I Remember the Screenshot, Not the Conversation: Adding Windows OCR to YuGPT

Remembering part of an answer is useful when you can search your saved conversations. But what if the words you remember were inside an image?

An error dialog. Part of a settings screen. A diagram you meant to revisit. If the words do not appear in the surrounding conversation, searching its text will not find them. A filename such as image.png offers little help either.

This is the story of making “perhaps it was in a screenshot” searchable.

I develop YuGPT, a Windows application for backing up ChatGPT and Codex history locally and searching across saved conversations. I have added image search and recognition of text inside images using Windows OCR to the Microsoft Store Edition.

Text search and image search sit beside each other at the entrance. Some conversation titles are blurred.

OCR stands for optical character recognition: reading characters from an image. The central API call is short. Making it useful in an application took more thought, particularly around Windows package identity and the relationship between recognition and searching.

Finding an image does not always start with OCR

You might remember that an image appeared during a discussion about GIMP, without remembering anything written in the image itself. YuGPT can first use conversation titles, bodies, and comments to find conversations containing images. Source, backup-set, and date filters help narrow the selection.

These results come from conversation text. “OCR not checked” means the image has not been checked for recognized text; this is not an OCR-result screenshot.

When the remembered words exist only in the picture, there is another route. Select “Search only text inside images” and press Search. YuGPT performs the required recognition within the current filters, then searches that text. Selecting the checkbox alone does not start processing.

This feature follows conversation context or recognized words. It does not search for pictures with a similar visual appearance.

From a result, you can open the image or return to its conversation. Sometimes the screenshot is what you remember, while the explanation you need is in the exchange around it.

Finding the image gives you a route back to its original context.

Why a WPF application needs to think about package identity

I used Windows.Media.Ocr.OcrEngine. Recognition uses Windows facilities rather than sending the image to an external AI service.

There is an important desktop requirement: Microsoft documents Windows.Media.Ocr as supported for desktop applications with package identity.

YuGPT is a WPF app. WPF is not the obstacle here; the process’s runtime identity matters. Microsoft describes deploying WPF and other desktop apps through MSIX to obtain that identity.

For YuGPT, this means running the installed MSIX application. Starting a built EXE directly is a different execution context. Buying through the Store is not itself the Windows API requirement. YuGPT’s choice to offer image search in its Store Edition is a separate product decision.

“Package ID” also deserves a little precision. Microsoft’s package-identity overview distinguishes these concepts:

YuGPT’s manifest uses YutakaHirata.YuGPT as its Identity Name. This is different from the display name or the product ID at the end of the Store URL. Merely having a manifest file beside an executable does not give a directly launched process package identity.

Check the environment and recognition languages first

The Windows adapter is in src/YuGPT.App/WindowsImageOcr.cs. The following excerpts come from the implementation inspected for this article. Line breaks and indentation have been adjusted for readability. They depend on surrounding application types and are not standalone programs.

First, the app checks the environment and usable languages:

internal static Availability CheckAvailability()
{
    try
    {
        // Preserve the actual error when OCR fails in an installed package.
        _ = Windows.ApplicationModel.Package.Current;
        var languages = OcrEngine.AvailableRecognizerLanguages
            .Where(language => OcrEngine.TryCreateFromLanguage(language) is not null).ToArray();
        return new(languages, languages.Length == 0 ? L.T("image.no_languages") : null);
    }
    catch (Exception ex) { return new(Array.Empty<Language>(), ex.Message); }
}

Package.Current obtains the current app’s package; it does not create an identity or install anything. Microsoft documents this property as supported in packaged processes. If the check fails, the method retains the exception message.

Having identity does not guarantee usable recognition languages. The code enumerates AvailableRecognizerLanguages and tests creation through TryCreateFromLanguage. That factory returns null when the requested language cannot be resolved to an available recognizer.

The app’s interface language and the language used to read an image are separate choices. YuGPT lets the user select from available recognition languages and processes the selected languages individually. It does not assume that every language visible in a picture will be recognized automatically.

An error also should not be flattened into “no languages installed” when something else failed. Keeping the reason gives the UI something more useful to explain.

Prepare the image before asking Windows to read it

For recognition, the session obtains an engine for the selected language. It keeps engines by language so they can be reused within that session:

_ = Windows.ApplicationModel.Package.Current;
if (!_engines.TryGetValue(tag, out var engine))
    _engines[tag] = engine =
        OcrEngine.TryCreateFromLanguage(new Language(tag))
        ?? throw new InvalidOperationException(L.T("image.no_languages"));

The image is then opened and decoded:

var file = await StorageFile.GetFileFromPathAsync(Path.GetFullPath(path)).AsTask(token);
using var stream = await file.OpenReadAsync().AsTask(token);
var decoder = await BitmapDecoder.CreateAsync(stream).AsTask(token);
var scale = Math.Min(1.0,
    (double)OcrEngine.MaxImageDimension /
    Math.Max(decoder.OrientedPixelWidth, decoder.OrientedPixelHeight));
var transform = new BitmapTransform
{
    ScaledWidth = Math.Max(1, (uint)(decoder.PixelWidth * scale)),
    ScaledHeight = Math.Max(1, (uint)(decoder.PixelHeight * scale))
};
_bitmap = await decoder.GetSoftwareBitmapAsync(
    BitmapPixelFormat.Bgra8, BitmapAlphaMode.Premultiplied,
    transform, ExifOrientationMode.RespectExifOrientation,
    ColorManagementMode.DoNotColorManage).AsTask(token);
_path = path;

The scale respects the engine’s maximum image dimension. Math.Min(1.0, ...) reduces oversized images without enlarging smaller ones.

A long screenshot has a very different aspect ratio from an ordinary photograph. The calculation uses the larger dimension, rather than checking width alone. The oriented dimensions account for orientation, and decoding specifies RespectExifOrientation too.

The BitmapDecoder API accepts pixel-format, alpha, transform, and orientation settings. This implementation requests Bgra8 with premultiplied alpha for the recognition bitmap. The original attachment is not resized and overwritten.

Only then does recognition happen:

var result = await engine.RecognizeAsync(_bitmap).AsTask(token);
return string.Join(Environment.NewLine, result.Lines.Select(line => line.Text));

The recognized lines become a string joined with newlines. AsTask(token) passes cancellation into the awaited operation; it is not a transaction that erases results already saved. The application handles recognition, saving, and cancellation as separate states.

The session disposes of the SoftwareBitmap when switching to another image and when the session ends. While processing one image in multiple languages, it reuses the decoded bitmap. Repeated preparation is easy to overlook when testing one image, but more noticeable across an archive.

You cannot filter by words you have not read yet

Connecting recognition to search creates another small trap.

Suppose the query is “error.” Would it be faster to run OCR only on images containing that word?

We do not know which images contain it yet.

In src/YuGPT.Core/ImageSearch.cs, recognition candidates therefore discard the query and excluded words:

var candidates = Candidates(options with
{
    Conversation = options.Conversation with
    {
        Query = "",
        Exclude = ""
    }
});

Source, backup set, date range, and an explicitly selected conversation still narrow the work. Within that scope, the app obtains the required recognition results before applying the text query. An unread image must not be treated as an image whose text failed to match.

This is search-application logic rather than an OCR API requirement. An optimization that removes the thing we wanted to find would be a rather unfortunate improvement.

Reuse recognition, preserve what it actually said

Reading the same image again on every search would add waiting time. YuGPT stores results using the image content’s SHA-256 hash and the recognition language:

private string CachePath(string hash, string language)
{
    // Hash language tags too: no user-controlled path segments.
    var tag = Convert.ToHexString(SHA256.HashData(
        System.Text.Encoding.UTF8.GetBytes(language.ToLowerInvariant())));
    return Path.Combine(_directory, hash + "-" + tag + ".json");
}

Two attachments called image.png can have different contents and therefore different hashes. One image recognized as Japanese and as English gets separate cached results. Hashing the language tag also avoids placing the tag directly into a filename component.

Results are JSON files under .image-ocr/v1 in the backup library. The indexing code checks the image hash again after recognition, so a file changed during processing does not receive results associated with its earlier contents.

The hash identifies content; it does not encrypt the recognized text. OCR results are stored as plaintext. Recognition and searching of saved images happen locally, but a shared or synced backup folder still needs appropriate handling. Direct backup from ChatGPT is a separate operation that requires connectivity.

OCR also makes mistakes. Japanese text can acquire spaces inside what a user regards as one word. YuGPT normalizes some whitespace between Japanese-script characters for matching, while retaining the original recognized text. It does not remove every space and turn English Microsoft Store into MicrosoftStore.

Rather than pretending every recognition is correct, I want the result to remain inspectable and disposable. Clearing OCR results leaves the original conversations and image attachments in place, allowing recognition to be performed again when needed.

The thing we are looking for is beyond the text

Looking back at this implementation, the central recognition call is surprisingly small. Before it come Windows identity and language checks. After it come storage, reuse, and decisions about what a search result means. Much of the work of making a useful tool lives around the API call.

Someone using the app does not need to remember package identities or image hashes. They need to get back to the image and its conversation. The developer can deal with the slightly awkward details. That seems a fair division of the work.

I added this feature in the hope that fewer “where was that image?” moments will interrupt whatever someone was trying to understand.

The YuGPT website has more details and screenshots. Image search and OCR described here are features of the Microsoft Store Edition; usable Windows recognition languages depend on the PC.

The app can now read words inside images. I would also like to find the image before I forget why I was looking for it.