23: Document Rescue and OCR - transcript
Jessica: The QUILL Cast, episode twenty-three. I'm Jessica, and today is the heavy document rescue episode. This is the one for the lease from two thousand eleven, the photocopied class handout, the court document from a scanner older than you, the photo of a page that arrived as a JPEG in an email. All those files that look fine on a sighted person's screen and produce nothing at all on a screen reader.
Liam: I'm Liam. Last episode, episode twenty-two, was the bridge to the rest of the world. The Word, EPUB, PDF, and friends episode. The headline there was the canonical source of truth. The editor buffer is plain text and Markdown, the hidden codes from episode twenty-one ride along, and Pandoc materializes those codes into real Word styles, real RTF runs, real HTML on the way out. The on-demand download we walked through, PDF and Office text extraction, is the same on-demand pattern that today's episode inherits, and the same honesty about what is and is not installed on your machine.
Jessica: Quick recap of the relevant piece for today. The on-demand PDF and Office text extraction is the package that gives QUILL the MarkItDown text extractor, the pdfplumber library, and the pypdf library. It is fetched from PyPI the first time you need it, sha-pinned, never silent. Last episode also named the gap: pdfplumber and pypdf only read the text that born-digital files already contain. A scanned PDF, a JPEG of a page, a faxed letter, none of those have a text layer. The honest answer from the PDF reader, the one we showed in episode twenty-two, is the message in the buffer: no selectable text was found, this is likely a scanned or image-only PDF, use File, Import, and choose OCR to read it. Today is that choice.
Liam: The one rule for this episode, the rule that governs every door in this tool, and the rule we want you to take into every future rescue attempt. Free first, local first, and nothing is ever uploaded without asking you. Free first means the cheapest tier is always tried first. Local first means the engine is on your machine, no internet round trip. And nothing is ever uploaded without asking means even the paid cloud tier, when it appears, requires an explicit yes for every single document. The TTS engine that reads you the prompt will read the same three sentences out loud every time you need to make this decision. That repetition is the design.
Jessica: Before the deep walkthrough, the do this now beat. Pause the audio, take thirty seconds, and do these four small things in QUILL. One, open the File menu and look at the Import submenu. Find the entry Import slash Convert Document with the parenthetical OCR after it, that is the door for today. Two, open the Tools menu and find OCR and Document Conversion, the submenu that holds OCR Image, Install Local OCR Engine with Tesseract in parentheses, OCR and Conversion Services, and Review Last OCR Result. Three, look at your screen and tell me, out loud or to yourself, which of those entries is grayed out or missing. Four, do not run anything yet. Press play, and we will walk the code-verified path that matches the menu you just saw.
Liam: The menu you are looking at is registered in quill-ui-main-frame, lines twenty-three-forty-eight to twenty-three-four-zero-five, and the labels are exact. Import slash Convert Document, the parenthetical OCR after the slash, is the file-dot-import-convert command. OCR Image, the file picker that runs the built-in Windows engine on a single image, is the tools-dot-ocr-image command. Install Local OCR Engine with the parenthetical Tesseract, is tools-dot-install-local-ocr. OCR and Conversion Services, the customer-facing plain-language page, is tools-dot-ocr-services. Review Last OCR Result, the spoken checklist for the last conversion, is tools-dot-review-last-ocr. And on the AI side, the one we will preview at the end, Describe Image is tools-dot-describe-image. Every one of those handlers lives in a mixin. The file-pick OCR and the clipboard and screen capture OCR live in quill-ui-main-frame-image. The Import slash Convert flow and the Tesseract install live in quill-ui-main-frame-docconvert. The Describe Image AI flow lives in the same image mixin. The point: this is real engineering, separated into testable mixins, not a single block of legacy code.
Jessica: The pipeline starts with Import slash Convert Document. The dialog supports born-digital formats, which are dot-docx, dot-pptx, dot-xlsx, dot-doc, dot-ppt, dot-xls, dot-html, dot-htm, dot-epub, dot-csv, dot-odt, dot-odp, and dot-ods, images, which are dot-png, dot-jpg, dot-jpeg, dot-tif, dot-tiff, dot-bmp, dot-gif, and dot-webp, and PDFs, which are their own ambiguous case. The code is in quill-io-docconvert, the helper supported-import-extensions returns the union. The set is a frozenset, the order is alphabetical inside the dialog mask, and the message in the dialog label reads All convertible documents with the extension list. There is no hidden tier. The dialog says what it will accept, the code says what it will accept, and the two match.
Liam: The router function is convert-document in quill-io-docconvert, line three-eighty-two. Read the docstring: born-digital types go to Tier one, images go straight to Tier two, PDFs try Tier one and come back flagged offer-local-ocr when the text layer looks empty. The caller owns the accessible escalation prompt, because QUILL never runs OCR without asking. The born-digital branch calls convert-born-digital, which calls convert-with-markitdown in quill-io-markitdown-bridge. The image branch and the PDF-empty branch call convert-with-local-ocr, which calls ocr-image-with-tesseract in quill-io-tesseract-ocr. The cloud branch, convert-with-cloud-ocr, calls convert-with-datalab from quill-core-datalab-ocr, and it is reached only through the escalation prompts we will get to in a moment.
Jessica: Tier one, the free local converter. The bridge is in quill-io-markitdown-bridge, the whole module is twenty lines, and what it does is exactly what its name says. It imports MarkItDown from the package of the same name, with enable-plugins false, calls convert on your file path, and returns text-content stripped and right-trimmed. The package itself is the Microsoft markitdown package on PyPI, MIT licensed, and the bridge raises a clean ImportError when the package is not on the import path. That is the moment you see the friendly error in the buffer. The error message is precise, and we will quote it. The free local converter, MarkItDown, is not installed. Open Help, Download Optional Components, and download PDF and Office text extraction to enable born-digital conversion.
Liam: Honest correction before we go further, because the brief said we should call out drift. The free local converter, MarkItDown, is not bundled with the QUILL installer. It used to be implied that way in older show notes, and the customer-facing services page in main-frame-docconvert still says installed with QUILL in its header. The current code says otherwise. The download is on-demand, about thirty megabytes, fetched from PyPI through the Download Optional Components hub, the same hub that handles Pandoc, the braille pack, Whisper, Kokoro, Piper, the spell dictionaries. That is the right model. Every byte that lands on your disk is a byte you approved. But if you read a line like installed with QUILL in the services page, treat it as aspirational copy that needs refresh, not as ground truth. The ground truth is the Download Optional Components dialog and the import-error message.
Jessica: The PDF branch has one more twist. When MarkItDown runs on a PDF and produces empty text, the function raises ValueError, the empty-output signal. The router catches that and returns a ConversionOutcome with text empty, tier markitdown, source the path, page-count the page count, and offer-local-ocr set to True. When MarkItDown runs on a PDF and produces some text but the text averages fewer than fifty characters per page, the heuristic text-layer-looks-empty, the same module line one-forty-six, returns True, and the outcome is flagged the same way. The constant is MIN-CHARS-PER-PAGE, fifty, and the check counts whitespace-stripped characters only, so a PDF of pure whitespace is correctly classified as scanned. Either signal, empty or too-sparse, hands control back to the caller with the offer-local-ocr flag set. The caller is the UI, the UI is quill-ui-main-frame-docconvert, and what it does with the flag is the user-facing escalation prompt.
Liam: The escalation prompt is the design moment of this episode. Read the wording in quill-ui-main-frame-docconvert, the function docconvert-finished, line one-fifty-seven. The exact message, with the filename interpolated, reads: QUILL could not find readable text in file-name. It looks scanned or image-based. Run free on-device OCR, local Tesseract? This stays on your computer and does not upload anything. Yes runs local OCR now. No opens the empty result anyway. Cancel stops. Three buttons, Yes, No, Cancel. Yes is the default, the focus is on Yes, and the message explicitly names the engine, Tesseract, and explicitly says the work stays on your computer. The TTS engine that reads you the prompt will read every word. The consent is not assumed, the consent is collected every time.
Jessica: The Yes branch is docconvert-run-local-ocr, the same file line one-eighty-one. The first check is tesseract-available, the function in quill-io-tesseract-ocr, line seventy-five. It is one call: discover-tesseract-executable with an optional settings override. The search order is the settings override, then the QUILL-managed component folder, which is app-data-dir slash ocr slash tesseract, then the system PATH, then the conventional Windows path C colon slash Program Files slash Tesseract-OCR slash tesseract.exe. If Tesseract is anywhere on this list, QUILL finds it. If not, the friendly message appears. The local OCR engine, Tesseract, is not installed yet. Install it now? QUILL downloads the official installer, about forty-eight megabytes, verifies it, and opens it for you to complete. Then run Import slash Convert Document again. The forty-eight figure is rounded copy in the prompt. The precise constant TESSERACT-DOWNLOAD-BYTES in quill-core-tesseract-install, line forty-six, is fifty million one hundred seventy-five thousand two hundred forty-eight bytes, which works out to about forty-eight megabytes, the rounded form is honest.
Liam: Setup, the one-time install, is its own command, the tools-dot-install-local-ocr entry. The downloader is in quill-core-tesseract-install. The first thing the installer does is refuse to run under safe mode, with the message Downloading the OCR engine is disabled in Safe Mode. The next check is platform, and the installer is Windows only. The URL is pinned. TESSERACT-DOWNLOAD-URL points to a re-publish of the official UB-Mannheim Windows installer, version five-dot-four-dot-zero, dated two thousand twenty-four June six, hosted on the Community-Access quill GitHub release under assets-v1, Apache-2.0 licensed. The fingerprint is pinned too. TESSERACT-DOWNLOAD-SHA256 is c-eight-eight-five-f-f-f-six-nine-nine-eight-e-zero-six-zero-eight-b-a-four-b-b-eight-a-b-five-one-four-three-six-e-one-c-six-seven-seven-five-c-two-b-a-f-c-two-five-five-nine-a-one-nine-b-four-two-three-e-one-eight-six-seven-eight-b-six-zero-c-nine. Verify the digest before the file is ever executed. The download is HTTPS with a verified TLS context.
Jessica: Then QUILL does the part that matters for trust. It does not silently install Tesseract. The install flow launches the official installer visibly, the same one you would have downloaded yourself, and you complete the install the way you would have completed it yourself. No surprise elevation, no background service, no system-wide configuration. After the installer finishes, QUILL finds the engine in its conventional location and you can run Import slash Convert Document again. The honest line in the announcement reads: the verified Tesseract installer is open. Complete it, then use Import slash Convert Document, QUILL will find the engine automatically. If you already have Tesseract on your machine, from anywhere, QUILL finds it on the next run with no download at all. On macOS, the help copy says, install it with Homebrew, brew install tesseract, and QUILL finds it on PATH.
Liam: When Tesseract is available and you said Yes, the OCR run submits to the QuillTaskManager background pool, so the editor stays responsive. The progress callback narrates page by page, recognizing page three of twelve, and the cancellation token is checked between pages. The PDF rasterization uses pypdfium2 at about one hundred fifty DPI, the same library the PDF reader uses for page extraction, and each page is recognized as an image with tesseract, language eng by default, and the recognized text is reassembled with the page markers you can search. The marker is exactly the HTML comment, less-than-bang-dash-dash space Page space N space dash-dash-greater-than, on its own line, with a blank line before and after the body text. Screen readers ignore the marker, search engines do not, and you can press Ctrl-F and type Page 7 to find page seven of the scan. The same convention is used by EPUB chapters when they are joined with form feeds, and the same convention is the page-navigation contract across the app.
Jessica: Now the honesty feature, the one that separates this from every OCR tool we have used. Tesseract reports per-word confidence on a zero to one hundred scale, and QUILL records it. The TSV parser in quill-io-tesseract-ocr, function parse-tsv, groups Tesseract's level-five rows into lines by page-block-paragraph-line keys, separates paragraphs with a blank line, and averages word confidences into a line confidence. The router function in quill-io-docconvert computes a mean across all lines, and the result is stored on the ConversionOutcome as mean-confidence. The threshold is WEAK-OCR-CONFIDENCE, sixty. Mean below sixty gets a warning, mean above sixty is trusted. The warning text reads, on-device OCR confidence is low, NN out of one hundred. Review the result carefully, a cleaner scan may recognize better. The threshold and the warning text are constants, and the warning is the only honest answer to a poor scan.
Liam: The announcement on the new tab reads, opened file-name from free on-device OCR, N pages, confidence NN out of one hundred. The page count is in the announcement only when there is more than one page. A single-image run says one page, confidence NN, and stops. A clean high-confidence scan comes back with a confidence in the eighties or nineties, and the warning tuple is empty. A crooked fax from twenty-five ninety-eight comes back flagged with a low confidence and a warning, and the new tab is opened anyway, modified equals True, because the rescued text exists nowhere else and the close-and-save prompts must protect it. Issue seven-ninety is the review that hardened this contract, and the comment in the code says exactly that. Closing a pathless unmodified tab would silently discard the whole result, and the conversion is too important for that.
Jessica: The next tool on the menu, Review Last OCR Result, is the screen-reader-first way to walk the trouble spots. The handler is review-last-ocr-result in quill-ui-main-frame-docconvert, line three-forty-five. It opens a named scratch tab with the conversion's provenance header. The header lines are, in order, OCR Review, blank line, Source colon file-name, Converted with colon one of three tier labels. The Tier one label is Free local converter, MarkItDown, no OCR was needed. The Tier two label is Free on-device OCR, Tesseract. The Tier three label is Datalab Chandra cloud OCR, consented upload. Then Pages colon N, then Mean recognition confidence colon NN out of one hundred, then any warnings. If the conversion came from Tier two and any line had confidence below the threshold, the body of the review lists every flagged line pre-formatted as Page N colon space, square-bracket-NN percent-square-bracket, space, the line text. Search the converted document for the page marker to jump there. That is the spoken checklist. Arrow through it once, search the converted document for the trouble spot, fix only that, save. No re-proofreading the whole result.
Liam: The OCR review surface has a more advanced cousin we will get to in the next episode, the OcrReviewDialog in quill-ui-ocr-review-dialog, with the Insert, Copy, and Discard buttons. That surface is the one the file-OCR and clipboard-OCR and screen-capture flows use, and it is the surface episode twenty-four extends. The Import slash Convert flow opens a real editor tab, the review-last-ocr-result flow opens a named scratch tab. Two different review surfaces, both accessible, both honest about what they are.
Jessica: The cloud tier. The brief asked us to be honest about drift, and the previous show notes for this episode called the cloud tier planned. It is no longer planned, it is shipped. The Datalab Chandra cloud service is reachable from Import slash Convert Document, but only after the free tiers fall short, and only with an explicit consent prompt for every single upload. The flow is the same file, the function docconvert-offer-cloud, line two-seventy-one. The exact prompt reads, on-device OCR finished, but the result looks low-quality or the layout is complex. For higher accuracy you can convert it with Datalab cloud OCR. This uploads the file to a cloud service and may cost money. Yes converts with cloud OCR. No keeps the local result. Two buttons, Yes, No, no default, the focus is on No. If you say Yes, the consent-gated upload prompt appears, the one named Cloud Upload Consent.
Liam: The consent prompt is the design moment. Read the wording, quill-ui-main-frame-docconvert line two-ninety-six. The body says, QUILL will send this document to Datalab for OCR and document conversion. Only continue if you are allowed to upload this file to that service. Consider whether the document contains private, medical, legal, educational, financial, employment, or confidential information. Datalab says conversion results are deleted from its servers about one hour after processing. QUILL retrieves the result promptly and keeps it only in the opened document. The filename is checked by looks-sensitive, the function in quill-core-datalab-ocr, and if the name suggests sensitive content, a CAUTION line is prepended. The dialog is wx-ICON-WARNING, the buttons are Yes and No, and No is the safe default. The button you press is logged in the status line. Cloud conversion cancelled, nothing was uploaded is the message when you back out. That is the consent culture applied uniformly, and the TTS engine reads every word.
Jessica: The cloud tier also has an API key, and the key is stored in the Windows credential vault, never in the settings file. The Datalab service is enabled in the AI Hub, on the Services tab. The mode setting, balanced or accurate, the output format setting, markdown, html, or json, and the paginate setting, are all honored on the upload. The Datalab account itself is your account, your monthly free allowance, your billing, and QUILL never sees your billing. The page in the services overview says so in plain language, and the page is reachable from Tools, OCR and Conversion Services, the tools-dot-ocr-services entry, the same handler show-ocr-services-overview that appends the current engine status to the markdown body. The live status is the only moving part. Everything else in the page is a static reference.
Liam: A few more honest corrections before homework. First, the old show notes conflated the built-in Windows OCR engine with Tesseract. They are different. The built-in Windows engine is the Windows.Media.Ocr runtime, exposed through the winrt packages, the ENGINE-WINDOWS id in quill-io-ocr, used by the file-OCR and clipboard-OCR and screen-capture flows. The Tesseract engine is the cross-platform Tesseract binary, the ENGINE-TESSERACT id in quill-io-tesseract-ocr, used by Import slash Convert Document. Two engines, two install stories, two review surfaces, one shared outcome model. If a past claim said they are the same, it conflated them.
Jessica: Second, the previous show notes for this episode said nothing is ever uploaded about a cloud tier that is planned. The cloud tier is not planned, it is shipped, and the consent is per-document, not per-session. The opt-in is in the AI Hub, the consent is at the moment of upload, and the deletion guarantee is from Datalab, not from QUILL. If a past claim said the cloud tier is future, it drifted. The code, the consent prompt, the SERVICES_OVERVIEW page, and the AI Hub Services tab all reflect the current state.
Liam: Third, the page markers are HTML comments, not Markdown headings. They are invisible to screen readers, visible to the search field, and stable across the round trip into Pandoc formats. The marker is the contract that makes Page seven searchable, that makes the OCR review checklist useful, and that makes the page navigation in the EPUB Navigator work the same way on PDF rescues. The marker is in the code, the marker is in the search, the marker is in the review. If a past claim called them Markdown headings, it drifted.
Jessica: Fourth, the temp folder. Every OCR run creates a per-run temporary directory with the prefix quill-ocr-dash, the conversion result is written to the opened document, the temp directory is cleaned up by the TemporaryDirectory context manager. The leftover-clearing tool, Delete OCR Temporary Files, clears the app-data ocr-jobs folder, the one used for the older single-image flow, and announces how many entries it removed. The honest answer when there is nothing to remove is, no OCR temporary files to delete. That is the design.
Jessica: Homework, four steps, the way every episode closes. One, open QUILL and run Tools, OCR and Conversion Services. The page is the customer-facing reference, and the bottom of the page shows the live engine status. Read the status. Is the local OCR engine installed? Is the Datalab cloud service enabled? The answers are written in plain language. If the local engine is missing and you want it, run Install Local OCR Engine with the Tesseract parenthetical, complete the installer, and run this step again.
Liam: Step two, find a real scanned document. The lease from two thousand eleven. A photo of a page from a class. The court form from a scanner older than you. Run File, Import, Import slash Convert Document, pick the file, and watch the pipeline. If the file is a real scan, the escalation prompt appears. Say Yes, listen for the per-page progress, wait for the announcement of mean confidence, and save the result. If the result is a single page, the confidence is the only quality signal. If the result is a multi-page scan, every page boundary is searchable.
Jessica: Step three, run Tools, Review Last OCR Result. Walk the review surface. If the mean confidence is below sixty, the body of the review lists every flagged line pre-formatted with its page and confidence. Search the converted document for the page marker on a flagged line, fix only that line, save. The point of the review is that you do not re-proofread the whole result. The honest answer to a low-confidence scan is targeted repair, not full retyping.
Liam: Step four, the opt-in test, only if you are comfortable. If the Datalab cloud service is enabled in your AI Hub, run a scan through Import slash Convert Document, watch the Tier two warning appear, and watch the cloud-tier prompt appear. Read the consent wording, decide which document you would actually upload, and most of the time, decide no. The point of the test is the consent, not the upload. The TTS engine that reads the prompt will read every word, and the muscle memory of clicking No is the part of the design that protects you. If you do say Yes, the per-document audit log is the safety net.
Jessica: The architecture lesson. The free-first, local-first, consent-first design law is the same law every QUILL feature that touches the network follows, and the document rescue pipeline is the hardest case. Born-digital goes to the cheapest tier. Empty or scanned escalates to the next free tier, with a prompt. Weak escalation goes to the cloud, with a per-document consent. The escalation prompts are accessible dialogs with focus on the safe default, the wording is read by the TTS engine, and the user's working copy is always in the most accessible format alive. The hidden codes from episode twenty-one do not appear in rescued text, the result is plain text and Markdown, and the screen reader always knows what to do.
Liam: Next episode, episode twenty-four, OCR and image describe. The lighter OCR flow, the file picker, the clipboard image, the screen capture, the three entry points that share the OcrReviewDialog with Insert, Copy, and Discard. And on top of that, the AI vision flow, the Describe Image command, the one that asks a connected vision model to describe a photo or a screenshot and offers the description for insertion, copy, or discard through the same review dialog. The OCR review surface we built in this episode is the foundation. Episode twenty-four adds the visible image input sources and the AI image description, and the two review surfaces converge. Same dialog, different content, same honesty about what was recognized and what was guessed.
Jessica: The QUILL Cast is a fifty-four episode course, and we are twenty-three of the way through. Past the first third, into the deep middle, where the systems are real and the engineering is something you can rely on. The audio arc returns after this two-episode deep middle, the AI hub follows, then the vault, then GLOW, then braille, then Quillins, then the finale. The road is long, the road is good, and we will walk it together.
Liam: I'm Liam.
Jessica: I'm Jessica. Free the words.
Liam: The QUILL Cast is a fifty-four episode course on QUILL. Subscribe wherever you listen, and we will see you in episode twenty-four.