Digitizing Devotional Poetry That Only Exists as Scanned Images

Why a Django app exists to scrape, OCR, and re-typeset Qasidas: because most of the source material is a photograph of a page, not a text file.

Digitizing Devotional Poetry That Only Exists as Scanned Images

The first thing I learned about this library is that "the text" is often not a piece of text at all. It's a JPEG of a typeset page, or a PDF where the letters were placed one glyph at a time with kashida padding, or an HTML page whose only useful content is a linked scan behind a document viewer. Qasidas — devotional poems in the naat and hamd traditions, mostly in Urdu and Arabic, some in Persian — circulate on small community sites that were never built to be read by anything but a browser and a human eye.

Five sites, three failure modes: photographs, a blocked origin, a shattered PDF text layer
Five sites, three failure modes: photographs, a blocked origin, a shattered PDF text layer

Five source sites, one archive: 3,302 Urdu works and 473 Arabic, most read off a page rather than typed.

Why this exists at all

The app is a Django 5.0.7 project (psycopg2-binary, Celery + Redis, BeautifulSoup4, PyMuPDF, Pillow, pytesseract, argos-translate) whose job is to crawl a small set of naat and qasida sites, extract whatever verse structure they publish, and store it in one normalized schema — title, native-script title, author, language, lyrics, transliteration, translation — so a reader can search, tag, and download a typeset PDF regardless of which site the poem originally came from.

None of that is interesting on its own. What makes it a real engineering problem is that "extract the text" turns out to mean five different things depending on the source:

  • mynaatbook.com ships its data inside a minified React JS bundle; the scraper fetches the bundle and regexes JSON-like objects out of it.
  • desertechoblog.wordpress.com is ordinary WordPress markup — the easy case, div.entry-content.
  • damas.nur.nu publishes qasidas as PDFs and JPG scans behind an embedded document viewer, discoverable through its WordPress REST API rather than its own archive page (which only shows the first 50 posts).
  • lyrics.midhah.com is a Next.js site with build-hashed class names — useless for markup scraping — but it embeds a full schema.org MusicComposition in JSON-LD, title/author/language/lyrics/stanza breaks intact.
  • qasidacollection.com sits behind a Vercel bot challenge that answers every path, including robots.txt, with a 429. It's imported from Internet Archive Wayback snapshots instead.

The two-part problem

Once a source is reachable at all, there are two separable problems: getting the text off the page, and knowing whether what you got is right. The damas PDFs are the sharpest case of the second. Some of them place every glyph as its own positioned character with kashida (tatweel) justification stretching the line — visually correct, structurally useless, because extracting in stream order gives you the right letters in the wrong order. That's not a crawler bug to fix with a better selector. It needs the glyph coordinates read back out and reassembled right-to-left, row by row, which is a different kind of code than a scraper — closer to a tiny layout engine than to BeautifulSoup.select().

And when even that fails — no usable text layer, only a photograph of a page — the fallback is to rasterize the page and hand it to Tesseract, with the Arabic (tesseract-ocr-ara), Urdu (tesseract-ocr-urd), and Persian (tesseract-ocr-fas) language packs installed in the container image. That's the subject of the next post: what the actual OCR pipeline looks like, and why a naive pytesseract.image_to_string() call is nowhere near enough.

The posture that follows from this

Because so much of the corpus arrives with some doubt attached — is this text OCR output, a reassembled PDF, or a clean JSON-LD lyric block? — the schema carries a text_quality field (ok, poor, ocr) rather than pretending every row is equally trustworthy. A work whose text was reconstructed automatically says so on the page and in the exported PDF:

if qasida.text_quality != 'ok':
    notes.append('This text was reconstructed automatically and may contain errors.')

That's not a hedge for legal reasons. It's the only honest thing to say when half your library never had a text file to begin with — it had a picture of a book, and a pipeline that tried its best to read it.

Series: Qasida App.