Resumable Background Indexing Against an API That Doesn't Support What You Need

Indexing a photo library means walking every file, resolving its capture date, and writing a row per file — five hundred at a time, in a background job that can be killed at any point by a server restart, a timeout, or a worker recycling. The scan has to be resumable, which usually means: keep a cursor, and next time ask for everything with an id greater than it. Photo Sweep's indexer can't do that, because the API it's built on won't let it ask.

Ordering by file id keeps an offset about as stable as an offset gets.

The constraint

Nextcloud's ISearchQuery — the interface behind Folder::search() — only accepts eq and in comparisons on the fileid column. There's no gt. A cursor of the form "give me everything past file id 41,802" simply cannot be expressed as a search query on this API, full stop. It's not a performance concession or an oversight to work around; it's not there.

So MediaFinder::findBatch() pages by offset instead, ordered ascending by file id:

$query = new SearchQuery(
    new SearchBinaryOperator(ISearchBinaryOperator::OPERATOR_OR, $mimeComparisons),
    $limit,
    $offset,
    [new SearchOrder(ISearchOrder::DIRECTION_ASCENDING, 'fileid')],
);

Why ordering by file id specifically matters

An offset cursor is normally fragile: insert or delete a row ahead of your current position and the whole window shifts, silently skipping or re-visiting entries. Ordering by file id is what keeps this particular offset usable despite that. New uploads always get a higher file id than anything already in the library, so they land past whatever window the scan is currently in rather than shifting it. A scan in progress never has new arrivals reshuffle underneath it — only deletions ahead of the cursor can, and even then the effect is bounded to letting one file slip past a single pass.

The scan state — offset, count found so far, whether the pass completed — is written to the database after every batch of 500, not just at the end:

$scan->setCursorOffset($scan->getCursorOffset() + count($files));
$scan->setFound($scan->getFound() + $indexed);
$scan->setUpdatedAt(time());
$this->scanMapper->save($scan);

A process killed mid-scan costs exactly one batch of up to 500 files, never the whole pass, and the next cron tick or web request picks up from the saved offset.

Two run budgets, for two callers

The same scan() method serves both a background cron job and an interactive "index now" button in the web UI, with a different ceiling on how many batches to run per call:

public const WEB_BATCHES = 8;
private const MAX_BATCHES_PER_RUN = 40;

A web request has to return inside the server's request timeout, so it gets a small budget and shows the user real progress — four thousand files, then hand back control — rather than sitting on a spinner and eventually failing. Cron gets a much larger budget (40 × 500 = 20,000 files per run) so one user's very first scan doesn't monopolize the whole cron cycle indefinitely; it just takes more ticks to finish.

There's also a stale-run guard: a running flag older than thirty minutes is treated as a crashed process rather than an active one, because without that check a killed worker would permanently lock a user out of ever scanning again.

The sweep that makes the cursor honest

An offset cursor can still let a file slip through — one deleted exactly at the boundary between two batches, in the narrow window before the next batch reads it. That's what finishPass() cleans up: once a full pass reaches the end of the library, everything still present has been re-stamped with the current timestamp, so anything in the index older than the pass's start time is provably a file that's gone:

private function finishPass(string $userId, Scan $scan): void {
    $removed = $this->mediaMapper->removeStale($userId, $scan->getStartedAt());
    // ...
    $scan->setComplete(true);
}

The cursor gets you through the scan cheaply; the sweep is what makes the end state correct regardless of what happened during it. Neither piece alone would be enough — the cursor without the sweep would slowly accumulate ghost entries for deleted files, and the sweep without the cursor would mean redoing the whole library on every resume.

Series: Photo Sweep.