Why AI-Core Separates Capture from Processing

An input record and the work performed on that input have different lifecycles. AI-Core makes that distinction explicit instead of letting a worker rewrite the original record while it processes it.

The current architecture is:

Capture
   ↓
processing_jobs
   ↓
processor / worker
   ↓
immutable processing result

A Capture is raw human input. A processing result is a versioned machine-generated interpretation. The queue state between them is durable execution state.

Capture is the evidence boundary

AI-Core stores the original source, text, URL, metadata, and identity of a Capture. Workers read that record; they do not update the captures row. The architecture documentation states this as an invariant: raw Capture is immutable and workers never update captures.

That rule is useful even for a small system. A processor may be replaced, fail halfway through, or produce a different interpretation after a version change. None of those events should rewrite the evidence that entered the system.

Capture creation also establishes the default processing job. The queue test verifies that one Capture creates one pending job for the default processor, and that repeating idempotent Capture creation does not create a second job.

The queue owns execution state

processing_jobs contains the state that changes during execution: status, attempts, error information, retry schedule, and completion information. The worker claims a pending job, reads the Capture, invokes a processor, validates the structured output, persists an immutable result, and completes the job.

The documented flow is intentionally conservative:

claim → read → process → validate
     → persist immutable result → complete

The current worker loop is one-at-a-time and ordered by existing creation time. There is no Redis, RabbitMQ, Kafka, priority queue, or concurrency layer in this implementation. SQLite is the durable queue for this phase because the authoritative state already lives there and the required queue operations are local transactions.

Retry is state, not a loop variable

A retryable processor failure sets the job to failed with next_retry_at. The worker does not immediately spin on it. The next attempt becomes eligible only after bounded exponential backoff and while the job remains below max_attempts.

Permanent or unknown failures do not receive an automatic retry timestamp. An internal manual retry can put a failed job back into pending, preserve the historical attempts and error, and extend the attempt budget for a terminal job.

The retry tests verify the schedule, the distinction between failed_waiting and failed_ready, the transition back to processing, the attempt counter, the terminal limit, and persistence after the database is reopened. These tests demonstrate the queue contract; they do not measure worker reliability in a deployed environment.

Stale recovery is separate from retry

A job in processing may be left behind when its worker dies. recover_stale_processing_jobs() returns a job whose started_at is older than the configured timeout to pending, clears the active start marker, and records recovered stale processing job as the last error.

That is logical work recovery. It is not the same as restarting the Python process. The process restart only gives a new worker a chance to inspect durable state; stale recovery changes a stranded job back into runnable queue state.

Results are replaceable in meaning, not mutable in history

The result includes processor identity, processor version, creation time, and a versioned enrichment contract. The result is persisted separately from the Capture and is protected by database rules against updates and deletes. A later processor can create a new interpretation without changing the original Capture model.

This also keeps the worker replaceable. A worker only needs to honor the existing contract: read a Capture, claim a job, produce validated structured output, persist the result, and mark completion. The storage model does not need to know whether the compute client is Hermes, a deterministic mock, or another worker.

What this architecture does not claim

The current source explicitly does not implement parallel processing, public processing/retry APIs, Redis, RabbitMQ, Kafka, dashboards, or a production-scale scheduler. Calling SQLite a durable queue here means that the queue state survives process boundaries and can be recovered by the existing worker; it does not imply a benchmarked distributed queue.

The separation is therefore a boundary, not a promise of unlimited scale: Capture preserves what arrived, processing_jobs records what must happen, and results preserve what a processor concluded.

Self-review

• The article describes only tables and flows present in AI-Core documentation and source.
• “Immutable” is used for Capture and result records where the schema/tests enforce that boundary; mutable queue state is described separately.
• Retry and stale recovery are identified as tested behavior, not production statistics.
• No unsupported worker topology or external queue was introduced.
• The open limitation is the deliberately single-worker, local SQLite phase.

Sources

• AI-Core repository: README.md
• AI-Core repository: architecture/README.md
• AI-Core repository: storage/database.py
• AI-Core repository: processing/worker.py
• AI-Core repository: tests/test_processing_queue.py
• AI-Core repository: tests/test_retry_policy.py