This page follows one bounded collection batch from its frozen plan to saved graph records. Each step is labeled as a local script, Codex research and judgment, or a human decision. It describes what is implemented now and where the path still needs orchestration.
Reviewed 14 October 2026 · Prototype documentationA collection can have many runs; a completed batch does not mean the topic is complete. Detailed parameter contracts live in the project’s docs/RESEARCH_COLLECTION_RUNS.md.
Local script means deterministic code that can be rerun and validated against saved files. Codex inference means the language model reads or interprets source material, makes a bounded scholarly judgment, or creates a structured packet. This does not mean the model has proved a claim. Human decision is reserved for starting or changing scope and exceptions that cannot safely be resolved from explicit evidence.
Select the collection, run purpose, seeds or assessed pairs, and named profile. The plan freezes the graph snapshot, parameters, target list and batch boundaries.
Local support: atlas.research_runs plan, validate, and dashboard status.
Traverse the selected graph under the saved neighborhood rules. Compare target pairs with current treatments, prior research decisions, source captures and the global draft library so already-covered work can be reused or skipped according to run type.
Local support: atlas.research_frontier, atlas.collection_assessment, pair-finding/history and run planning.
For tier 1, search the saved SEP/IEP indexes and exact-phrase fallback, score suitable roots, then inspect their pages for eligible direct links. The frozen profile controls root count, per-root followed-page allowance and depth; depth remains one.
Local support: atlas.research_source_discovery. Selection is a lead, not evidence that the page contains a treatment.
Deduplicate canonical URLs and versions, attach target and draft-reuse matches, preserve why each source was selected, and validate all document limits before fetching.
Local support: atlas.research_worklists; a worklist revision preserves the actual root and followed URLs.
Fetch only URLs in the frozen worklist. Save local HTML captures, hashes, response metadata, per-document timing and failures; reuse an existing verified capture where possible.
Local support: atlas.research_acquire. It does not infer, add graph records, or discover extra links.
Convert acquired HTML into section-labeled reading copies and save counts and hashes in a receipt. This is formatting for review, not summarization.
Local support: atlas.research_text. Reading copies remain operational files; source provenance remains in the tracked artifacts.
Read the bounded source pages, record a disposition for each, and extract concise supported treatments and relevant entities. Include dates with their basis and precision, source locators, limitations, and useful adjacent summaries. A passing mention is not a treatment or a reusable draft.
Output: a structured interpretation packet, not a direct database write.
Compare proposed records with current graph identities and drafts. Decide whether each complete item belongs in this collection, should remain a global draft, is a duplicate, or needs an exception. Resolve dates and endpoint dependencies; do not invent dates or merge uncertain identities by label alone.
Codex supplies the evidence-based decisions; validators enforce fields, source links, date review and endpoint closure. Ambiguous identities or missing evidence are human-review exceptions.
Check packet structure, one disposition per acquired page, citations to acquired sources, stable IDs, treatment dependencies, date-review completion and graph consistency. Validation errors block publication.
Local support: atlas.research_interpretation, atlas.page_harvest, and atlas.treatment_batch validators.
Save source-supported adjacent records to the shared draft library; publish selected, validated records as additive graph snapshots; pin manifests and receipts to the batch; then synchronize and close the run when its work is accounted for.
Local support: atlas.research_interpretation, atlas.treatment_batch, and atlas.research_runs sync/close. Publication is additive and provenance-preserving.
Review the Collection dashboard and Review workspace. Correct a wrong or weak record by marking it uncertain/removal-recommended and pruning it from the active graph; restore it if appropriate. Step in earlier only when a script fails, sources are insufficient, identity/dating is ambiguous, or the plan needs to change.
The goal is a post-hoc quality pass, with interruption for defined exceptions—not approval of every routine record.
| Handoff | Saved artifact | Why it matters |
|---|---|---|
| Plan → sources | Versioned run and batch JSON, profile and input snapshot pins | Recreates which targets and settings the run used. |
| Discovery → acquisition | Source-choice receipts, frozen worklist revisions, local source captures and acquisition receipt | Shows what was selected, followed, reused, skipped or failed. |
| Acquisition → interpretation | Reading-copy receipt and interpretation packet | Connects each summary and locator to a particular captured page. |
| Interpretation → graph/drafts | Validated additive batch, manifest, draft-library update and run results | Separates published graph additions from reusable drafts and records their provenance. |
SQLite serves the app, but committed JSON packets and source metadata are the reproducible research record. A lost local database can be rebuilt from the preserved source and additive artifacts.
| Area | Current state | Remaining work for routine unattended execution |
|---|---|---|
| Planning and reuse | Scripted, bounded | Invoke the existing planner and reuse checks from one run controller; stop on stale inputs or invalid parameters. |
| Tier-1 source discovery and acquisition | Scripted, deterministic | Chain discovery, worklist freeze, acquisition and text preparation with resumable receipts. |
| Source interpretation | Codex-led | Keep interpretation as the explicit LLM stage. It is currently produced through this Codex workflow, not an automatic local model call. |
| Reconciliation and publication | Validated, partly orchestrated | Automatically run validators, route unambiguous ready items, publish, and advance status; stop on unresolved identity/date/provenance exceptions. |
| Human review | Post-hoc + exception policy | Define exception thresholds and expose a clear paused/needs-attention state; retain removal and restoration as reversible corrections. |
Codex currently performs the interpretive stages, while local scripts handle acquisition and validation. Model configuration is not a parameter of the current research runner. A future controller could coordinate these stages through their saved inputs and outputs; this is an implementation direction rather than a capability of the current dashboard.
Each source in a new interpretation packet records whether a specific relationship was found, none was supported, or the source was insufficient. Saved treatment summaries can also support original conceptual comparisons without another source crawl. These model-inferred parallels and contrasts retain their reasoning and source links, and remain distinct from historical response, adoption or influence. Unpublished relationship proposals stay with the source harvest for later reuse.
The research timing ledger records wall-clock intervals for source selection, worklist, acquisition, interpretation, text preparation, source reading, drafting, reconciliation, validation and publication. Automated scripts can also report their own elapsed time, such as fetch time per document. The ledger’s human-scale intervals may include waiting, review and handoffs; they are not measurements of model compute or private reasoning time.
When a stage fails, preserve its receipt and error, leave the batch at its last verified stage, and resume from the saved artifact rather than repeating successful acquisition. A run should pause only for explicit failures or unresolved evidence decisions. Budgets are ceilings, never quotas: a run reports shortfalls instead of claiming work it did not do.
Implementation references: docs/RESEARCH_COLLECTION_RUNS.md, docs/RESEARCH_WORKLISTS.md, docs/PAGE_HARVEST_PIPELINE.md, and atlas/research_timing.py.
A completed research run adds knowledge and decision history that can support further traversal, comparisons and articles. Story production is a separate workflow over a saved active graph snapshot. It selects accounts, plans sections and writes from existing research, preserving historical claims separately from editorial comparisons.
Stories: workflow and options describes these downstream stages; Purpose and approach explains how software, human direction and model judgment cooperate across the project.
A controller will determine the next unfinished stage from existing run state and validated receipts, carry out routine local steps, and prepare bounded interpretation work for Codex. The first increment is a read-only next-action planner. A later pilot will verify safe interruption, resumption and publication without duplicated work.
A coverage index will record which captured source sections were actually interpreted, under what scope and recipe, and which treatments or drafts resulted. Future collections can reuse those outputs and examine only the remaining gaps. Downloaded pages alone will not count as interpreted, and negative findings will retain their bounded scope.
Both milestones are planned. Their acceptance criteria include completed batches proposing no new work, interrupted batches preserving successful stages, and overlapping collections reusing valid interpretation outputs. They extend the current workflow without requiring an LLM API integration.