Skip to content

Day-Two Operations

Use this guide after a successful first scope run. The manual cadence below is operational guidance, not Dagster schedule configuration.

Suggested manual cadence

Activity When to consider it
Inspect *_observability freshness and data-quality relations Before trusting prices for analysis or after any failed job
Manual hourly odds ingest for a scope When you want newer prices without enabling schedules
Enable schedules Only after manual jobs and dbt builds are healthy
Validate and recover After failures, schema conflicts, or suspicious gaps
Prune or compact via Scripts When the local warehouse grows large or odds history needs trimming
Full warehouse reset (rm oddsfox.duckdb*) After layout upgrades or unrecoverable corruption

Example manual odds refresh after the first full run:

uv run python scripts/run_scope.py polymarket:wc2026 --step odds
uv run python scripts/run_scope.py polymarket:wc2026 --step dbt

Soccer performance-layout rollout

The soccer current projections and private incremental minute marts require a clean local state. Leave the active warehouse untouched and configure new, versioned DUCKDB_PATH and ODDSFOX_RUNTIME_ROOT locations on the external SSD. Validate the replacement in this order:

  1. Run make soccer-catalog-audit.
  2. Run make soccer-minute-live-smoke.
  3. Run make soccer-minute-backfill.
  4. Run uv run python scripts/run_scope.py polymarket:soccer --step dbt and make soccer-production-health.
  5. Switch Dagster to the new paths only after health is noncritical.

Use make soccer-minute-performance-benchmark for a disposable cold/warm comparison. It reports bounded-fetch high-water marks, due-token audit amplification, dirty-market rebuild counts, and output hashes below the active SSD runtime root. Retire the old state manually only after the replacement is accepted; the repository does not delete or update it in place.

Freshness And Trust

  1. Prefer read-only inspection: uv run python scripts/profile_warehouse.py
  2. Check matching *_data_quality and *_ingestion_run_observability relations.
  3. For live Kalshi analysis, prefer is_actionable_live_market and inspect current_price_status on kalshi_wc2026_stage_markets / kalshi_wc2026_group_winner_markets (not Polymarket hourly marts).

Lock Hygiene

Only one read-write connection should hold the DuckDB file. Stop Dagster and other writers before repairs. Use uv run python scripts/profile_warehouse.py --snapshot-copy when you must inspect while another process is active. See Troubleshooting.

Schedules

Keep schedule flags false in .env until you intentionally enable them. Polygon settlement backfill and audit-release jobs are unscheduled and have no schedule-enable flags.

Advanced Paths