Checking credentials…
Gated · Pipeline
Training schedules & pipeline health
~635 nightly tasks across morning (06:00 UTC), evening (18:00 UTC), and continuous crons (/10 min matchers, /15 min tensor snapshots, /4h tier-A interferometry). Every task writes to /var/log/deepmap/ and reports status here.
Recurring crons (AX41 + CPX41)
| Schedule (UTC) | Task | Server | Purpose |
|---|---|---|---|
| */10 * * * * | Auto-matchers (22 in parallel) | AX41 | Pull USGS / EMSC / NOAA / GeyserTimes etc., match to predictions |
| */15 * * * * | Earth State Tensor snapshot | AX41 | 9,700-dim state vector + anomaly scoring |
| 5,20,35,50 * * * * | Tensor solar flare predictor | AX41 | Per-snapshot solar flare classification |
| 13 * * * * | Convergence gate predictor (NEW) | AX41 | N ≥ 3 multi-physics channel agreement |
| 5,20,35,50 * * * * | Tensor EQ per-region predictor | AX41 | Per-region nowcast after each snapshot |
| 0 3 * * * | Eigenmode compute | AX41 | PCA over all tensor snapshots |
| 5 3 * * * | Sync AX41 → CPX41 | AX41 | prediction_ledger.db + earth_state_tensor.db |
| 30 3 * * * | Cross-correlation trainer | AX41 | 21×21 VQ pairwise correlation matrix |
| 0 4 * * * | Solar + DM pipeline | AX41 | Annual modulation analysis + permutation nulls |
| 0 6 * * * | Morning pipeline | AX41 | ~115 core + 157 auto-discovered monitors + 224 trainers |
| 0 18 * * * | Evening pipeline | AX41 | ~157 monitors + 224 trainers |
| 01,05,09,13,21 h | Tier-A interferometry (50K pairs each) | AX41 | Europe/Asia/global dv/v, 250K/day from AX41 |
| 03,07,11,15,19 h | Americas interferometry (10K each) | CPX41 | Americas-only dv/v, 50K/day |
| 30 4 * * 0 | Sunday full retrain (2000 samples) | AX41 | tensor_eq + solar classifiers retrain from scratch |
| 35 4 * * 0 | Per-region EQ retrain | AX41 | 56 regions × USGS M3.5+ labels |
| 35 5 * * * | Per-region retry wrapper | AX41 | Daily fallback if weekly hit rate-limits |
| */15 * * * * | CPX41 auto-deploy | CPX41 | git fetch + SIGHUP uvicorn if new commits |
| 3 7 * * * | Overnight health check | CPX41 | service status + DB freshness + integrity_check |
Task counts
115+
Morning core tasks
157+
Auto-discovered monitors
224+
ML trainers
300K
Pairs / day (dv/v)
Live log-tailing view of the latest morning / evening pipeline run is the next build. For now, ssh into AX41 and tail -f /var/log/deepmap/morning.log for the live feed.