Intel update. V1.1

gemini.md

@@ -9,32 +9,49 @@ This document is the consolidated source of truth for AI agents working on the F
 **Objective:** A modular, scenario-driven simulation framework built on CARLA 0.9.16 for demonstrating structured ADAS driving scenarios with multi-modal sensor data.
 
 **The Pipeline:**
-`CARLA Simulator → Multi-Sensor Capture → Dataset (PNG/NPY/JSONL) → MCAP Conversion → Foxglove Visualization`
+`CARLA Simulator → Multi-Sensor Capture → Dataset (PNG/NPY/JSONL) → [SHENRON SYNTHESIS] → MCAP Conversion → Foxglove Visualization`
 
 ### Repository Layout
 - `dashboard.bat`: One-click launcher for the Flask GUI orchestrator.
 - `run.bat`: One-click runner for the simulation (activates `carla312` conda env).
 - `config.py`: Global simulation constants (FPS, sensor specs, ego model).
-- `src/main.py`: Scenario-agnostic orchestrator and entry point.
+- `src/main.py`: Thin CLI wrapper for the `PipelineManager`.
+- `src/pipeline/`: **[NEW]** Core stage-based architecture (Manager + Stages).
+- `src/processing/`: **[NEW]** Centralized physics and sensor augmentation logic.
 - `src/sensors.py`: `SensorManager` — Handles camera, radar, and lidar sync.
 - `src/recorder.py`: `Recorder` — Serializes raw frames to disk asynchronously.
 - `scenarios/`: Contains scenario implementations (e.g., `braking.py`, `cutin.py`, `showcase.py`).
-- `scripts/`: Utility scripts for data processing (e.g., `data_to_mcap.py`).
+- `scripts/`: Utility scripts for data processing (e.g., `data_to_mcap.py`, `generate_shenron.py`).
 - `dashboard/`: Flask backend and static web assets for the GUI.
 - `intel/`: Detailed deep-dive documentation for specific components.
 - `intel/CHRONICLES.md`: **Project History** — Weekly evolution and major milestones.
 - `intel/memory_update.md`: **Agent Protocol** — Standardized guide for repo memory updates.
 - `intel/radar/`: Deep dives into radar physics (e.g. `research/isotropic_illumination_problem.md`).
-- `scripts/`: Utility scripts for data processing and **Metrology Analysis** (`track_full_state.py`, etc).
+- `scripts/ISOLATE/`: High-fidelity radar simulation engine and plotting utils.
 
 ---
 
 ## 🛠️ 2. Component Reference
 
-### `src/main.py` (The Orchestrator)
-- **Role:** Handles CARLA connection, ego spawning, sensor initialization, and main simulation loop.
-- **Rules:** Never import a scenario by name. Uses `scenario_loader.py` to dynamically load classes.
-- **Workflow:** Parse CLI args → Initialize `ScenarioBase` subclass → Tick world once to settle physics → `scenario.setup()` → Loop `world.tick()`, `recorder.save()`, `scenario.step()` → `scenario.cleanup()`.
+### `src/pipeline/manager.py` (The Director)
+- **Role:** Sequential executor for pipeline stages.
+- **Workflow:** Reads `PipelineContext` → Skips stages if requested → Executes `Sim` → `Shenron` → `MCAP` → `Video`.
+- **Error Handling:** Halts the pipeline on critical stage failures but ensures cleanup for all started stages.
+
+### `src/main.py` (CLI Entry Point)
+- **Role:** Parses arguments and initializes the `PipelineManager`.
+- **Features:** Supports selective execution via `--only-mcap` or `--skip-shenron`.
+
+### `src/pipeline/stages/` (The Workers)
+- `SimulationStage`: Handles CARLA connection, ego spawning, and the capture loop.
+- `ShenronStage`: Triggers physics-based radar synthesis via `generate_shenron.py`.
+- `McapStage`: Performs Foxglove serialization via upgraded `data_to_mcap.py`.
+
+### `src/processing/physics.py` (The Math Layer)
+- Centralized source of truth for:
+    - **Radial Velocity Injection**: Projecting relative velocity onto LOS.
+    - **ADAS Metrics**: Real-time Range, Azimuth, and Closing Velocity.
+    - **Actor Categorization**: Classifying CARLA actors for telemetry.
 
 ### `src/sensors.py` (SensorManager)
 - Manages Camera (RGB), Radar, and LiDAR.
@@ -108,4 +125,4 @@ This document is the consolidated source of truth for AI agents working on the F
 
 ---
 
-*Generated by Antigravity AI | Last Updated: 2026-04-15 | Refer to .cursorrules for start-of-turn instructions.*
+*Generated by Antigravity AI | Last Updated: 2026-04-23 | Refer to .cursorrules for start-of-turn instructions.*

intel/CHRONICLES.md

@@ -74,6 +74,13 @@ Prior to Week 3, we used artificial scaling to keep the signal "visible."
 - **Calculation:** $P_{rec} = \frac{P_{tx} \cdot G^2 \cdot \lambda^2 \cdot \sigma}{(4\pi)^3 \cdot R^4}$
 - **Outcome:** The simulation now correctly models how energy drops over distance, allowing for realistic sensitivity-threshold testing.
 
+### 🧩 The Stage-Based Pipeline (Fox v1.1 "Ares")
+As the repository grew, the monolithic `main.py` became a maintenance hazard.
+**The Refactor:** In v1.1, we transitioned to a **Modular Stage Architecture**.
+- **Decision:** Encapsulated Simulation, Radar Synthesis (Shenron), and Data Export (MCAP) into discrete `PipelineStage` classes.
+- **The "Why":** This allows for selective execution (e.g., re-running MCAP on old data) and ensures that failures in one stage (like a Shenron crash) don't corrupt the raw simulation data.
+- **Outcome:** Reduced `main.py` complexity by 85% and enabled robust background resource management.
+
 ### 🖥️ GPU Resource Control (Idle Mode)
 The Shenron engine is heavy. Running it alongside a 30 FPS CARLA simulation causes thermal throttling.
 **The Fix:** We implemented **"Idle Mode."**
@@ -122,7 +129,21 @@ After 30 iterations of parameter tuning, we identified the fundamental reason fo
 
 ## 📜 6. Detailed Daily Chronology (Git Absolute)
 
-### **April 15 (Today)**
+### **April 23 (Milestone: Fox v1.1 "Ares")**
+- **Task**: Stage-Based Pipeline & Radar Parity Sync.
+- **Architecture**: Completed the full refactor to a modular stage-based pipeline. Isolated `SimulationStage`, `ShenronStage`, and `McapStage`.
+- **Logic**: Centralized all radar physics and ADAS metrology in `src/processing/physics.py`.
+- **Robustness**: Migrated all path resolution to `Pathlib` to fix the `stop.flag` detection bug and ensure cross-platform compatibility.
+- **Sync**: Achieved 100% feature parity between the simulation pipeline and `test_shenron.py` diagnostics.
+- **Metrology**: Integrated `FastHeatmapEngine` for stateful rendering, dual RA plots (Static vs. Dynamic), and hardware-specific 3D FOV frustum wireframes.
+- **UX**: Bumped Dashboard version to v1.1.
+
+### **April 22**
+- **Task**: Radar Visualization Modularization.
+- **Action**: Extracted all plotting and scan-conversion logic from `data_to_mcap.py` into a reusable `sim_radar_utils/plots.py` module.
+- **UX**: Implemented `FastHeatmapEngine` to enable high-speed diagnostic rendering without UI blocking.
+
+### **April 15**
 - **Task**: Modular Architecture & Calibration Refinement.
 - **Action**: Transitioned Shenron to a fully modular "Knobs and Dials" architecture.
 - **Modularity**: Isolated radar hardware profiles (AWRL1432, TI-Cascade, Radarbook) in `ConfigureRadar.py`.

intel/radar/diagnostics/Shenron_debug.md

@@ -113,6 +113,8 @@ This log tracks the surgical evolution of the C-Shenron pipeline, from a broken
 | **20** | **Sector Scan-Convert**| 120° Fan-shaped Projection | Corrected rectangular RA distortion; objects now appear at correct geometry. |
 | **21** | **Symmetry Lock** | FFT Index Centering | **TILT FIXED:** Resolved the ~15° FOV shift; fan is perfectly symmetric. |
 | **26** | **The 1/R⁴ Milestone** | Pure Physical Alignment | **FINAL BASELINE:** Stripped legacy 1/1000 norm. Reality-matching power law. |
+| **32** | **FastHeatmap Sync** | Parity with Testbench | **STABILITY:** Migrated to stateful `FastHeatmapEngine` for noise-free dB metrology. |
+| **33** | **Stage Architecture** | Modular Orchestration | **V1.1 "ARES":** Decoupled synthesis into `ShenronStage`. Robust error isolation. |
 
 ---
 

intel/scenarios/dashboard.md

@@ -58,7 +58,15 @@ When the `/api/run` endpoint is hit:
 
 4. **Event Streaming (Yield):**
    Instead of waiting for the process to return an exit code, Flask uses a specific Python generator (`def generate():`) paired with a standard `Response(mimetype="text/event-stream")`. 
-   As `process.stdout.readline()` pulls new text from the unbuffered stream, Flask yields it to the browser wrapped in `data: ... \n\n` syntax. The JS EventSource catches these fragments and appends them to the `<div id="console">` log box.
+   As `process.stdout.readline()` pulls new text from the unbuffered stream, Flask yields it to the browser.
+   
+   **Visual Feedback Tags:**
+   The dashboard JS (`app.js`) specifically listens for bracketed tags to drive UI components:
+   - `[FRAME X/Y]`: Drives the primary simulation progress bar.
+   - `[SHENRON_INIT]`: Sends hardware specs (GPU, VRAM) to the Telemetry HUD.
+   - `[SHENRON_STEP]`: Drives the secondary "Synthesis" progress bar and real-time signal metrics (SNR, Pts).
+   - `[AUTO-MCAP]`: Signals the transition to the final data serialization phase.
+   - `[PROCESS_COMPLETED]`: Triggers the final cleanup and re-enables the "Launch" button.
 
 ---