# Fox CARLA ADAS Simulation Project: Agent Intelligence (gemini.md)

This document is the consolidated source of truth for AI agents working on the Fox CARLA Simulation project. It combines high-level architecture, component-level details, and best practices.

---

## 🏗️ 1. Project Overview & Architecture

**Objective:** A modular, scenario-driven simulation framework built on CARLA 0.9.16 for demonstrating structured ADAS driving scenarios with multi-modal sensor data.

**The Pipeline:**
`CARLA Simulator → Multi-Sensor Capture → Dataset (PNG/NPY/JSONL) → [SHENRON SYNTHESIS] → MCAP Conversion → Foxglove Visualization`

### Repository Layout
- `dashboard.bat`: One-click launcher for the Flask GUI orchestrator.
- `run.bat`: One-click runner for the simulation (activates `carla312` conda env).
- `config.py`: Global simulation constants (FPS, sensor specs, ego model).
- `src/main.py`: Thin CLI wrapper for the `PipelineManager`.
- `src/pipeline/`: Core stage-based architecture (Manager + Stages).
- `src/pipeline/base.py`: `PipelineContext` dataclass and `PipelineStage` ABC.
- `src/pipeline/manager.py`: Sequential stage executor with cleanup guarantees.
- `src/pipeline/stages/`: Individual stage implementations.
- `src/processing/`: Centralized physics and sensor augmentation logic.
- `src/processing/physics.py`: Radial velocity injection, ADAS metrics, actor classification.
- `src/sensors.py`: `SensorManager` — Handles camera, radar, and lidar sync.
- `src/recorder.py`: `Recorder` — Serializes raw frames to disk asynchronously.
- `scenarios/`: Contains scenario implementations (e.g., `braking.py`, `cutin.py`, `showcase.py`).
- `scripts/data_to_mcap.py`: Production MCAP converter with full metrology suite.
- `scripts/generate_shenron.py`: Orchestrates Shenron radar synthesis per session.
- `scripts/test_shenron.py`: Standalone diagnostic testbench for radar iteration.
- `scripts/ISOLATE/`: High-fidelity radar simulation engine.
- `scripts/ISOLATE/sim_radar_utils/`: Visualization and DSP utilities.
- `scripts/ISOLATE/shenron_orchestrator.py`: **Unified Orchestration Engine** — Centralized processing loop for physics-based radar synthesis. Ensures parity between `generate_shenron.py` (Production) and `test_shenron.py` (Iterative Lab).
- `scripts/ISOLATE/sim_radar_utils/plots.py`: **Single source of truth** for all radar heatmap rendering (`FastHeatmapEngine`, `postprocess_ra`, `scan_convert_ra`).
- `scripts/ISOLATE/sim_radar_utils/config.yaml`: DSP processor and plot limit configuration.
- `dashboard/`: Flask backend and static web assets for the GUI.
- `intel/`: Detailed deep-dive documentation for specific components.
- `intel/CHRONICLES.md`: **Project History** — Weekly evolution and major milestones.
- `intel/memory_update.md`: **Agent Protocol (v1.1)** — Standardized guide for repo memory updates.
- `intel/internal/[version]/`: Versioned release changelogs and walkthroughs.
- `intel/radar/`: Deep dives into radar physics (e.g. `research/isotropic_illumination_problem.md`).

---

## 🛠️ 2. Component Reference

### `src/pipeline/manager.py` (The Director)
- **Role:** Sequential executor for pipeline stages.
- **Workflow:** Reads `PipelineContext` → Skips stages if requested → Executes `Sim` → `Shenron` → `MCAP` → `Video`.
- **Error Handling:** Halts the pipeline on critical stage failures but ensures `cleanup()` is called for all started stages (reverse order).

### `src/main.py` (CLI Entry Point)
- **Role:** Parses arguments and initializes the `PipelineManager`.
- **Features:** Supports selective execution via `--only-mcap`, `--only-shenron`, `--skip-shenron`, `--skip-mcap`, `--skip-sim`.
- **Session Reuse:** `--session <path>` allows re-processing existing data without re-running the simulation.

### `src/pipeline/stages/` (The Workers)
- `SimulationStage` (`sim_stage.py`): Handles CARLA connection, ego spawning, sensor capture loop, and `stop.flag` detection.
- `ShenronStage` (`shenron_stage.py`): Triggers physics-based radar synthesis via `generate_shenron.py`. Non-fatal on error.
- `McapStage` (`mcap_stage.py`): Performs Foxglove serialization via `data_to_mcap.py`.
- `VideoStage` (`video_stage.py`): Stitches captured camera frames into `.mp4` preview videos (dash + third-person).

### `scripts/ISOLATE/shenron_orchestrator.py` (The Unified Engine)
- **Role**: Centralized orchestration for physics-based radar synthesis.
- **Features**: 
    - Ensures 100% parity between `generate_shenron.py` and `test_shenron.py`.
    - Manages directory structures and multi-modal serialization (ADC, PCD, Metrology).
    - Decouples core physics processing from UI-specific telemetry via a callback pattern.

### `src/processing/physics.py` (The Math Layer)
- Centralized source of truth for:
    - **Radial Velocity Injection**: Projecting relative velocity onto LOS via `calculate_radial_velocity()`.
    - **ADAS Metrics**: Real-time Range, Azimuth, and Closing Velocity via `calculate_relative_metrics()`.
    - **Actor Categorization**: Classifying CARLA actors into `vehicle`, `vru`, `pedestrian` via `get_actor_class()`.

### `src/sensors.py` (SensorManager)
- Manages Camera (RGB), Radar, and LiDAR.
- Uses synchronized queues to ensure frame alignment.
- Coordinate System: Left-handed (CARLA Default).

### `scenarios/base.py` (The Plugin Contract)
- All scenarios must inherit from `ScenarioBase`.
- Required methods: `setup()`, `step()`, `cleanup()`, and `name` property.
- Optional hooks: `on_ego_spawned()`, `get_scenario_metadata()`, `max_frames`, `weather`, `ego_spawn_point`.
- Support for CLI parameter injection via `apply_parameters(args.params)`.

---

## 🛰️ 3. Data & Visualization

### Data Recording (`src/recorder.py`)
- Saves images as `.png`, radar/lidar as `.npy`, and metadata as `.jsonl`.
- **Relative ADAS Metrics:** Calculates range, azimuth, and closing velocity for all actors via `src/processing/physics.py`.

### MCAP Conversion (`scripts/data_to_mcap.py`)
- Converts raw dataset folders into Foxglove-compatible `.mcap` files.
- **Coordinate Conversion:** Negates Y and Yaw to switch from CARLA left-handed to ROS/Foxglove right-handed convention.
- **Radar Math:** Converts spherical (Depth/Az/Alt) to Cartesian (XYZ) for PointCloud representation.
- **Metrology Suite (v1.1):**
    - **RD Heatmaps:** dB-converted via `10*log10(data) - 68.0` system gain offset.
    - **RA Dual Plots:** Static (absolute bounds) + Dynamic (auto-scaled peak tracking) via `FastHeatmapEngine`.
    - **CFAR Visualization:** dB-converted threshold mask.
    - **Telemetry:** Flattened metrics schema for native Foxglove Plot panel graphing.
    - **3D FOV Frustums:** Hardware-accurate LINE_LIST wireframes per radar type.

### Visualization Engine (`scripts/ISOLATE/sim_radar_utils/plots.py`)
- `render_heatmap()`: Stateless single-shot colormap rendering.
- `FastHeatmapEngine`: Stateful matplotlib renderer that reuses figure memory for high-speed frame-by-frame rendering.
- `postprocess_ra()`: Clutter subtraction, log compression, and fixed dynamic range clipping.
- `scan_convert_ra()`: Polar-to-Cartesian sector projection for RA BEV display.

---

## 🖥️ 4. GUI Dashboard Architecture

- **Backend:** Flask (`app.py`) running on port 5000.
- **Version:** Orchestrator v1.1.
- **Simulator Lifecycle:** Dashboard polls `/api/simulator/status` to detect if CARLA is Offline/Ready. Provides one-click initialization via the UI.
- **GPU Resource Control:** Incorporates "Idle Mode" (Synchronous Throttling) to drop CARLA GPU usage to ~0% when inactive or while Shenron processing is active.
- **Streaming:** Uses **Server-Sent Events (SSE)** to stream `stdout` to the browser.
- **Visual Feedback Tags:** `[SHENRON_INIT]`, `[SHENRON_STEP]`, `[AUTO-MCAP]`, `[PROCESS_COMPLETED]` drive progress bars and telemetry HUD.
- **Execution:** Spawns `run.bat` as a subprocess.
- **Crucial Rule:** Always set `PYTHONUNBUFFERED=1` to prevent log delays in the GUI console.

---

## 🧠 5. Scenario Design & Best Practices

### The "Deterministic Pattern"
- Use manual coordinate-based control and `set_target_velocity` instead of Traffic Manager for repeatable maneuvers.
- **Spawn-and-Move:** Spawn ego at a safe point, then `set_transform` to the exact start location.
- **Z-Axis Lift:** Always spawn NPCs with a **+0.5m offset** to avoid ground-mesh collision errors ("Spot occupied").

### Common Pitfalls
- **Stale Physics:** Always `world.tick()` once after spawning the Ego but before starting scenario logic.
- **Graceful Shutdown:** Use the "Stop Simulation" button in the dashboard. It writes a `tmp/stop.flag` which the orchestrator checks to ensure all recorders and data converters (Shenron/MCAP) finalize correctly.
- **Progress Bar Clash:** Use `pbar.write()` for logs if a `tqdm` progress bar is active.
- **Asynchronous I/O:** Use `ThreadPoolExecutor` for disk writes (images) to prevent simulation frame drops.

---

## 🚀 6. Execution Commands

- **Run Scenario:** `cmd /c run.bat <scenario_name> --frames <N> --params "KEY=VAL" --weather <Preset>`
- **List Scenarios:** `cmd /c run.bat --list-scenarios`
- **Run Testbench:** `cmd /c "call C:\ProgramData\miniconda3\Scripts\activate.bat carla312 && python scripts/test_shenron.py --iter <Name>"`
- **Re-process MCAP Only:** `cmd /c run.bat --only-mcap --session data/<session_folder>`

## 🤖 7. Agent Execution Rules

- **Shell Environment:** This repository is hosted on **Windows**. All commands MUST use **PowerShell** syntax for general file operations and directory management.
- **Python Execution:** All simulation and scenario commands MUST be launched via the `run.bat` orchestrator using the `cmd /c` prefix. 
- **Standalone Scripts:** Scripts like `test_shenron.py` or diagnostic tools must also use `cmd /c` with explicit environment activation to ensure compatibility:
    - **Correct:** `cmd /c "call C:\ProgramData\miniconda3\Scripts\activate.bat carla312 && python scripts/test_shenron.py --iter test"`
    - **Incorrect:** `python scripts/test_shenron.py --iter test`
- **Cmdlet Preference:** Prioritize native PowerShell cmdlets or robust Windows utilities:
    - Use `Select-String` or `ripgrep` instead of `grep`.
    - Use `;` to chain commands instead of `&&`.
    - Use `move`/`Copy-Item` instead of `mv`/`cp`.
    - Use `tree /f` for recursive directory visualization.

---

*Generated by Antigravity AI | Last Updated: 2026-04-23 | Refer to .cursorrules for start-of-turn instructions.*