Simulator/gemini.md

Fox CARLA ADAS Simulation Project: Agent Intelligence (gemini.md)

This document is the consolidated source of truth for AI agents working on the Fox CARLA Simulation project. It combines high-level architecture, component-level details, and best practices.

---

🏗️ 1. Project Overview & Architecture

Objective: A modular, scenario-driven simulation framework built on CARLA 0.9.16 for demonstrating structured ADAS driving scenarios with multi-modal sensor data.

The Pipeline: CARLA Simulator → Multi-Sensor Capture → Dataset (PNG/NPY/JSONL) → [SHENRON SYNTHESIS] → MCAP Conversion → Foxglove Visualization

Repository Layout

• dashboard.bat: One-click launcher for the Flask GUI orchestrator.
• run.bat: One-click runner for the simulation (activates carla312 conda env).
• config.py: Global simulation constants (FPS, sensor specs, ego model).
• src/main.py: Thin CLI wrapper for the PipelineManager.
• src/pipeline/: Core stage-based architecture (Manager + Stages).
• src/pipeline/base.py: PipelineContext dataclass and PipelineStage ABC.
• src/pipeline/manager.py: Sequential stage executor with cleanup guarantees.
• src/pipeline/stages/: Individual stage implementations.
• src/processing/: Centralized physics and sensor augmentation logic.
• src/processing/physics.py: Radial velocity injection, ADAS metrics, actor classification.
• src/sensors.py: SensorManager — Handles camera, radar, and lidar sync.
• src/recorder.py: Recorder — Serializes raw frames to disk asynchronously.
• scenarios/: Contains scenario implementations (e.g., braking.py, cutin.py, showcase.py).
• scripts/data_to_mcap.py: Production MCAP converter with full metrology suite.
• scripts/generate_shenron.py: Orchestrates Shenron radar synthesis per session.
• scripts/test_shenron.py: Standalone diagnostic testbench for radar iteration.
• scripts/ISOLATE/: High-fidelity radar simulation engine.
• scripts/ISOLATE/sim_radar_utils/: Visualization and DSP utilities.
• scripts/ISOLATE/shenron_orchestrator.py: Unified Orchestration Engine — Centralized processing loop for physics-based radar synthesis. Ensures parity between generate_shenron.py (Production) and test_shenron.py (Iterative Lab).
• scripts/ISOLATE/sim_radar_utils/plots.py: Single source of truth for all radar heatmap rendering (FastHeatmapEngine, postprocess_ra, scan_convert_ra).
• scripts/ISOLATE/sim_radar_utils/config.yaml: DSP processor and plot limit configuration.
• dashboard/: Flask backend and static web assets for the GUI.
• intel/: Detailed deep-dive documentation for specific components.
• intel/CHRONICLES.md: Project History — Weekly evolution and major milestones.
• intel/memory_update.md: Agent Protocol (v1.1) — Standardized guide for repo memory updates.
• intel/internal/[version]/: Versioned release changelogs and walkthroughs.
• intel/radar/: Deep dives into radar physics (e.g. research/isotropic_illumination_problem.md).

---

🛠️ 2. Component Reference

src/pipeline/manager.py (The Director)

• Role: Sequential executor for pipeline stages.
• Workflow: Reads PipelineContext → Skips stages if requested → Executes Sim → Shenron → MCAP → Video.
• Error Handling: Halts the pipeline on critical stage failures but ensures cleanup() is called for all started stages (reverse order).

src/main.py (CLI Entry Point)

• Role: Parses arguments and initializes the PipelineManager.
• Features: Supports selective execution via --only-mcap, --only-shenron, --skip-shenron, --skip-mcap, --skip-sim.
• Session Reuse: --session <path> allows re-processing existing data without re-running the simulation.

src/pipeline/stages/ (The Workers)

• SimulationStage (sim_stage.py): Handles CARLA connection, ego spawning, sensor capture loop, and stop.flag detection.
• ShenronStage (shenron_stage.py): Triggers physics-based radar synthesis via generate_shenron.py. Non-fatal on error.
• McapStage (mcap_stage.py): Performs Foxglove serialization via data_to_mcap.py.
• VideoStage (video_stage.py): Stitches captured camera frames into .mp4 preview videos (dash + third-person).

scripts/ISOLATE/shenron_orchestrator.py (The Unified Engine)

• Role: Centralized orchestration for physics-based radar synthesis.
• Features:
  • Ensures 100% parity between generate_shenron.py and test_shenron.py.
  • Manages directory structures and multi-modal serialization (ADC, PCD, Metrology).
  • Decouples core physics processing from UI-specific telemetry via a callback pattern.

src/processing/physics.py (The Math Layer)

• Centralized source of truth for:
  • Radial Velocity Injection: Projecting relative velocity onto LOS via calculate_radial_velocity().
  • ADAS Metrics: Real-time Range, Azimuth, and Closing Velocity via calculate_relative_metrics().
  • Actor Categorization: Classifying CARLA actors into vehicle, vru, pedestrian via get_actor_class().

src/sensors.py (SensorManager)

• Manages Camera (RGB), Radar, and LiDAR.
• Uses synchronized queues to ensure frame alignment.
• Coordinate System: Left-handed (CARLA Default).

scenarios/base.py (The Plugin Contract)

• All scenarios must inherit from ScenarioBase.
• Required methods: setup(), step(), cleanup(), and name property.
• Optional hooks: on_ego_spawned(), get_scenario_metadata(), max_frames, weather, ego_spawn_point.
• Support for CLI parameter injection via apply_parameters(args.params).

---

🛰️ 3. Data & Visualization

Data Recording (src/recorder.py)

• Saves images as .png, radar/lidar as .npy, and metadata as .jsonl.
• Relative ADAS Metrics: Calculates range, azimuth, and closing velocity for all actors via src/processing/physics.py.

MCAP Conversion (scripts/data_to_mcap.py)

• Converts raw dataset folders into Foxglove-compatible .mcap files.
• Coordinate Conversion: Negates Y and Yaw to switch from CARLA left-handed to ROS/Foxglove right-handed convention.
• Radar Math: Converts spherical (Depth/Az/Alt) to Cartesian (XYZ) for PointCloud representation.
• Metrology Suite (v1.1):
  • RD Heatmaps: dB-converted via 10*log10(data) - 68.0 system gain offset.
  • RA Dual Plots: Static (absolute bounds) + Dynamic (auto-scaled peak tracking) via FastHeatmapEngine.
  • CFAR Visualization: dB-converted threshold mask.
  • Telemetry: Flattened metrics schema for native Foxglove Plot panel graphing.
  • 3D FOV Frustums: Hardware-accurate LINE_LIST wireframes per radar type.

Visualization Engine (scripts/ISOLATE/sim_radar_utils/plots.py)

• render_heatmap(): Stateless single-shot colormap rendering.
• FastHeatmapEngine: Stateful matplotlib renderer that reuses figure memory for high-speed frame-by-frame rendering.
• postprocess_ra(): Clutter subtraction, log compression, and fixed dynamic range clipping.
• scan_convert_ra(): Polar-to-Cartesian sector projection for RA BEV display.

---

🖥️ 4. GUI Dashboard Architecture

• Backend: Flask (app.py) running on port 5000.
• Version: Orchestrator v1.1.
• Simulator Lifecycle: Dashboard polls /api/simulator/status to detect if CARLA is Offline/Ready. Provides one-click initialization via the UI.
• GPU Resource Control: Incorporates "Idle Mode" (Synchronous Throttling) to drop CARLA GPU usage to ~0% when inactive or while Shenron processing is active.
• Streaming: Uses Server-Sent Events (SSE) to stream stdout to the browser.
• Visual Feedback Tags: [SHENRON_INIT], [SHENRON_STEP], [AUTO-MCAP], [PROCESS_COMPLETED] drive progress bars and telemetry HUD.
• Execution: Spawns run.bat as a subprocess.
• Crucial Rule: Always set PYTHONUNBUFFERED=1 to prevent log delays in the GUI console.

---

🧠 5. Scenario Design & Best Practices

The "Deterministic Pattern"

• Use manual coordinate-based control and set_target_velocity instead of Traffic Manager for repeatable maneuvers.
• Spawn-and-Move: Spawn ego at a safe point, then set_transform to the exact start location.
• Z-Axis Lift: Always spawn NPCs with a +0.5m offset to avoid ground-mesh collision errors ("Spot occupied").

Common Pitfalls

• Stale Physics: Always world.tick() once after spawning the Ego but before starting scenario logic.
• Graceful Shutdown: Use the "Stop Simulation" button in the dashboard. It writes a tmp/stop.flag which the orchestrator checks to ensure all recorders and data converters (Shenron/MCAP) finalize correctly.
• Progress Bar Clash: Use pbar.write() for logs if a tqdm progress bar is active.
• Asynchronous I/O: Use ThreadPoolExecutor for disk writes (images) to prevent simulation frame drops.

---

🚀 6. Execution Commands

• Run Scenario: cmd /c run.bat <scenario_name> --frames <N> --params "KEY=VAL" --weather <Preset>
• List Scenarios: cmd /c run.bat --list-scenarios
• Run Testbench: cmd /c "call C:\ProgramData\miniconda3\Scripts\activate.bat carla312 && python scripts/test_shenron.py --iter <Name>"
• Re-process MCAP Only: cmd /c run.bat --only-mcap --session data/<session_folder>

🤖 7. Agent Execution Rules

• Shell Environment: This repository is hosted on Windows. All commands MUST use PowerShell syntax for general file operations and directory management.
• Python Execution: All simulation and scenario commands MUST be launched via the run.bat orchestrator using the cmd /c prefix.
• Standalone Scripts: Scripts like test_shenron.py or diagnostic tools must also use cmd /c with explicit environment activation to ensure compatibility:
  • Correct: cmd /c "call C:\ProgramData\miniconda3\Scripts\activate.bat carla312 && python scripts/test_shenron.py --iter test"
  • Incorrect: python scripts/test_shenron.py --iter test
• Cmdlet Preference: Prioritize native PowerShell cmdlets or robust Windows utilities:
  • Use Select-String or ripgrep instead of grep.
  • Use ; to chain commands instead of &&.
  • Use move/Copy-Item instead of mv/cp.
  • Use tree /f for recursive directory visualization.

---

Generated by Antigravity AI | Last Updated: 2026-04-23 | Refer to .cursorrules for start-of-turn instructions.