Simulator/intel/internal/context.md

Project Context — Fox CARLA ADAS Simulation Pipeline

This document is the primary reference for AI agents and developers navigating this codebase. It covers project purpose, architecture, file-by-file roles, data flows, and extension patterns.

---

Project Purpose

A modular, scenario-driven simulation framework built on CARLA 0.9.16.

End-to-end pipeline:

CARLA Simulator → Multi-Sensor Capture → Dataset (PNG / NPY / JSONL)
               → MCAP Conversion → Foxglove Visualization

Primary goal: Demonstrate structured ADAS driving scenarios with synchronized, multi-modal sensor data that can be visualized and analysed in Foxglove Studio.

---

Repository Layout

Fox/
├── dashboard.bat            ← One-click launcher for the GUI orchestrator (Flask)
├── run.bat                  ← One-click launcher (activates carla312 conda env)
├── config.py                ← All tuneable constants (FPS, sensor params, ego models)
├── gemini.md                ← The primary agent instruction protocol
├── README.md                ← Project setup and replication guide
│
├── src/
│   ├── main.py              ← Thin CLI wrapper for the PipelineManager
│   ├── pipeline/            ← Stage-based orchestration (Sim → Shenron → MCAP → Video)
│   ├── processing/          ← Physics utilities (radial velocity, ADAS metrology)
│   ├── sensors.py           ← SensorManager (camera, radar, lidar sync queues)
│   ├── recorder.py          ← Asynchronous frame recorder (PNG / NPY / JSONL)
│   ├── scenario_loader.py   ← Dynamic scenario loader via importlib
│   └── utils.py             ← Shared project helpers (e.g., weather mapping)
│
├── scripts/
│   ├── ISOLATE/             ← High-fidelity Shenron radar physics engine
│   │   ├── shenron_orchestrator.py ← Unified processing loop (production & testbench)
│   │   ├── model_wrapper.py        ← Hardware spec syncer and physics execution
│   │   └── sim_radar_utils/        ← DSP 3D-FFT chain and visualization heatmaps
│   ├── generate_shenron.py  ← Production wrapper invoked by the pipeline (with SSE telemetry)
│   ├── test_shenron.py      ← Standalone iterative lab for radar testing
│   └── data_to_mcap.py      ← Converts datasets into Foxglove `.mcap` formats
│
├── dashboard/               ← Flask backend and web frontend (GUI)
│   ├── app.py               ← Web server bridging API requests to run.bat
│   ├── static/              ← Frontend logic (app.js with SSE parser)
│   └── templates/           ← HTML views (index.html with ADAS Explorer panel)
│
├── scenarios/
│   ├── __init__.py          
│   ├── base.py              ← ScenarioBase abstract class (the plugin contract)
│   ├── braking.py           ← Lead vehicle hard braking scenario
│   ├── cutin.py             ← Adjacent lane cut-in scenario
│   ├── obstacle.py          ← Static obstacle (traffic cone) scenario
│   └── showcase.py          ← Complex Left-Turn Across Path demo
│
├── data/                    ← Auto-created; one subfolder per recording session
│   └── <scenario>_YYYYMMDD_HHMMSS/
│       ├── camera/          ← frame_XXXXXX.png
│       ├── radar/           ← frame_XXXXXX.npy  (shape: [N, 4] — depth/az/alt/vel)
│       ├── lidar/           ← frame_XXXXXX.npy  (shape: [N, 4] — x/y/z/intensity)
│       └── frames.jsonl     ← One JSON record per frame (metadata + scenario info)
│
├── carla_examples/          ← Standard CARLA PythonAPI examples (not part of core Fox pipeline)
│
├── tmp/                     ← Staging area for IPC flags (e.g., stop.flag)
│
└── intel/                   ← Project Knowledge Base & Developer Guides
    ├── radar/               ← Deep-dives into physics math and RCS diagnostics
    │   ├── core/            ← Calibration and architecture guides
    │   ├── diagnostics/     ← Shenron_debug.md and energy suppression logs
    │   ├── metrology_suite/ ← Heatmap and CFAR integration docs
    │   ├── research/        ← Mathematical deep-dives (e.g. Isotropic Illumination)
    │   └── ADC_Data.md      ← FMCW and I/Q sampling reference
    ├── scenarios/           ← Operational manuals for driving simulations
    │   ├── braking.md
    │   ├── dashboard.md
    │   └── showcase.md
    ├── internal/            ← Context manifest and legacy architecture references
    │   ├── context.md       ← This file
    │   └── old_implement.md
    └── CHRONICLES.md        ← Running timeline of weekly updates and feature completions

---

Key Files — Detailed Reference

dashboard.bat & dashboard/app.py — Web GUI Orchestrator

A Flask-based web dashboard (app.py) that provides an intuitive interface for running CARLA scenarios without the CLI.

• API Endpoints: Dynamically fetches available scenarios (/api/config), scenario-specific parameters (/api/scenario_params/<name>), and manages the CARLA simulator lifecycle (launching, killing, and putting the GPU into "idle" mode).
• Execution: Translates user choices into run.bat commands spawned as background subprocesses.
• Streaming: Streams the unbuffered Python stdout text back to the browser using Server-Sent Events (SSE).

Full Architecture details: See intel/scenarios/dashboard.md for a complete breakdown of API routing and extension guidelines.

---

config.py

Single source of truth for all simulation-wide defaults. It NO LONGER contains scenario-specific constants.

Key	Default	Purpose
FPS	30	Simulation tick rate
DELTA_SECONDS	0.033	CARLA fixed delta (= 1/FPS)
DEFAULT_EGO_MODEL	"tesla.model3"	Ego vehicle blueprint
DEFAULT_WEATHER	"ClearNoon"	Global weather preset
CAM_WIDTH/HEIGHT/FOV	1280×720, 90°	Camera resolution & field of view
RADAR_RANGE	100 m	Radar max range
LIDAR_CHANNELS	32	LiDAR beam count
MAX_FRAMES	200	Default run length
DEFAULT_SCENARIO	"braking"	Used when no --scenario flag provided

---

src/pipeline/ — Stage-Based Architecture

The simulation pipeline has been refactored into modular, sequential stages.

• src/pipeline/base.py: Defines the PipelineContext (shared state container) and the PipelineStage abstract base class (requires name, run(), and cleanup()).
• src/pipeline/manager.py: PipelineManager orchestrates the sequential execution of stages. It handles skip_stages flags, propagates stop.flag early halts, and guarantees cleanup() is called in reverse order for all started stages if an error occurs.
• src/pipeline/stages/: Individual worker implementations.
  • sim_stage.py (SimulationStage): Runs the live CARLA capture loop. Connects to CARLA, handles Ego/NPC spawning, applies weather, runs the world.tick() loop, reads sensor queues, and records frames via the Recorder. Detects tmp/stop.flag for graceful shutdown.
  • shenron_stage.py (ShenronStage): Runs the physics-based radar synthesis over the recorded dataset (calls generate_shenron.py). Preserves SSE progress tags ([SHENRON_INIT]) for the dashboard.
  • mcap_stage.py (McapStage): Performs Foxglove serialization via data_to_mcap.py.
  • video_stage.py (VideoStage): Stitches captured camera frames into MP4 previews.

---

src/main.py — Entry Point

Role: Thin CLI wrapper that initializes the PipelineManager. Features:

• Parses arguments and initializes the PipelineContext.
• Supports selective execution via flags like --only-mcap, --only-shenron, --skip-shenron, --skip-mcap, --skip-sim.
• Supports session reuse via --session <path> to re-process existing data (e.g., running Shenron or MCAP conversions on old recordings without re-running CARLA).

---

src/utils.py — Utilities

Contains shared project helpers, currently featuring get_weather_preset(name), which safely maps simple string names (e.g., "Clear", "Rain") to the corresponding carla.WeatherParameters objects.

---

src/processing/physics.py — Centralized Math & Physics

Standalone utilities for sensor data augmentation and ADAS metrology.

• calculate_radial_velocity(): Injects radial speed onto LiDAR points by projecting relative velocity onto the line-of-sight vector (used heavily by Shenron synthesis).
• calculate_relative_metrics(): Computes ground-truth Range, Azimuth, and Closing Velocity between the Ego and NPCs.
• get_actor_class(): Categorizes CARLA actors into broad ADAS classes (vehicle, vru, pedestrian).

---

src/sensors.py — SensorManager

Manages three sensors attached to the ego vehicle. All sensors write into queue.Queue objects. get_data() blocks until one item from each queue is available, then asserts frame alignment.

Sensor	CARLA Blueprint	Attachment Point	Output
Camera	sensor.camera.rgb	x=1.5, z=2.4	BGRA image → camera_queue
Radar	sensor.other.radar	x=2.0, z=1.0	Detection list → radar_queue
LiDAR	sensor.lidar.ray_cast	x=0.0, z=2.5	Point buffer → lidar_queue

get_data() asserts cam.frame == radar.frame == lidar.frame — any mismatch raises immediately.

---

src/recorder.py — Recorder

Writes one frame's worth of data to disk each tick.

Output per frame:

• camera/frame_XXXXXX.png — BGR image via OpenCV
• radar/frame_XXXXXX.npy — float64 array [N, 4]: depth, azimuth, altitude, velocity
• lidar/frame_XXXXXX.npy — float32 array [N, 4]: x, y, z, intensity
• One line appended to frames.jsonl:

{
  "frame_id": 82,
  "timestamp": 1234.56,
  "ego_pose": {"x": 12.3, "y": 4.5, "z": 0.0, "yaw": -91.2},
  "ground_truth": [
    {
      "id": 123,
      "class": "vehicle",
      "type": "vehicle.tesla.model3",
      "transform": {"x": 10.5, "y": 2.1, "z": 0.5, "yaw": 90.0, ...},
      "velocity": {"vx": 5.0, "vy": 0.0, "vz": 0.0, "speed": 5.0},
      "acceleration": {"ax": 0.1, "ay": 0.0, "az": 0.0},
      "bounding_box": {"l": 4.5, "w": 2.0, "h": 1.5},
      "relative": {
        "range": 15.2,
        "azimuth": -2.5,
        "closing_velocity": 1.2
      }
    }
  ],
  "scenario": "braking",
  "brake_frame": 80
}

ADAS Relative Metrics:

• range: Euclidean distance (m).
• azimuth: Angle in ego-forward frame (degrees).
• closing_velocity: Rate of approach (m/s). Positive means getting closer.

Scope: Now tracks both vehicle.* and walker.* (pedestrians).

extra_meta pattern: save() accepts extra_meta: dict which is merged into the record. Scenarios use get_scenario_metadata() to supply this — no recorder changes needed per scenario.

---

src/scenario_loader.py — Dynamic Loader

Uses importlib to load scenarios.<name> at runtime. Inspects the module for a concrete ScenarioBase subclass and returns an instance.

from scenario_loader import load_scenario, list_scenarios

scenario = load_scenario("braking")       # → BrakingScenario()
names    = list_scenarios()               # → ['braking', 'cutin', 'obstacle']

list_scenarios() uses pkgutil.iter_modules on the scenarios/ package — auto-discovers new files with no configuration changes.

---

scenarios/base.py — ScenarioBase (Abstract)

The plugin contract all scenarios must fulfil.

Required abstracts:      name (property), setup(), step(), cleanup()
Optional overrides:      ego_spawn_point, weather, max_frames (properties), on_ego_spawned(), get_scenario_metadata()
Shared helpers:          _destroy_actors(), _get_waypoint_ahead(distance, lane_offset), apply_parameters(params)
Protected state:         self._world, self._ego, self._tm, self._actors (list)

**Deterministic Spawning:**
Subclasses should override `ego_spawn_point` (return a `carla.Transform`) to ensure the scenario always starts at a specific intersection or road segment, regardless of the map's default spawn points.

**Z-Axis Safety:**
NPCs should be spawned with a **0.5m Z-offset (lift)** relative to the road waypoint to prevent bounding-box collision with the ground mesh (fixed "Spot occupied" errors).

---

### Implemented Scenarios

| File | Class | Default Effect | Deterministic? |
|---|---|---|---|
| `braking.py` | `BrakingScenario` | Lead vehicle brakes at frame 80 | Yes (Spawn-and-Move) |
| `cutin.py` | `CutInScenario` | NPC cuts into lane at frame 60 | Yes (Spawn-and-Move) |
| `obstacle.py` | `ObstacleScenario` | Cone placed 30 m ahead | Yes (Spawn-and-Move) |
| `showcase.py` | `ShowcaseScenario` | Complex Left-Turn Across Path demo | Yes (Manual Pathing) |

All scenarios now encapsulate their own defaults and support CLI injection via `--params`.

---

### `scripts/` — Production Utilities & Shenron Engine
The `scripts/` folder houses post-processing and conversion utilities, most notably the physics-based radar synthesis engine (Shenron).

#### Shenron Engine (`scripts/ISOLATE/`)
- **`scripts/ISOLATE/shenron_orchestrator.py`**: The unified orchestration engine. It ensures total parity between the production pipeline and iterative lab tests. Manages the directory structures, initializes radar models, processes LiDAR frames, and saves ADC data, pointclouds, and metrology `.npy` files.
- **`scripts/generate_shenron.py`**: The production wrapper invoked by the `ShenronStage`. Feeds the orchestrator with LiDAR data from the session and handles telemetry string reporting (`[SHENRON_INIT]`, `[SHENRON_STEP]`) back to the Flask dashboard.
- **`scripts/ISOLATE/model_wrapper.py`**: Defines `ShenronRadarModel`, providing an object-oriented interface over the underlying physics engine and DSP. It synchronizes hardware specifications (bandwidth, chirps) from `config.yaml` to the global configuration used by the processor.
- **`scripts/ISOLATE/sim_radar_utils/radar_processor.py`**: The core DSP chain. Simulates the TI mmWave hardware accelerators by executing the Range FFT, Doppler FFT, CFAR detection, peak grouping (NMS), and Angle/Azimuth beamforming (3D-FFT) to convert raw ADC cubes into a final 3D point cloud.
- **`scripts/ISOLATE/sim_radar_utils/plots.py`**: Visualization engine. Contains `FastHeatmapEngine` (a stateful matplotlib renderer optimized for high-speed frame-by-frame rendering by reusing figure memory) and other rendering functions for RA/RD heatmaps.

#### Foxglove Serialization
- **`scripts/data_to_mcap.py`**: Converts the raw data folders (`data/<session>/`) into Foxglove-compatible `.mcap` files.

**Foxglove topics produced by `data_to_mcap.py`:**

| Topic | Schema | Content |
|---|---|---|
| `/camera` | `foxglove.CompressedImage` | Base64-encoded PNG |
| `/lidar` | `foxglove.PointCloud` | X/Y/Z float32, Y-axis flipped for ROS convention |
| `/radar/*` | `foxglove.PointCloud` | Synthesized Shenron pointclouds |
| `/ego_pose` | `foxglove.Pose` | Position + quaternion from yaw angle |
| `/metrology/*` | `foxglove.Grid` | Heatmaps and CFAR diagnostic matrices |

> **Coordinate system note:** CARLA uses left-handed coords (Y increases right).
> The converter negates Y and yaw to match ROS/Foxglove right-handed convention.

---

## Full Pipeline — End-to-End

1. CARLA server running (CarlaUE4.exe)
2. run.bat braking → [SIMULATION] → data/braking_/ (PNG + NPY + JSONL)

3.                      → [SHENRON]      → synthesizes radar physics → data/braking_<ts>/<radar_type>/
   

4.                      → [MCAP]         → scripts/data_to_mcap.py → data/braking_<ts>/<session>.mcap
   

5.                      → [VIDEO]        → mp4 preview stitcher
   

6. Open .mcap in Foxglove Studio

---

## How to Add a New Scenario

1. Create `scenarios/my_scenario.py`
2. Subclass `ScenarioBase`, implement the four required members
3. Append any spawned actors to `self._actors` so `_destroy_actors()` handles cleanup
4. Use `self._get_waypoint_ahead(d)` for all NPC placement
5. Add config constants in `config.py` (optional but recommended)
6. Run `python src/main.py --list-scenarios` — it appears automatically

```python
from scenarios.base import ScenarioBase

class MyScenario(ScenarioBase):

    @property
    def name(self): return "my_scenario"

    def setup(self, world, ego_vehicle, traffic_manager):
        self._world, self._ego, self._tm = world, ego_vehicle, traffic_manager
        wp = self._get_waypoint_ahead(20)
        npc = world.try_spawn_actor(bp, wp.transform)
        if npc:
            self._actors.append(npc)

    def step(self, frame, ego_vehicle):
        if frame == 100:
            pass  # trigger event

    def cleanup(self):
        self._destroy_actors()

    def get_scenario_metadata(self):
        return {"scenario": self.name}

---

Environment & Dependencies

Item	Value
CARLA version	0.9.16
Python environment	conda env carla312 (miniconda)
Activation	run.bat calls activate.bat carla312 automatically
Key Python packages	carla, numpy, opencv-python (cv2), mcap
CARLA server address	localhost:2000 (hardcoded in main.py)
Traffic Manager port	8000 (hardcoded in main.py)

---

Knowledge Base (intel/)

The intel/ directory is the project's brain, storing deep-dive documentation, research logs, and agent protocols. Unlike the codebase sections above, these are markdown files (.md).

• intel/radar/: The core research repository for the Shenron physics engine.
  • ADC_Data.md: Comprehensive technical reference on raw ADC data, FMCW, I/Q sampling, and signal synthesis.
  • core/: High-level architecture of the Shenron engine, antenna gain calibration, and implementation guides.
  • diagnostics/: Crucial debugging logs (e.g., Shenron_debug.md is the source of truth for RCS calibration and 3D energy suppression issues).
  • metrology_suite/: Documentation on the Foxglove integration (RARD, CFAR heatmaps, Auto MCAP).
  • research/: Mathematical deep-dives into physics problems like isotropic illumination and symmetric RCS reflection.
• intel/scenarios/: Operational manuals for driving simulations (e.g., dashboard architecture, showcase walkthroughs).
• intel/internal/: Project-wide context (like this file) and legacy implementation archives.

---

Known Limitations & Future Work

Area	Status	Notes
MCAP encoding	JSON (functional)	Should migrate to Protobuf/typed schemas for performance
Intersection scenario	In Progress	See scenarios/showcase.py for LTAP implementation
Foxglove layouts	Manual	Future: .json layout presets per scenario
Multi-ego support	Not implemented	Single ego vehicle assumed throughout
Radar Foxglove schema	Re-uses PointCloud	Correct but non-semantic; dedicated radar schema planned

---

🤖 AI Agent Navigation Guide

When working on this repository, prioritize documentation based on your specific task:

• Radar Physics or Calibration: READ intel/radar/diagnostics/Shenron_debug.md FIRST. This is the source of truth for all FMCW and material RCS milestones.
• Scenario Creation: READ intel/internal/context.md for the plugin contract and intel/scenarios/braking.md for spawning examples.
• Dashboard or GUI Logic: READ intel/scenarios/dashboard.md for SSE and Flask-to-Subprocess architecture.
• Historical Context: Check intel/internal/old_implement.md if the user refers to legacy "Transfuser++" patterns.

---

Last updated: 2026-04-10 | Pipeline version: Scenario-Centric Deterministic Architecture