Simulator/intel/internal/context.md

Project Context — Fox CARLA ADAS Simulation Pipeline

This document is the primary reference for AI agents and developers navigating this codebase. It covers project purpose, architecture, file-by-file roles, data flows, and extension patterns.

---

Project Purpose

A modular, scenario-driven simulation framework built on CARLA 0.9.16.

End-to-end pipeline:

CARLA Simulator → Multi-Sensor Capture → Dataset (PNG / NPY / JSONL)
               → MCAP Conversion → Foxglove Visualization

Primary goal: Demonstrate structured ADAS driving scenarios with synchronized, multi-modal sensor data that can be visualized and analysed in Foxglove Studio.

---

Repository Layout

Fox/
├── dashboard.bat            ← One-click launcher for the GUI orchestrator (Flask)
├── run.bat                  ← One-click launcher (activates carla312 conda env)
├── config.py                ← All tuneable constants (FPS, sensor params, scenario defaults)
│
├── scripts/
│   ├── data_to_mcap.py      ← Converts recorded dataset folders → .mcap files
│   └── data_inspector.py    ← Utility to inspect / debug recorded datasets
│
├── dashboard/               ← Flask backend and web frontend (GUI)
│   ├── app.py               ← Web server bridging API requests to run.bat
│   ├── static/              ← Frontend logic and styling assets
│   └── templates/           ← HTML views
│
├── src/
│   ├── main.py              ← Orchestrator — scenario-agnostic entry point
│   ├── sensors.py           ← SensorManager (camera, radar, lidar setup + sync queues)
│   ├── recorder.py          ← Recorder (writes PNG / NPY / JSONL per frame)
│   ├── scenario_loader.py   ← Dynamic scenario loader via importlib
│   └── utils.py             ← Shared project helpers (weather mapping, etc.)
│
├── scenarios/
│   ├── __init__.py          ← Package marker (empty to avoid side-effect imports)
│   ├── base.py              ← ScenarioBase abstract class (the plugin contract)
│   ├── braking.py           ← Lead vehicle hard braking scenario
│   ├── cutin.py             ← Adjacent lane cut-in scenario
│   └── obstacle.py          ← Static obstacle (traffic cone) scenario
│
├── data/                    ← Auto-created; one subfolder per recording session
│   └── <scenario>_YYYYMMDD_HHMMSS/
│       ├── camera/          ← frame_XXXXXX.png
│       ├── radar/           ← frame_XXXXXX.npy  (shape: [N, 4] — depth/az/alt/vel)
│       ├── lidar/           ← frame_XXXXXX.npy  (shape: [N, 4] — x/y/z/intensity)
│       └── frames.jsonl     ← One JSON record per frame (metadata + scenario info)
│
├── tmp/                     ← Staging area for IPC flags (e.g., stop.flag)
│
└── intel/
    ├── radar/              ← [CRITICAL] Physics, Material RCS, and Debug logs
    │   ├── Shenron_debug.md      ← The "Source of Truth" for radar calibration
    │   └── Sceneset_deepdive.md  ← Reflection & Electromagnetic math
    ├── scenarios/          ← Operational manuals for driving simulations
    │   ├── dashboard.md          ← Web GUI Architecture
    │   └── showcase.md           ← Scenario-specific post-mortems
    └── internal/           ← Project-wide context and developer guides
        └── context.md            ← This file (Primary entry point)

---

Key Files — Detailed Reference

dashboard.bat & dashboard/ — Web GUI Orchestrator

A Flask-based web dashboard that provides an intuitive interface for running CARLA scenarios without the CLI. It dynamically fetches available scenarios and config params, translates user choices into run.bat commands as a background subprocess, and heavily streams the unbuffered Python stdout text back to the browser using Server-Sent Events (SSE).

Full Architecture details: See intel/scenarios/dashboard.md for a complete breakdown of API routing and extension guidelines.

---

config.py

Single source of truth for all simulation-wide defaults. It NO LONGER contains scenario-specific constants.

Key	Default	Purpose
FPS	30	Simulation tick rate
DELTA_SECONDS	0.033	CARLA fixed delta (= 1/FPS)
DEFAULT_EGO_MODEL	"tesla.model3"	Ego vehicle blueprint
DEFAULT_WEATHER	"ClearNoon"	Global weather preset
CAM_WIDTH/HEIGHT/FOV	1280×720, 90°	Camera resolution & field of view
RADAR_RANGE	100 m	Radar max range
LIDAR_CHANNELS	32	LiDAR beam count
MAX_FRAMES	200	Default run length
DEFAULT_SCENARIO	"braking"	Used when no --scenario flag provided

---

src/main.py — Orchestrator

Responsibilities: CARLA connection, ego spawn, sensor init, scenario load, main loop, shutdown. Does NOT contain any scenario-specific logic.

CLI:

python src/main.py --scenario braking --params "BRAKE_FRAME=100"
python src/main.py --scenario cutin --frames 120 --weather Rain
python src/main.py --list-scenarios

New Flags:

• --params: Scenario-specific adjustments (e.g. SPEED=40).
• --weather: Override scenario default (e.g. Sunset, Wet).
• --no-record: Dry-run simulation tracking with no disk I/O.

Execution flow:

1. Parse args → load_scenario(name) → returns ScenarioBase instance
2. Handle Parameter Injection: Call scenario.apply_parameters(args.params)
3. Connect CARLA, settle sync mode, apply chosen weather (CLI > Scenario > Config)
4. Clear existing actors, spawn ego at ego_spawn_point (deterministic index or Transform)
5. SensorManager.spawn_sensors() → attach camera / radar / lidar to ego
6. Recorder(scenario_name=scenario.name) → creates data/<scenario>_<ts>/
7. Settle Physics: Call world.tick() once to synchronize Ego location before setup
8. scenario.setup(world, ego, traffic_manager)
9. Main loop per frame: world.tick() → IPC Stop Check (tmp/stop.flag) → sensor_manager.get_data() → recorder.save(..., extra_meta=scenario.get_scenario_metadata()) → scenario.step(frame, ego, pbar)
10. finally: scenario.cleanup() → sensor_manager.destroy() → restore async mode

Key invariant: main.py never imports a scenario module by name. The CARLA import itself is deferred until after --list-scenarios early-exit so the dry-run works without a running CARLA server.

---

src/sensors.py — SensorManager

Manages three sensors attached to the ego vehicle. All sensors write into queue.Queue objects. get_data() blocks until one item from each queue is available, then asserts frame alignment.

Sensor	CARLA Blueprint	Attachment Point	Output
Camera	sensor.camera.rgb	x=1.5, z=2.4	BGRA image → camera_queue
Radar	sensor.other.radar	x=2.0, z=1.0	Detection list → radar_queue
LiDAR	sensor.lidar.ray_cast	x=0.0, z=2.5	Point buffer → lidar_queue

get_data() asserts cam.frame == radar.frame == lidar.frame — any mismatch raises immediately.

---

src/recorder.py — Recorder

Writes one frame's worth of data to disk each tick.

Output per frame:

• camera/frame_XXXXXX.png — BGR image via OpenCV
• radar/frame_XXXXXX.npy — float64 array [N, 4]: depth, azimuth, altitude, velocity
• lidar/frame_XXXXXX.npy — float32 array [N, 4]: x, y, z, intensity
• One line appended to frames.jsonl:

{
  "frame_id": 82,
  "timestamp": 1234.56,
  "ego_pose": {"x": 12.3, "y": 4.5, "z": 0.0, "yaw": -91.2},
  "ground_truth": [
    {
      "id": 123,
      "class": "vehicle",
      "type": "vehicle.tesla.model3",
      "transform": {"x": 10.5, "y": 2.1, "z": 0.5, "yaw": 90.0, ...},
      "velocity": {"vx": 5.0, "vy": 0.0, "vz": 0.0, "speed": 5.0},
      "acceleration": {"ax": 0.1, "ay": 0.0, "az": 0.0},
      "bounding_box": {"l": 4.5, "w": 2.0, "h": 1.5},
      "relative": {
        "range": 15.2,
        "azimuth": -2.5,
        "closing_velocity": 1.2
      }
    }
  ],
  "scenario": "braking",
  "brake_frame": 80
}

ADAS Relative Metrics:

• range: Euclidean distance (m).
• azimuth: Angle in ego-forward frame (degrees).
• closing_velocity: Rate of approach (m/s). Positive means getting closer.

Scope: Now tracks both vehicle.* and walker.* (pedestrians).

extra_meta pattern: save() accepts extra_meta: dict which is merged into the record. Scenarios use get_scenario_metadata() to supply this — no recorder changes needed per scenario.

---

src/scenario_loader.py — Dynamic Loader

Uses importlib to load scenarios.<name> at runtime. Inspects the module for a concrete ScenarioBase subclass and returns an instance.

from scenario_loader import load_scenario, list_scenarios

scenario = load_scenario("braking")       # → BrakingScenario()
names    = list_scenarios()               # → ['braking', 'cutin', 'obstacle']

list_scenarios() uses pkgutil.iter_modules on the scenarios/ package — auto-discovers new files with no configuration changes.

---

scenarios/base.py — ScenarioBase (Abstract)

The plugin contract all scenarios must fulfil.

Required abstracts:      name (property), setup(), step(), cleanup()
Optional overrides:      ego_spawn_point, weather, max_frames (properties), on_ego_spawned(), get_scenario_metadata()
Shared helpers:          _destroy_actors(), _get_waypoint_ahead(distance, lane_offset), apply_parameters(params)
Protected state:         self._world, self._ego, self._tm, self._actors (list)

**Deterministic Spawning:**
Subclasses should override `ego_spawn_point` (return a `carla.Transform`) to ensure the scenario always starts at a specific intersection or road segment, regardless of the map's default spawn points.

**Z-Axis Safety:**
NPCs should be spawned with a **0.5m Z-offset (lift)** relative to the road waypoint to prevent bounding-box collision with the ground mesh (fixed "Spot occupied" errors).

---

### Implemented Scenarios

| File | Class | Trigger | Effect |
| File | Class | Default Effect | Deterministic? |
|---|---|---|---|
| `braking.py` | `BrakingScenario` | Lead vehicle brakes at frame 80 | Yes (Spawn-and-Move) |
| `cutin.py` | `CutInScenario` | NPC cuts into lane at frame 60 | Yes (Spawn-and-Move) |
| `obstacle.py` | `ObstacleScenario` | Cone placed 30 m ahead | Yes (Spawn-and-Move) |
| `showcase.py` | `ShowcaseScenario` | Complex Left-Turn Across Path demo | Yes (Manual Pathing) |

All scenarios now encapsulate their own defaults and support CLI injection via `--params`.

---

### `scripts/data_to_mcap.py` — MCAP Converter
Scans `data/` for subfolders containing `frames.jsonl` and converts each to a `.mcap` file.
Output is written as `data/<session>/<session>.mcap` (skips if already exists).

**Foxglove topics produced:**

| Topic | Schema | Content |
|---|---|---|
| `/camera` | `foxglove.CompressedImage` | Base64-encoded PNG |
| `/lidar` | `foxglove.PointCloud` | X/Y/Z float32, Y-axis flipped for ROS convention |
| `/radar` | `foxglove.PointCloud` | Spherical → Cartesian conversion, Y-flipped |
| `/ego_pose` | `foxglove.Pose` | Position + quaternion from yaw angle |

> **Coordinate system note:** CARLA uses left-handed coords (Y increases right).
> The converter negates Y and yaw to match ROS/Foxglove right-handed convention.

**Run:**

python scripts/data_to_mcap.py

Processes all unprocessed session folders in `data/` automatically.

---

## Full Pipeline — End-to-End

1. CARLA server running (CarlaUE4.exe)
2. run.bat braking → data/braking_/ (PNG + NPY + JSONL)
3. run.bat cutin → data/cutin_/
4. run.bat obstacle → data/obstacle_/
5. python scripts/data_to_mcap.py → data/*/.mcap
6. Open .mcap in Foxglove Studio

---

## How to Add a New Scenario

1. Create `scenarios/my_scenario.py`
2. Subclass `ScenarioBase`, implement the four required members
3. Append any spawned actors to `self._actors` so `_destroy_actors()` handles cleanup
4. Use `self._get_waypoint_ahead(d)` for all NPC placement
5. Add config constants in `config.py` (optional but recommended)
6. Run `python src/main.py --list-scenarios` — it appears automatically

```python
from scenarios.base import ScenarioBase

class MyScenario(ScenarioBase):

    @property
    def name(self): return "my_scenario"

    def setup(self, world, ego_vehicle, traffic_manager):
        self._world, self._ego, self._tm = world, ego_vehicle, traffic_manager
        wp = self._get_waypoint_ahead(20)
        npc = world.try_spawn_actor(bp, wp.transform)
        if npc:
            self._actors.append(npc)

    def step(self, frame, ego_vehicle):
        if frame == 100:
            pass  # trigger event

    def cleanup(self):
        self._destroy_actors()

    def get_scenario_metadata(self):
        return {"scenario": self.name}

---

Environment & Dependencies

Item	Value
CARLA version	0.9.16
Python environment	conda env carla312 (miniconda)
Activation	run.bat calls activate.bat carla312 automatically
Key Python packages	carla, numpy, opencv-python (cv2), mcap
CARLA server address	localhost:2000 (hardcoded in main.py)
Traffic Manager port	8000 (hardcoded in main.py)

---

Known Limitations & Future Work

Area	Status	Notes
MCAP encoding	JSON (functional)	Should migrate to Protobuf/typed schemas for performance
Intersection scenario	In Progress	See scenarios/showcase.py for LTAP implementation
Foxglove layouts	Manual	Future: .json layout presets per scenario
Multi-ego support	Not implemented	Single ego vehicle assumed throughout
Radar Foxglove schema	Re-uses PointCloud	Correct but non-semantic; dedicated radar schema planned

---

🤖 AI Agent Navigation Guide

When working on this repository, prioritize documentation based on your specific task:

• Radar Physics or Calibration: READ intel/radar/Shenron_debug.md FIRST. This is the source of truth for all FMCW and material RCS milestones.
• Scenario Creation: READ intel/internal/context.md for the plugin contract and intel/scenarios/braking.md for spawning examples.
• Dashboard or GUI Logic: READ intel/scenarios/dashboard.md for SSE and Flask-to-Subprocess architecture.
• Historical Context: Check intel/internal/old_implement.md if the user refers to legacy "Transfuser++" patterns.

---

Last updated: 2026-04-10 | Pipeline version: Scenario-Centric Deterministic Architecture