# Project Context — Fox CARLA ADAS Simulation Pipeline

> This document is the primary reference for AI agents and developers navigating this codebase.
> It covers project purpose, architecture, file-by-file roles, data flows, and extension patterns.

---

## Project Purpose

A modular, scenario-driven simulation framework built on CARLA 0.9.16.

**End-to-end pipeline:**
```
CARLA Simulator → Multi-Sensor Capture → Dataset (PNG / NPY / JSONL)
               → MCAP Conversion → Foxglove Visualization
```

**Primary goal:** Demonstrate structured ADAS driving scenarios with synchronized,
multi-modal sensor data that can be visualized and analysed in Foxglove Studio.

---

## Repository Layout

```
Fox/
├── dashboard.bat            ← One-click launcher for the GUI orchestrator (Flask)
├── run.bat                  ← One-click launcher (activates carla312 conda env)
├── config.py                ← All tuneable constants (FPS, sensor params, scenario defaults)
│
├── scripts/
│   ├── data_to_mcap.py      ← Converts recorded dataset folders → .mcap files
│   └── data_inspector.py    ← Utility to inspect / debug recorded datasets
│
├── dashboard/               ← Flask backend and web frontend (GUI)
│   ├── app.py               ← Web server bridging API requests to run.bat
│   ├── static/              ← Frontend logic and styling assets
│   └── templates/           ← HTML views
│
├── src/
│   ├── main.py              ← Orchestrator — scenario-agnostic entry point
│   ├── sensors.py           ← SensorManager (camera, radar, lidar setup + sync queues)
│   ├── recorder.py          ← Recorder (writes PNG / NPY / JSONL per frame)
│   ├── scenario_loader.py   ← Dynamic scenario loader via importlib
│   └── utils.py             ← Shared project helpers (weather mapping, etc.)
│
├── scenarios/
│   ├── __init__.py          ← Package marker (empty to avoid side-effect imports)
│   ├── base.py              ← ScenarioBase abstract class (the plugin contract)
│   ├── braking.py           ← Lead vehicle hard braking scenario
│   ├── cutin.py             ← Adjacent lane cut-in scenario
│   └── obstacle.py          ← Static obstacle (traffic cone) scenario
│
├── data/                    ← Auto-created; one subfolder per recording session
│   └── <scenario>_YYYYMMDD_HHMMSS/
│       ├── camera/          ← frame_XXXXXX.png
│       ├── radar/           ← frame_XXXXXX.npy  (shape: [N, 4] — depth/az/alt/vel)
│       ├── lidar/           ← frame_XXXXXX.npy  (shape: [N, 4] — x/y/z/intensity)
│       └── frames.jsonl     ← One JSON record per frame (metadata + scenario info)
│
└── intel/
    ├── radar/              ← [CRITICAL] Physics, Material RCS, and Debug logs
    │   ├── Shenron_debug.md      ← The "Source of Truth" for radar calibration
    │   └── Sceneset_deepdive.md  ← Reflection & Electromagnetic math
    ├── scenarios/          ← Operational manuals for driving simulations
    │   ├── dashboard.md          ← Web GUI Architecture
    │   └── showcase.md           ← Scenario-specific post-mortems
    └── internal/           ← Project-wide context and developer guides
        └── context.md            ← This file (Primary entry point)
```

---

## Key Files — Detailed Reference

### `dashboard.bat` & `dashboard/` — Web GUI Orchestrator
A Flask-based web dashboard that provides an intuitive interface for running CARLA scenarios without the CLI. It dynamically fetches available scenarios and config params, translates user choices into `run.bat` commands as a background subprocess, and heavily streams the unbuffered Python stdout text back to the browser using Server-Sent Events (SSE).
> **Full Architecture details:** See `intel/scenarios/dashboard.md` for a complete breakdown of API routing and extension guidelines.

---

### `config.py`
Single source of truth for all simulation-wide defaults. It NO LONGER contains scenario-specific constants.

| Key | Default | Purpose |
|---|---|---|
| `FPS` | 30 | Simulation tick rate |
| `DELTA_SECONDS` | 0.033 | CARLA fixed delta (= 1/FPS) |
| `DEFAULT_EGO_MODEL` | `"tesla.model3"` | Ego vehicle blueprint |
| `DEFAULT_WEATHER` | `"ClearNoon"` | Global weather preset |
| `CAM_WIDTH/HEIGHT/FOV` | 1280×720, 90° | Camera resolution & field of view |
| `RADAR_RANGE` | 100 m | Radar max range |
| `LIDAR_CHANNELS` | 32 | LiDAR beam count |
| `MAX_FRAMES` | 200 | Default run length |
| `DEFAULT_SCENARIO` | `"braking"` | Used when no `--scenario` flag provided |

---

### `src/main.py` — Orchestrator
**Responsibilities:** CARLA connection, ego spawn, sensor init, scenario load, main loop, shutdown.
**Does NOT contain any scenario-specific logic.**

**CLI:**
```powershell
python src/main.py --scenario braking --params "BRAKE_FRAME=100"
python src/main.py --scenario cutin --frames 120 --weather Rain
python src/main.py --list-scenarios
```

**New Flags:**
- `--params`: Scenario-specific adjustments (e.g. `SPEED=40`).
- `--weather`: Override scenario default (e.g. `Sunset`, `Wet`).
- `--no-record`: Dry-run simulation tracking with no disk I/O.

**Execution flow:**
1. Parse args → `load_scenario(name)` → returns `ScenarioBase` instance
2. **Handle Parameter Injection**: Call `scenario.apply_parameters(args.params)`
3. Connect CARLA, settle sync mode, apply chosen weather (CLI > Scenario > Config)
4. Clear existing actors, spawn ego at **`ego_spawn_point`** (deterministic index or Transform)
5. `SensorManager.spawn_sensors()` → attach camera / radar / lidar to ego
6. `Recorder(scenario_name=scenario.name)` → creates `data/<scenario>_<ts>/`
7. **Settle Physics**: Call `world.tick()` once to synchronize Ego location before setup
8. `scenario.setup(world, ego, traffic_manager)`
9. **Main loop** per frame: `world.tick()` → `sensor_manager.get_data()` →
   `recorder.save(..., extra_meta=scenario.get_scenario_metadata())` → `scenario.step(frame, ego, pbar)`
10. `finally`: `scenario.cleanup()` → `sensor_manager.destroy()` → restore async mode

> **Key invariant:** `main.py` never imports a scenario module by name.
> The CARLA import itself is deferred until after `--list-scenarios` early-exit so the dry-run
> works without a running CARLA server.

---

### `src/sensors.py` — SensorManager
Manages three sensors attached to the ego vehicle. All sensors write into `queue.Queue` objects.
`get_data()` blocks until one item from each queue is available, then asserts frame alignment.

| Sensor | CARLA Blueprint | Attachment Point | Output |
|---|---|---|---|
| Camera | `sensor.camera.rgb` | x=1.5, z=2.4 | BGRA image → `camera_queue` |
| Radar | `sensor.other.radar` | x=2.0, z=1.0 | Detection list → `radar_queue` |
| LiDAR | `sensor.lidar.ray_cast` | x=0.0, z=2.5 | Point buffer → `lidar_queue` |

> `get_data()` asserts `cam.frame == radar.frame == lidar.frame` — any mismatch raises immediately.

---

### `src/recorder.py` — Recorder
Writes one frame's worth of data to disk each tick.

**Output per frame:**
- `camera/frame_XXXXXX.png` — BGR image via OpenCV
- `radar/frame_XXXXXX.npy` — `float64` array `[N, 4]`: depth, azimuth, altitude, velocity
- `lidar/frame_XXXXXX.npy` — `float32` array `[N, 4]`: x, y, z, intensity
- One line appended to `frames.jsonl`:

```json
{
  "frame_id": 82,
  "timestamp": 1234.56,
  "ego_pose": {"x": 12.3, "y": 4.5, "z": 0.0, "yaw": -91.2},
  "ground_truth": [
    {
      "id": 123,
      "class": "vehicle",
      "type": "vehicle.tesla.model3",
      "transform": {"x": 10.5, "y": 2.1, "z": 0.5, "yaw": 90.0, ...},
      "velocity": {"vx": 5.0, "vy": 0.0, "vz": 0.0, "speed": 5.0},
      "acceleration": {"ax": 0.1, "ay": 0.0, "az": 0.0},
      "bounding_box": {"l": 4.5, "w": 2.0, "h": 1.5},
      "relative": {
        "range": 15.2,
        "azimuth": -2.5,
        "closing_velocity": 1.2
      }
    }
  ],
  "scenario": "braking",
  "brake_frame": 80
}
```

**ADAS Relative Metrics**:
- `range`: Euclidean distance (m).
- `azimuth`: Angle in ego-forward frame (degrees).
- `closing_velocity`: Rate of approach (m/s). Positive means getting closer.

**Scope**: Now tracks both `vehicle.*` and `walker.*` (pedestrians).

**`extra_meta` pattern:** `save()` accepts `extra_meta: dict` which is merged into the record.
Scenarios use `get_scenario_metadata()` to supply this — no recorder changes needed per scenario.

---

### `src/scenario_loader.py` — Dynamic Loader
Uses `importlib` to load `scenarios.<name>` at runtime. Inspects the module for a concrete
`ScenarioBase` subclass and returns an instance.

```python
from scenario_loader import load_scenario, list_scenarios

scenario = load_scenario("braking")       # → BrakingScenario()
names    = list_scenarios()               # → ['braking', 'cutin', 'obstacle']
```

`list_scenarios()` uses `pkgutil.iter_modules` on the `scenarios/` package — auto-discovers
new files with no configuration changes.

---

### `scenarios/base.py` — ScenarioBase (Abstract)
The **plugin contract** all scenarios must fulfil.

```
Required abstracts:      name (property), setup(), step(), cleanup()
Optional overrides:      ego_spawn_point, weather, max_frames (properties), on_ego_spawned(), get_scenario_metadata()
Shared helpers:          _destroy_actors(), _get_waypoint_ahead(distance, lane_offset), apply_parameters(params)
Protected state:         self._world, self._ego, self._tm, self._actors (list)

**Deterministic Spawning:**
Subclasses should override `ego_spawn_point` (return a `carla.Transform`) to ensure the scenario always starts at a specific intersection or road segment, regardless of the map's default spawn points.

**Z-Axis Safety:**
NPCs should be spawned with a **0.5m Z-offset (lift)** relative to the road waypoint to prevent bounding-box collision with the ground mesh (fixed "Spot occupied" errors).

---

### Implemented Scenarios

| File | Class | Trigger | Effect |
| File | Class | Default Effect | Deterministic? |
|---|---|---|---|
| `braking.py` | `BrakingScenario` | Lead vehicle brakes at frame 80 | Yes (Spawn-and-Move) |
| `cutin.py` | `CutInScenario` | NPC cuts into lane at frame 60 | Yes (Spawn-and-Move) |
| `obstacle.py` | `ObstacleScenario` | Cone placed 30 m ahead | Yes (Spawn-and-Move) |
| `showcase.py` | `ShowcaseScenario` | Complex Left-Turn Across Path demo | Yes (Manual Pathing) |

All scenarios now encapsulate their own defaults and support CLI injection via `--params`.

---

### `scripts/data_to_mcap.py` — MCAP Converter
Scans `data/` for subfolders containing `frames.jsonl` and converts each to a `.mcap` file.
Output is written as `data/<session>/<session>.mcap` (skips if already exists).

**Foxglove topics produced:**

| Topic | Schema | Content |
|---|---|---|
| `/camera` | `foxglove.CompressedImage` | Base64-encoded PNG |
| `/lidar` | `foxglove.PointCloud` | X/Y/Z float32, Y-axis flipped for ROS convention |
| `/radar` | `foxglove.PointCloud` | Spherical → Cartesian conversion, Y-flipped |
| `/ego_pose` | `foxglove.Pose` | Position + quaternion from yaw angle |

> **Coordinate system note:** CARLA uses left-handed coords (Y increases right).
> The converter negates Y and yaw to match ROS/Foxglove right-handed convention.

**Run:**
```
python scripts/data_to_mcap.py
```
Processes all unprocessed session folders in `data/` automatically.

---

## Full Pipeline — End-to-End

```
1. CARLA server running (CarlaUE4.exe)
2. run.bat braking          → data/braking_<ts>/ (PNG + NPY + JSONL)
3. run.bat cutin            → data/cutin_<ts>/
4. run.bat obstacle         → data/obstacle_<ts>/
5. python scripts/data_to_mcap.py   → data/*/<session>.mcap
6. Open .mcap in Foxglove Studio
```

---

## How to Add a New Scenario

1. Create `scenarios/my_scenario.py`
2. Subclass `ScenarioBase`, implement the four required members
3. Append any spawned actors to `self._actors` so `_destroy_actors()` handles cleanup
4. Use `self._get_waypoint_ahead(d)` for all NPC placement
5. Add config constants in `config.py` (optional but recommended)
6. Run `python src/main.py --list-scenarios` — it appears automatically

```python
from scenarios.base import ScenarioBase

class MyScenario(ScenarioBase):

    @property
    def name(self): return "my_scenario"

    def setup(self, world, ego_vehicle, traffic_manager):
        self._world, self._ego, self._tm = world, ego_vehicle, traffic_manager
        wp = self._get_waypoint_ahead(20)
        npc = world.try_spawn_actor(bp, wp.transform)
        if npc:
            self._actors.append(npc)

    def step(self, frame, ego_vehicle):
        if frame == 100:
            pass  # trigger event

    def cleanup(self):
        self._destroy_actors()

    def get_scenario_metadata(self):
        return {"scenario": self.name}
```

---

## Environment & Dependencies

| Item | Value |
|---|---|
| CARLA version | 0.9.16 |
| Python environment | conda env `carla312` (miniconda) |
| Activation | `run.bat` calls `activate.bat carla312` automatically |
| Key Python packages | `carla`, `numpy`, `opencv-python` (`cv2`), `mcap` |
| CARLA server address | `localhost:2000` (hardcoded in `main.py`) |
| Traffic Manager port | `8000` (hardcoded in `main.py`) |

---

## Known Limitations & Future Work

| Area | Status | Notes |
|---|---|---|
| MCAP encoding | JSON (functional) | Should migrate to Protobuf/typed schemas for performance |
| Intersection scenario | In Progress | See `scenarios/showcase.py` for LTAP implementation |
| Foxglove layouts | Manual | Future: `.json` layout presets per scenario |
| Multi-ego support | Not implemented | Single ego vehicle assumed throughout |
| Radar Foxglove schema | Re-uses PointCloud | Correct but non-semantic; dedicated radar schema planned |

---

## 🤖 AI Agent Navigation Guide

When working on this repository, prioritize documentation based on your specific task:

- **Radar Physics or Calibration:** READ `intel/radar/Shenron_debug.md` FIRST. This is the source of truth for all FMCW and material RCS milestones.
- **Scenario Creation:** READ `intel/internal/context.md` for the plugin contract and `intel/scenarios/braking.md` for spawning examples.
- **Dashboard or GUI Logic:** READ `intel/scenarios/dashboard.md` for SSE and Flask-to-Subprocess architecture.
- **Historical Context:** Check `intel/internal/old_implement.md` if the user refers to legacy "Transfuser++" patterns.

---

*Last updated: 2026-04-06 | Pipeline version: Scenario-Centric Deterministic Architecture*