docs: update context.md manifest and fix radar SNR/telemetry logic
- Expanded intel/internal/context.md with modern stage-based architecture and internal documentation structure.
- Corrected SNR calculation in model_wrapper.py by dividing out the CFAR threshold.
- Tuned CFAR threshold to 19dB for increased point cloud density.
- Integrated stop flag detection into PipelineManager and ShenronOrchestrator for graceful halting.
- Synchronized metrology gain offsets (-78.0dB) across production and testbench scripts.
- Added 'ti_cascade' support to default radar list and testbench visualization.
│ └── ADC_Data.md ← FMCW and I/Q sampling reference
├── scenarios/ ← Operational manuals for driving simulations
│ ├── braking.md
│ ├── dashboard.md
│ └── showcase.md
├── internal/ ← Context manifest and legacy architecture references
│ ├── context.md ← This file
│ └── old_implement.md
└── CHRONICLES.md ← Running timeline of weekly updates and feature completions
```
```
---
---
## Key Files — Detailed Reference
## Key Files — Detailed Reference
### `dashboard.bat`&`dashboard/` — Web GUI Orchestrator
A Flask-based web dashboard that provides an intuitive interface for running CARLA scenarios without the CLI. It dynamically fetches available scenarios and config params, translates user choices into `run.bat` commands as a background subprocess, and heavily streams the unbuffered Python stdout text back to the browser using Server-Sent Events (SSE).
### `dashboard.bat`&`dashboard/app.py` — Web GUI Orchestrator
A Flask-based web dashboard (`app.py`) that provides an intuitive interface for running CARLA scenarios without the CLI.
- **API Endpoints**: Dynamically fetches available scenarios (`/api/config`), scenario-specific parameters (`/api/scenario_params/<name>`), and manages the CARLA simulator lifecycle (launching, killing, and putting the GPU into "idle" mode).
- **Execution**: Translates user choices into `run.bat` commands spawned as background subprocesses.
- **Streaming**: Streams the unbuffered Python stdout text back to the browser using Server-Sent Events (SSE).
> **Full Architecture details:** See `intel/scenarios/dashboard.md` for a complete breakdown of API routing and extension guidelines.
> **Full Architecture details:** See `intel/scenarios/dashboard.md` for a complete breakdown of API routing and extension guidelines.
---
---
@ -98,38 +120,37 @@ Single source of truth for all simulation-wide defaults. It NO LONGER contains s
The simulation pipeline has been refactored into modular, sequential stages.
- **`src/pipeline/base.py`**: Defines the `PipelineContext` (shared state container) and the `PipelineStage` abstract base class (requires `name`, `run()`, and `cleanup()`).
- **`src/pipeline/manager.py`**: `PipelineManager` orchestrates the sequential execution of stages. It handles `skip_stages` flags, propagates `stop.flag` early halts, and guarantees `cleanup()` is called in reverse order for all started stages if an error occurs.
- `sim_stage.py` (`SimulationStage`): Runs the live CARLA capture loop. Connects to CARLA, handles Ego/NPC spawning, applies weather, runs the `world.tick()` loop, reads sensor queues, and records frames via the `Recorder`. Detects `tmp/stop.flag` for graceful shutdown.
- `shenron_stage.py` (`ShenronStage`): Runs the physics-based radar synthesis over the recorded dataset (calls `generate_shenron.py`). Preserves SSE progress tags (`[SHENRON_INIT]`) for the dashboard.
- `mcap_stage.py` (`McapStage`): Performs Foxglove serialization via `data_to_mcap.py`.
- `video_stage.py` (`VideoStage`): Stitches captured camera frames into MP4 previews.
**Role:** Thin CLI wrapper that initializes the `PipelineManager`.
**Features:**
- Parses arguments and initializes the `PipelineContext`.
- Supports selective execution via flags like `--only-mcap`, `--only-shenron`, `--skip-shenron`, `--skip-mcap`, `--skip-sim`.
- Supports session reuse via `--session <path>` to re-process existing data (e.g., running Shenron or MCAP conversions on old recordings without re-running CARLA).
---
### `src/utils.py` — Utilities
Contains shared project helpers, currently featuring `get_weather_preset(name)`, which safely maps simple string names (e.g., "Clear", "Rain") to the corresponding `carla.WeatherParameters` objects.
@ -229,7 +250,6 @@ NPCs should be spawned with a **0.5m Z-offset (lift)** relative to the road wayp
### Implemented Scenarios
### Implemented Scenarios
| File | Class | Trigger | Effect |
| File | Class | Default Effect | Deterministic? |
| File | Class | Default Effect | Deterministic? |
|---|---|---|---|
|---|---|---|---|
| `braking.py` | `BrakingScenario` | Lead vehicle brakes at frame 80 | Yes (Spawn-and-Move) |
| `braking.py` | `BrakingScenario` | Lead vehicle brakes at frame 80 | Yes (Spawn-and-Move) |
@ -241,38 +261,42 @@ All scenarios now encapsulate their own defaults and support CLI injection via `
---
---
### `scripts/data_to_mcap.py` — MCAP Converter
Scans `data/` for subfolders containing `frames.jsonl` and converts each to a `.mcap` file.
Output is written as `data/<session>/<session>.mcap` (skips if already exists).
### `scripts/` — Production Utilities & Shenron Engine
The `scripts/` folder houses post-processing and conversion utilities, most notably the physics-based radar synthesis engine (Shenron).
#### Shenron Engine (`scripts/ISOLATE/`)
- **`scripts/ISOLATE/shenron_orchestrator.py`**: The unified orchestration engine. It ensures total parity between the production pipeline and iterative lab tests. Manages the directory structures, initializes radar models, processes LiDAR frames, and saves ADC data, pointclouds, and metrology `.npy` files.
- **`scripts/generate_shenron.py`**: The production wrapper invoked by the `ShenronStage`. Feeds the orchestrator with LiDAR data from the session and handles telemetry string reporting (`[SHENRON_INIT]`, `[SHENRON_STEP]`) back to the Flask dashboard.
- **`scripts/ISOLATE/model_wrapper.py`**: Defines `ShenronRadarModel`, providing an object-oriented interface over the underlying physics engine and DSP. It synchronizes hardware specifications (bandwidth, chirps) from `config.yaml` to the global configuration used by the processor.
- **`scripts/ISOLATE/sim_radar_utils/radar_processor.py`**: The core DSP chain. Simulates the TI mmWave hardware accelerators by executing the Range FFT, Doppler FFT, CFAR detection, peak grouping (NMS), and Angle/Azimuth beamforming (3D-FFT) to convert raw ADC cubes into a final 3D point cloud.
- **`scripts/ISOLATE/sim_radar_utils/plots.py`**: Visualization engine. Contains `FastHeatmapEngine` (a stateful matplotlib renderer optimized for high-speed frame-by-frame rendering by reusing figure memory) and other rendering functions for RA/RD heatmaps.
#### Foxglove Serialization
- **`scripts/data_to_mcap.py`**: Converts the raw data folders (`data/<session>/`) into Foxglove-compatible `.mcap` files.
**Foxglove topics produced:**
**Foxglove topics produced by `data_to_mcap.py`:**
@ -328,6 +352,20 @@ class MyScenario(ScenarioBase):
---
---
## Knowledge Base (`intel/`)
The `intel/` directory is the project's brain, storing deep-dive documentation, research logs, and agent protocols. Unlike the codebase sections above, these are markdown files (`.md`).
- **`intel/radar/`**: The core research repository for the Shenron physics engine.
- `ADC_Data.md`: Comprehensive technical reference on raw ADC data, FMCW, I/Q sampling, and signal synthesis.
- `core/`: High-level architecture of the Shenron engine, antenna gain calibration, and implementation guides.
- `diagnostics/`: Crucial debugging logs (e.g., `Shenron_debug.md` is the source of truth for RCS calibration and 3D energy suppression issues).
- `metrology_suite/`: Documentation on the Foxglove integration (RARD, CFAR heatmaps, Auto MCAP).
- `research/`: Mathematical deep-dives into physics problems like isotropic illumination and symmetric RCS reflection.
