# Implementation Plan: Stage-Based Pipeline Refactor

Refactor the Fox CARLA ADAS simulation pipeline to separate concerns across environment management, data processing, and serialization. This moves the project from a monolithic `main.py` to a modular, stage-based architecture.

## User Review Required

> [!IMPORTANT]
> **Dashboard Telemetry Persistence**: The Dashboard backend (`app.py`) parses specific `stdout` patterns like `[SHENRON_STEP]` and `[AUTO-MCAP]`. We must ensure the new `PipelineManager` or the individual stages continue to emit these strings to avoid breaking the UI progress bars.

> [!WARNING]
> **Resource Management**: During the transition from `SimulationStage` to `ShenronStage`, we must explicitly ensure CARLA's synchronous mode is either released or the process is handled such that GPU memory is freed for the heavy Shenron Torch operations.

## Proposed Changes

---

### [NEW] Pipeline Core (`src/pipeline/`)

Establish the foundational classes for stage-based execution.

#### [NEW] [base.py](file:///d:/CARLA/CARLA_0.9.16/PythonAPI/Fox/src/pipeline/base.py)
- Define `PipelineContext`: Dataclass to carry `session_path`, `scenario_config`, and `args`.
- Define `PipelineStage`: Abstract Base Class with `run(context)` and `cleanup()` methods.

#### [NEW] [manager.py](file:///d:/CARLA/CARLA_0.9.16/PythonAPI/Fox/src/pipeline/manager.py)
- Implement `PipelineManager`: Orchestrates the execution of a list of `PipelineStage` objects.
- Handles error propagation and graceful halts.

---

### [NEW] Processing Layer (`src/processing/`)

Extract physics and data-augmentation logic to make it reusable and testable.

#### [NEW] [physics.py](file:///d:/CARLA/CARLA_0.9.16/PythonAPI/Fox/src/processing/physics.py)
- **`calculate_radial_velocity`**: Extracted from `recorder.py`. Calculates line-of-sight velocity for LiDAR points.
- **`compute_adas_metrics`**: Extracted from `recorder.py`. Calculates range, azimuth, and closing velocity for NPCs.

---

### [MODIFY] Simulation & Recording

Clean up existing components to focus on their primary responsibilities.

#### [MODIFY] [recorder.py](file:///d:/CARLA/CARLA_0.9.16/PythonAPI/Fox/src/recorder.py)
- Remove physics calculation logic.
- Update `save()` to accept pre-calculated metrics.
- Move `_generate_videos` to a utility function.

#### [MODIFY] [main.py](file:///d:/CARLA/CARLA_0.9.16/PythonAPI/Fox/src/main.py)
- Remove monolithic simulation loop.
- Transform into a thin wrapper that initializes `PipelineManager` with `SimStage`, `ShenronStage`, and `McapStage`.

---

### [NEW] Pipeline Stages (`src/pipeline/stages/`)

Encapsulate logic into discrete execution units.

#### [NEW] [sim_stage.py](file:///d:/CARLA/CARLA_0.9.16/PythonAPI/Fox/src/pipeline/stages/sim_stage.py)
- Port the CARLA simulation loop from `main.py`.
- Manage `SensorManager` and `Recorder` lifecycles.

#### [NEW] [shenron_stage.py](file:///d:/CARLA/CARLA_0.9.16/PythonAPI/Fox/src/pipeline/stages/shenron_stage.py)
- Wrap `scripts/generate_shenron.py` logic.
- Ensure `[SHENRON_STEP]` telemetry is emitted.

#### [NEW] [mcap_stage.py](file:///d:/CARLA/CARLA_0.9.16/PythonAPI/Fox/src/pipeline/stages/mcap_stage.py)
- Wrap `scripts/data_to_mcap.py` logic.
- Register Foxglove schemas and handle final serialization.

---

## Verification Plan

### Automated Tests
- **Physics Parity Test**: Create a script in `scratch/` that runs the old `recorder.py` logic and the new `physics.py` logic on the same frame to ensure bit-identical results for radial velocity and ADAS metrics.
- **Pipeline Dry-Run**: Run the pipeline with `--skip-sim` on an existing data folder to verify that `ShenronStage` and `McapStage` correctly identify and process the files.

### Manual Verification
- **Dashboard Validation**: Launch the Dashboard via `dashboard.bat`, trigger a simulation, and verify that:
    1. The console log shows the new stage transitions.
    2. The progress bars for Shenron and MCAP update correctly.
- **Foxglove Check**: Open the generated `.mcap` in Foxglove and verify that the LiDAR (with radial velocity) and Radar pointclouds are visualized correctly.