**Date:** 2026-04-03 — 2026-04-06 (Calibration Update)  
**Engineer:** Fox ADAS Pipeline | Antigravity AI  
**Objective:** Resolve physics-based discrepancies in the C-Shenron synthetic radar pipeline, calibrate physical materials for turn-stability, and align sensor coordinate frames for zero-parallax synchronization.

---

## 📐 Architecture Overview

The C-Shenron pipeline is a **physics-based FMCW radar simulator** isolated in `scripts/ISOLATE/`. It takes CARLA Semantic LiDAR as input and produces a synthetic radar point cloud as output.

```
CARLA (Semantic LiDAR) 
  → lidar.py (Ingestion + Material Mapping)
  → Sceneset.py (Fresnel RCS Physics)
  → heatmap_gen_fast.py (GPU FMCW ADC Synthesis)
  → radar_processor.py (Range/Doppler FFT + CFAR)
  → [x, y, z, velocity, magnitude] PointCloud
```

**Key Coordinate Convention (LOCKED):**
- CARLA Frame: `X = Forward`, `Y = Right (LHS)`, `Z = Up`
- Shenron Internal Frame: **`Index 1 = Forward`**, `Index 0 = Side`
- The swap `points[:, 0] = CARLA_Y, points[:, 1] = CARLA_X` in `lidar.py` is **intentional and must not be removed.**
- MCAP/Foxglove Output: Negate Y to convert from CARLA LHS → ROS RHS.

---

## 🐛 Bugs Fixed

### Bug #1: The "Cos(Cos)" Reflection Error
**File:** `scripts/ISOLATE/e2e_agent_sem_lidar2shenron_package/shenron/Sceneset.py`  
**Root Cause:** The `specularpoints()` method received pre-computed cosine values from CARLA's Semantic LiDAR (`cos_inc_angle`), but then ran `np.cos()` on them again inside `get_loss_3()`. This caused a "double-cosine" effect.  
**Effect:** A perfect perpendicular bounce (true `cos = 1.0`) was calculated as `cos(1.0) = 0.54`, implying a 57-degree glancing angle. This caused a massive ~46% RCS energy drop on every flat surface (car hoods, road, walls).  
**Fix:** Removed the erroneous `np.cos()` call. The engine now directly uses the pre-computed cosine values as intended. Also corrected the specular reflection mask threshold for 2-degree incident angles.

---

### Bug #2: Zeroed-Out Thermal Noise Floor
**File:** `scripts/ISOLATE/e2e_agent_sem_lidar2shenron_package/ConfigureRadar.py`  
**Root Cause:** The noise generator was multiplied by zero: `signal_Noisy = 0 * (...)`.  
**Effect:** With a zero noise floor, the CFAR detector became violently unstable. It detected infinite false peaks in empty bins because the background noise variance was literally zero, producing random "sparkle" detections with no correlation to actual targets.  
**Fix:** Restored the AWGN noise generator with a proper complex noise implementation using `noise_amp * (np.random.randn(...) + 1j * np.random.randn(...))`.

---

### Bug #3: LiDAR Metadata Bit-View Error
**File:** `src/recorder.py`  
**Root Cause:** CARLA Semantic LiDAR stores `object_idx` and `semantic_tag` as `uint32` values packed into `float32` bit fields. The recorder was reading them as plain floats, producing garbage values (e.g., Object ID of `0.0` for every actor).  
**Effect:** The velocity projection calculation `v_radial = dot(actor_vel - ego_vel, direction)` always failed the actor lookup, resulting in zero radial velocity stored for every point.  
**Fix:** Applied `np.view(np.uint32)` to correctly unpack the integer IDs from the float32 bitstream. The same fix was propagated to the MCAP packaging scripts (`test_shenron.py`, `data_to_mcap.py`).

---

### Bug #4: Stale `RadarProcessor` Axes (Range Scaling Mismatch)
**File:** `scripts/ISOLATE/model_wrapper.py`  
**Root Cause:** The `RadarProcessor` only computed its range axis once at startup, using the default `config.yaml` values (24GHz, 250MHz). When processing the `awrl1432` profile (77GHz, 400MHz), the processor still used the old stale axes to convert FFT bins to meters.  
**Effect:** A ~2x range "stretching" — objects at 50m appeared at ~100m. The bandwidth mismatch caused incorrect scaling.  
**Fix:** Added `self.processor.__init__()` call inside `_sync_configs()` to force the processor to rebuild its internal range and velocity axes every time the hardware profile changes.

---

### Bug #5: Semantic Tag Mismatch (CARLA 0.9.16)
**File:** `scripts/ISOLATE/e2e_agent_sem_lidar2shenron_package/lidar.py`  
**Root Cause:** The semantic lookup table was using legacy CARLA 0.9.5 tags. In CARLA 0.9.16, NPC vehicles moved to **Tag 14**, and Terrain moved to **Tag 10**. 
**Effect:** All NPC cars were being deleted from the point cloud (filtered out as "unknown"), while the ground terrain was being incorrectly simulated as a high-RCS "Metal" object, creating massive false alarms.
**Fix:** Updated the `map_carla_semantic_lidar_latest` mapping to align with modern 0.9.16 standards.

---

## 📡 Hardware Profile: AWRL1432 (Iteration 07 — Current Best)

The `awrl1432` profile in `ConfigureRadar.py` has been tuned to match the real-world **TI AWRL1432BOOST** professional ADAS configuration:

| Parameter | Previous (Iter 05) | Current (Iter 07) | Notes |
| :--- | :--- | :--- | :--- |
| **Frequency** | 77 GHz | 77 GHz | Same |
| **Bandwidth (B)** | 137.2 MHz | **400 MHz** | 3x wider |
| **Range Resolution** | ~109 cm | **37.5 cm** | 3x sharper |
| **Max Range** | ~279 m | **~96 m** | Calibrated to ADAS zone |
| **Chirps (Np)** | 128 | **64** | 2x faster, maintained SNR |
| **Samples (N)** | 256 | 256 | Unchanged |
| **Antennas (nRx)** | 6 | 6 | Unchanged |

**Signal Processor Config** (`sim_radar_utils/config.yaml`):
- `fStrt: 77.0e9`, `fStop: 77.4e9` (matching 400MHz bandwidth)
- `NFFT: 256` (matching N_sample)

---

## 🚀 The Iterative Journey (Milestone History)

This log tracks the surgical evolution of the C-Shenron pipeline, from a broken legacy port to a high-fidelity ADAS testbench.

| Iteration | Challenge | Core Change | Key Result |
| :--- | :--- | :--- | :--- |
| **01** | **System Crash** | Native Port from Transfuser++ | Pipeline initialized but outputs only "black" or random point clouds. |
| **02** | **"Sparkle" Noise** | Restored Thermal Noise Floor | **Bug #2 Fixed:** CFAR detector stabilized; random ghost points removed. |
| **03** | **Low RCS Fidelity** | Fixed "Cos(Cos)" Reflection Math | **Bug #1 Fixed:** Proper Fresnel gain restored (46% energy recovery). |
| **04** | **Static Velocity** | Resolved Metadata Bit-View | **Bug #3 Fixed:** Dynamic velocities (V_radial) finally visible on NPCs. |
| **05** | **Range Stretching** | Synchronized Processor Axes | **Bug #4 Fixed:** Objects at 50m finally appear at 50m (Corrected 77GHz scaling). |
| **06** | **Latency** | Reduced Chirps (Np = 32) | Achieved ~2 FPS simulation speed for rapid prototyping. |
| **07** | **Resolution Loss** | Balanced Chirps (Np = 64) | **AWRL1432 Baseline:** 37.5cm range res with stable SNR. |
| **08** | **Incomplete Tags** | Initial 0.9.16 Tag Exploration | Identified critical mismatch in CARLA 0.9.16 semantic lookup bits. |
| **09** | **Target Vanishing** | Corrected 0.9.16 Mapping | **Bug #5 Fixed:** Target vehicles (Tag 14) are now fully detectable. |
| **10** | **Turning Drop-off** | Metal Roughness (0.0001 $\to$ 0.02) | Vehicles act as "targets" not mirrors. Stable tracks during sharp turns. |
| **11** | **Parallax Shift** | LiDAR Forward Shift (-2.0m) | Radar points spatially locked to bumper. **Lag eliminated.** |
| **11.b**| **"Bumper Clutter"**| Blackman-Harris Windowing | -92dB rejection kills ghost trails bleeding from high-SNR targets. |
| **12** | **Ground Clutter** | Aggressive Z-Filter (Z > -1.5m) | Flat road reflections removed. Identified "Lower-Half Blindness." |
| **13** | **Golden Mix** | Optimized Z-Filter (Z > -2.2m) | **30cm Clearance:** Maintains car tires while stopping road clutter. |
| **17** | **Auto-Metrology** | Persistent RD/RA Heatmaps | Integrated diagnostic maps into automated simulation-to-MCAP pipeline. |
| **18** | **Phase Recovery** | Coherent Doppler-Slice | **BREAKTHROUGH:** Preserved complex phase for Angle-FFT (Fixed 'Blurring'). |
| **19** | **Blue Aesthetic** | Viridis + Global Norm | Achieving professional "Radar Blue" style for diagnostic visualization. |
| **20** | **Sector Scan-Convert**| 120° Fan-shaped Projection | Corrected rectangular RA distortion; objects now appear at correct geometry. |
| **21** | **Symmetry Lock** | FFT Index Centering | **TILT FIXED:** Resolved the ~15° FOV shift; fan is perfectly symmetric. |
| **26** | **The 1/R⁴ Milestone** | Pure Physical Alignment | **FINAL BASELINE:** Stripped legacy 1/1000 norm. Reality-matching power law. |
| **32** | **FastHeatmap Sync** | Parity with Testbench | **STABILITY:** Migrated to stateful `FastHeatmapEngine` for noise-free dB metrology. |
| **33** | **Stage Architecture** | Modular Orchestration | **V1.1 "ARES":** Decoupled synthesis into `ShenronStage`. Robust error isolation. |

---

## 🚀 Physical Calibration History (Session 04-06)

Following the baseline stabilization on April 3rd, the focus shifted to high-fidelity physical realism and sensor synchronization.

| Iteration | Challenge | Core Physics Change | Critical Result |
| :--- | :--- | :--- | :--- |
| **09** | **Target Vanishing** | Corrected 0.9.16 Mapping | NPC Cars (Tag 14) now appear; Road Terrain noise suppressed. |
| **10** | **Specular Deflection**| Metal Roughness (0.0001 $\to$ 0.02) | Vehicles act as "targets" not "mirrors." Visible during sharp turns. |
| **11** | **Parallax Error** | LiDAR-to-Radar Shift (-2.0m) | **ZERO-LAG:** Radar points spatially locked to the front bumper. |
| **11.b**| **Sidelobe Leakage** | Blackman-Harris Windowing | -92dB rejection kills "ghost trails" pointing back at the ego. |
| **12** | **Ground Clutter** | Aggressive Z-Filter (Z > -1.5m) | Total removal of road noise. Blinds lower half of vehicles. |
| **13** | **Golden Mix** | Optimized Z-Filter (Z > -2.2m) | **30cm Clearance:** Maintains car tires/bottom-lip while stopping ground clutter. |
| **14.a**| **Vertical Clumping** | Gaussian Elevation Damping | **20° Beamwidth:** Suppresses tree-top clutter by >90% while preserving boresight targets. |
| **14.b**| **Resolution Trap** | Global $1/N$ Normalization | Attempted to fix scaling; caused context-dependency (buildings dimming cars). |
| **15** | **Range Blindness** | Physical Sensitivity Floor (0.0005) | **DEPRECATED:** Selective bandage that removed distant targets in high-noise frames. |
| **16** | **Refined Physics** | Area-Density Integration | **BREAKTHROUGH:** Replaced dynamic $1/N$ with fixed Density Ref. **+234% Magnitude Recovery.** |
| **24** | **RA Post-Chain** | Log-Scale + Clutter Sub | Professional metrology processing; removes noise-rings and compresses range. |
| **25** | **Mag Calibration** | 1/R^2 Receiver Attenuation | ADC Voltage now correctly follows R^-2; total trip follows R^-4 power law. |
| **26** | **Pure 1/R⁴ Model** | Stripped Legacy Norms | **TRUE PHYSICS:** Removed 1/1000 division. Magnitude now purely distance-dependent. |

---

## 📦 Data Pipeline Fixes

### Sensor Mount Calibration (MCAP Visualization)
Both `scripts/test_shenron.py` and `scripts/data_to_mcap.py` now include correct physical mount offsets so sensors align properly in Foxglove:
- **LiDAR:** `Z = +2.5m` (Roof mount)
- **Radar (Native + Shenron):** `X = +2.0m, Z = +1.0m` (Front bumper/grille)

### LiDAR Point Cloud — Full 7-Column Support
Both MCAP scripts now publish all 7 Semantic LiDAR fields:
```
[x, y, z, velocity, cos_inc_angle, object_id, semantic_tag]
```
`object_id` and `semantic_tag` are correctly decoded via `.view(np.uint32)` before packaging.

---

## 🏃 Performance Optimization

**Testbench Speed (for 9 seconds of data):**
| State | Time | Cause |
| :--- | :--- | :--- |
| Before Today | ~20 min | 128 chirps × 3 models |
| Iteration 06 | ~5 min | 32 chirps × 3 models |
| **Iteration 07 (Current)** | **~5-7 min** | **64 chirps × 1 model (awrl1432 only)** |

To re-enable all three radar models, update `test_shenron.py` line 84:
```python
radar_types = ['awrl1432', 'radarbook', 'ti_cascade']
```

---

## 🧭 Coordinate System (Final Reference — DO NOT CHANGE)

This is the authoritative mapping that must be maintained across all files:

```
CARLA LiDAR Output:
  col[0] = X (Forward)
  col[1] = Y (Right, LHS)
  col[2] = Z (Up)

Shenron Internal Input (after lidar.py swap):
  col[0] = CARLA Y (Side)       <-- Index 0 is Side
  col[1] = CARLA X (Forward)    <-- Index 1 is Forward
  col[2] = CARLA Z (Up)

Shenron Cropping Filter (Cropped_forRadar):
  skew_pc[:, 1] > 0.5           <-- Forward filter (Y=Index 1)
  skew_pc[:, 0] in [-30, +30]   <-- Side filter (X=Index 0)
  skew_pc[:, 1] < 120           <-- Max forward range

MCAP Output (ROS/Foxglove RHS):
  Negate Y column before packaging
  Apply sensor mount pose offsets
```

---

## ▶️ Running Future Iterations

```powershell
# Activate Environment
conda activate carla312

# Run the testbench (replace XXXX with your iteration name)
python scripts/test_shenron.py --iter "07_high_def_sync"

# Output: Shenron_debug/iterations/07_high_def_sync/07_high_def_sync.mcap
```

### Known Pending Issues (Next Steps)
1. **Turn Lag:** Shenron points appear to trail ~0.5-1 frame behind the CARLA native radar during sharp turns. Suspected cause: LiDAR data captured at `T-1` but rendered with ego pose at `T`. Requires timestamp sync investigation in `recorder.py`.
2. **Angular FOV Validation:** Compare Shenron angular output vs. AWRL1432BOOST hardware spec (`+/- 60°`) to ensure angular clipping is not removing valid detections.
3. **CFAR Threshold Tuning:** Iteration 16 magnitude boost (+234%) has significantly improved SNR. `threshold: 20` in `config.yaml` should be recalibrated to maintain precision.
4. **Target Classification:** Proof of bimodal Magnitude Distribution in Iteration 16 allows for potential MLC (Machine Learning Classifier) based on point intensity maps.

---

## 📁 Key Files Reference

| File | Role |
| :--- | :--- |
| `scripts/ISOLATE/model_wrapper.py` | Public API — single entry point for Shenron |
| `scripts/ISOLATE/e2e_agent_sem_lidar2shenron_package/lidar.py` | LiDAR ingestion, semantic mapping, axis swap |
| `scripts/ISOLATE/e2e_agent_sem_lidar2shenron_package/ConfigureRadar.py` | Hardware profiles (awrl1432, radarbook, ti_cascade) |
| `scripts/ISOLATE/e2e_agent_sem_lidar2shenron_package/shenron/Sceneset.py` | Fresnel reflection + RCS physics |
| `scripts/ISOLATE/e2e_agent_sem_lidar2shenron_package/shenron/heatmap_gen_fast.py` | GPU FMCW ADC synthesis |
| `scripts/ISOLATE/sim_radar_utils/radar_processor.py` | Range/Doppler FFT + CFAR detection |
| `scripts/ISOLATE/sim_radar_utils/config.yaml` | DSP processor configuration |
| `scripts/test_shenron.py` | Testbench — generates and packages iterations |
| `scripts/data_to_mcap.py` | Main MCAP converter (Dashboard path) |
| `src/recorder.py` | Data capture — includes velocity + semantic metadata |
| `intel/radar/archive/shenron_architecture_deepdive.html` | Visual HTML architecture guide |

---

*Generated by Antigravity AI | Fox CARLA ADAS Pipeline | 2026-04-06 (Calibration Milestone)*