- Expanded intel/internal/context.md with modern stage-based architecture and internal documentation structure.
- Corrected SNR calculation in model_wrapper.py by dividing out the CFAR threshold.
- Tuned CFAR threshold to 19dB for increased point cloud density.
- Integrated stop flag detection into PipelineManager and ShenronOrchestrator for graceful halting.
- Synchronized metrology gain offsets (-78.0dB) across production and testbench scripts.
- Added 'ti_cascade' support to default radar list and testbench visualization.
Established a common orchestration framework to eliminate logic
duplication between production data generation and the iterative
testbench.
Key changes:
- Created 'ShenronOrchestrator' in scripts/ISOLATE/shenron_orchestrator.py
to serve as the single source of truth for the processing loop.
- Refactored 'generate_shenron.py' to use the orchestrator, ensuring
production data benefit from research-level DSP improvements.
- Refactored 'test_shenron.py' to use the orchestrator, guaranteeing
that debug iterations are bit-identical to production outputs.
- Centralized LiDAR padding, model execution, and metrology/metric
serialization logic.
- Preserved Dashboard SSE telemetry patterns ([SHENRON_INIT/STEP])
to maintain full UI compatibility.
This restructuring ensures that any iterative changes made in the
'test_shenron' lab are automatically and safely inherited by the
Dashboard's automated simulation pipeline.
Restores sharp target peaks and improves signal clarity in Shenron radar outputs
by optimizing windowing functions and disabling smoothing-heavy interpolation.
- radar_processor.py: Switched windowing from Hann to Blackman-Harris for range
and velocity processing to achieve deeper sidelobe rejection (-92 dB) and
suppress horizontal artifacts.
- test_shenron.py: Replaced 'bicubic' interpolation with 'nearest' for Range-Doppler
(RD) and CFAR maps to restore diagnostic clarity and prevent target blurring.
- config.yaml: Removed hardcoded RD plot limits (xRange/yRange) to ensure the
renderer utilizes the true physical axis (e.g., ±26.8 m/s) derived from hardware specs.
- Added technical comments explaining the trade-off between main-lobe width and
dynamic range in the new DSP windowing.
- [Radar Cfg] Increase gain by +10dB (110dB -> 120dB) for all radar models in ConfigureRadar.py
- [Root Cause] Compensate for the SNR drop caused by restoring the 1/R^4 physics law (Iteration 37), which removed the artificial R^2 area boost and made RD maps 'hazy' or 'out of focus'.
- [Viz] Update SYSTEM_GAIN_OFFSET from 68.0 to 78.0 in plots.py and test_shenron.py
- [Viz] Increase dynamic range clipping ceiling from 45dB to 55dB
- [Root Cause] Prevents RA heatmap saturation. The +10dB gain increase caused signals to hit the previous 45dB hard-clip ceiling, making the dynamic plots look identical to the static ones (blown out).
- [Physics] Disable R^2 area expansion in Sceneset.get_loss_3 to restore
physically correct 1/R^4 two-way radar power law (point scatterer model)
- [Metrology] Normalize RA heatmap [0,1] before variance calculation in
model_wrapper.py to prevent 1e20 scalar overflow values
- [Metrology] Add peak_azimuth_spread_deg (Half-Power Beamwidth proxy)
to signal metrics for per-frame angular resolution monitoring
- [DSP] Switch spatial Angle-FFT window from Hanning to Hamming in
radar_processor.py for sharper beamforming on small antenna arrays (6-8 Rx)
- [Telemetry] Expose az_std and spread in [SHENRON_STEP] dashboard stream
for both generate_shenron.py and test_shenron.py pipelines
- [BugFix] Add missing sys/os imports in src/main.py and video_stage.py
that caused NameError crashes in the stage-based pipeline orchestrator
- Refactored heatmap rendering, stateful engine, and scan conversion from test_shenron.py into new sim_radar_utils.plots module
- Updated FastHeatmapEngine to automatically respect xRange/yRange overrides sourced from config.yaml
- Calibrated Range-Doppler plot limits to ±8 m/s in config.yaml for optimized visual inspection
- Cleaned up test_shenron.py by removing ~150 lines of redundant visualization logic and heavy Matplotlib imports
- Added __init__.py to sim_radar_utils to ensure proper package identification
- physics: increased metal roughness in Sceneset.py to 1.5mm for better diffuse scattering
- cfar: widened guard cells to 5x5 in config.yaml to isolate clutter; threshold kept at 20dB
- signal-processing: implemented 3x3 local maximum filter (NMS) in radar_processor.py to eliminate sidelobes
- config: enabled peak_grouping toggle in config.yaml
This physics engine overhaul introduces two major fidelity upgrades to the Shenron pipeline: 1. Physical Area Integration: Tethers incident power natively to the solid angle expanding physical area (dTheta*dPhi*R^2) ensuring cross-section energy output scales independently of LiDAR point cloud resolution. 2. Millimeter Jitter: Maps physical property roughness (e.g., 5mm concrete, 50um metal) as a sub-wavelength Gaussian distance distorter. This fractures the 'perfect mathematical mirror' bug inherent to flat CARLA meshes by substituting naive coherent summation (N^2) with physical diffuse Rayleigh scattering (N). Includes addition of interactive 3-slide HTML and dark-mode simulation presentation decks for project reporting.
Transitioned the Shenron radar engine from a rigid architecture to a modular,
tunable "Knobs and Dials" framework. This update establishes a physics-first
baseline derived from real-world electromagnetic behavior.
Core Changes:
- Modular Radar Profiles: Pre-configured profiles for TI Cascade, Radarbook,
and AWRL1432 with hardware-specific Bandwidth, Chirp, and nRx parameters.
- Physics Core (Sceneset.py):
- Full implementation of Fresnel and Beckmann-Spizzichino scattering.
- Pure 1/R^4 power law (1/R^2 transmit, 1/R^2 receive) via legacy scaling removal.
- Fixed cos(cos(theta)) bug in CARLA semantic lidar mapping.
- Antenna Gain Integration:
- Implemented separable Azimuth/Elevation gain patterns.
- Added Symmetric Azimuth LUT interpolation and Vertical FOV Hard Cutoff.
- Signal Processing:
- Optimized GPU-accelerated signal synthesis using PyTorch.
- Standardized 110 dB System Calibration Constant for hardware SNR matching.
Architecture & Documentation (intel/radar):
- Reorganized radar intel directory into structured subfolders (core, research,
diagnostics, archive) for better scalability.
- Added SHENRON_MODULAR_ARCHITECTURE.md (System overview).
- Added SHENRON_ANTENNA_GAIN_CALIBRATION.md (Gain physics deep-dive).
- Modernized package README with Windows/Conda specific usage instructions.
- Synchronized all internal and external documentation links.
Calibration: Iteration 18 (Antenna Gain Baseline)
Note: Remaining magic numbers in get_loss_3 are noted for future migration.
Transitioned the Shenron radar engine from a rigid architecture to a modular,
tunable "Knobs and Dials" framework. This update establishes a physics-first
baseline derived from real-world electromagnetic behavior.
Core Changes:
- Modular Radar Profiles: Pre-configured profiles for TI Cascade, Radarbook,
and AWRL1432 with hardware-specific Bandwidth, Chirp, and nRx parameters.
- Physics Core (Sceneset.py):
- Full implementation of Fresnel and Beckmann-Spizzichino scattering.
- Pure 1/R^4 power law (1/R^2 transmit, 1/R^2 receive) via legacy scaling removal.
- Fixed cos(cos(theta)) bug in CARLA semantic lidar mapping.
- Antenna Gain Integration:
- Implemented separable Azimuth/Elevation gain patterns.
- Added Symmetric Azimuth LUT interpolation and Vertical FOV Hard Cutoff.
- Signal Processing:
- Optimized GPU-accelerated signal synthesis using PyTorch.
- Standardized 110 dB System Calibration Constant for hardware SNR matching.
Additional Documentation:
- intel/radar/SHENRON_MODULAR_ARCHITECTURE.md: Architecture and "Knobs/Dials" overview.
- intel/radar/SHENRON_ANTENNA_GAIN_CALIBRATION.md: Physics of Antenna Gain and 1/R^4 logic.
Note: Remaining magic numbers in get_loss_3 (K_sq, scat_normalization, lobe_frac)
are noted for future migration into ConfigureRadar.py.
Standardizes the metrology diagnostic suite for high-fidelity focus and
consistent dB scaling.
Core Improvements:
- Signal Processing: Switched Range and Doppler FFTs to Hann windowing,
reducing peak width by ~50% for sharper target focus (Iteration 27).
- Visualization Style: Implemented a fixed-scale MATLAB-style Jet colormap
(-5 to 45 dB). Disabled per-frame auto-scaling to preserve temporal
intensity consistency and physical R^-4 decay.
- Calibration: Applied a -68.0 dB System Gain Offset in post-processing
(calculated via statistical analysis of raw power levels in Iter 28)
to align noise floors (~7 dB) and target peaks (~41 dB).
- Pipeline: Standardized these focus and calibration settings across both
the testbench (test_shenron.py) and packaging pipeline (data_to_mcap.py).
- Docs: Verified and restored gemini.md at the repository root as the
master project context.
This milestone (Iteration 29) finalizes the visual interpretability of the
physics-based Shenron radar engine.
Restores high-fidelity physical realism to the C-Shenron radar engine by
aligning the synthesis pipeline with the pure Radar Range Equation.
Core Improvements:
- Physics: Restored 1/R^4 power-delay law (1/R^2 voltage) in Sceneset.py
and heatmap_gen_fast.py. Stripped legacy 1/1000 normalizations and
R^2 area-growth workarounds.
- Geometry: Fixed FFT index asymmetry in radar_processor.py, achieving a
perfectly symmetric 120° FOV sector.
- Metrology: Implemented "Radar Blue" (Viridis) 120° fan-projection for
diagnostic Range-Azimuth heatmaps.
- Automation: Integrated RD/RA/CFAR heatmap persistence into the
automated simulation-to-MCAP pipeline (data_to_mcap.py).
- Docs: Comprehensive update of intel/ directory, including Iterations 17-26
and the Physics/Symmetry Milestone deep-dive.
This milestone ensures that target brightness and spatial positioning
correctly mimic real-world TI AWRL1432 radar hardware.
- Engine: Modified CFAR and Processor to extract raw Range-Doppler and Range-Azimuth energy heatmaps.
- Visuals: Integrated high-fidelity Matplotlib colormapping (Viridis/Magma) for Foxglove image streaming.
- Data: Implemented 32-bit raw .npy persistence and JSONL telemetry for frame-level SNR analysis.
- Tools: Added a dedicated verification utility for end-to-end signal flow validation.
- Docs: Comprehensive documentation for the new 'Radar Lab' architecture in /intel/.
Implemented a major architectural refactor of the C-SHENRON radar engine to achieve physically stable, context-independent detections.
1. Physics Engine Refactor (Iteration 16):
- Sceneset.py: Replaced global 1/N normalization with a fixed Area-Density Integration model. This prevents Energy Starvation where buildings previously suppressed car detections.
- Gaussian Damping: Integrated a 20 deg vertical beamwidth profile to physically suppress tree-top clutter while preserving boresight targets.
- ConfigureRadar.py: Standardized hardware profiles with calibrated 110dB gain and removed Iteration 15 bandages.
2. Repository Reorganization:
- Moved analysis and verification scripts to a new scripts/analysis/ directory.
- Updated path resolution in moved scripts to ensure project-wide stability.
3. Documentation & Metrology:
- Created 3D vertical energy suppression.md detailing the Resolution Independence breakthrough.
- Updated Shenron_debug.md with Iterations 14-16, recording a +234% Target Magnitude recovery.
- Updated Shenron_Debug_Plan.md and possible_issue_resolution.md to reflect resolved technical blockers.
Implemented Gaussian vertical-gain damping (-2.77 constant) to address the 'Super-Reflector' tree density problem.
- [ConfigureRadar] Defined total vertical_beamwidth for AWRL1432 (30.0) and Radarbook (60.0).
- [Sceneset] Added G_vertical damping math in get_loss_3 to suppress high-elevation energy.
- [Sceneset] Retained debug logs (G_vertical average/min/max) for Iteration 14b calibration.
Hiccups:
- Resolved environment activation (carla312) in verification shell.
- Corrected test_shenron.py command syntax.
Comprehensive integration of the Shenron radar model, enabling high-fidelity
physics-driven radar synthesis from semantic LiDAR data.
Core Changes:
- Sensor Migration: Updated EGO vehicle to use 'sensor.lidar.ray_cast_semantic'.
- Recorder Enhancement: Modified src/recorder.py to dynamically capture
6/7nd-column semantic LiDAR buffers [x, y, z, cos, obj, tag].
- Orchestration: Created scripts/ISOLATE/model_wrapper.py with the
ShenronRadarModel class for a unified physics-to-detection API.
- Processor Logic: Fixed broadcasting bugs and hardcoded antenna counts
in RadarProcessor to support dynamic radar models (radarbook/ti_cascade).
- Rich Data Output: Updated extraction logic to include [velocity, magnitude]
parameters for ADAS performance analysis.
- Batch Pipeline: Added scripts/generate_shenron.py to synthesize radar
data as a post-processing step, maintaining simulation performance.
- Visualization: Updated scripts/data_to_mcap.py to support the
new /radar/shenron topic with a 5-field rich PointCloud schema.
Stability & Fixes:
- Implemented CUDA-enabled PyTorch checks with a robust CPU fallback.
- Corrected CARLA 0.9.16 semantic column mapping in ISOLATE/lidar.py.
- Handled indexing and range-slicing errors in the CFAR detection routine.
