Backlot
Turn any Unreal Engine 5 scene into pixel-aligned, ML-ready training data.
Overview
Backlot generates ML training data from virtual scenes: a CaptureRig owns four SceneCapture2D actors locked to one pose per frame, emitting RGB, float16 EXR depth, world-space normals, and albedo — plus camera intrinsics/extrinsics and per-actor 3D bounds projected to 2D boxes.
Captures ingest into PostgreSQL/PostGIS and serve through FastAPI to a Next.js explorer with class filters, bbox overlays, 3D proximity queries, and an animated camera-trajectory viewer. The whole demo stack runs with two Docker Compose commands and ships with sample data.
Full stack
- Python 3.11 · UE5.5+ embedded Python · SceneCapture2D
- FastAPI · SQLAlchemy 2.0 typed ORM · Pydantic v2
- PostgreSQL 16 + PostGIS 3.5 · GeoAlchemy2
- Next.js 16 (RSC) · React 19 · React Three Fiber
- OpenCV + NumPy EXR pipeline · Docker Compose
Engineering highlights
The parts a code review would find interesting.
01Zero-drift multi-modal capture
The four-camera rig was chosen after systematically abandoning UE5 Remote Execution (macOS multicast issues) and async screenshots (pose drift between modalities). The pivot to embedded Python cut debugging time by roughly 90% — and the decision log explains why.
02Two-runtime Python architecture
UE5's embedded interpreter (no pip) handles capture and subprocesses out to a host venv for OpenCV/NumPy EXR conversion — with forced module reloads to defeat UE5's import caching and a runtime assertion guarding against stale projection code.
03Projection math you can unit-test
Pinhole projection — UE5 left-handed rotator basis → extrinsics → 3D AABB-to-2D bbox with near-plane culling — is pure stdlib, no numpy, no unreal import, deliberately testable outside the editor.
04Idempotent at every layer
Deterministic uuid5 session IDs make re-captures upsert instead of duplicate; ON CONFLICT DO UPDATE for sessions, composite upserts for frames, delete-and-bulk-insert for per-frame objects. Re-run anything, orphan nothing.
05Spatial queries surfaced to the UI
Camera poses are stored as PostGIS PointZ geometry under a GiST index; ST_3DDWithin proximity queries combine with ORM-level class filters and flow all the way up into the React filter panel and a 3D trajectory viewer that remaps UE5 coordinates into Three.js space.