SetLab
From a text prompt to a production-ready 3D film set — deployed straight into Unreal Engine 5.
Overview
SetLab turns a plain-language description into a structured, editable 3D set. It orchestrates a division of labor across specialized models — Claude for schema-validated scene layout, Hyper3D Rodin Gen-2 for text-to-3D meshes and PBR retexturing, FLUX 1.1 Pro and Gemini for reference imagery, World Labs Marble for full 3D environments — and previews everything in a browser-based first-person walkthrough.
One click deploys to Unreal Engine 5: auto-import, spec-based actor placement, Megascans ground materials, and a full Lumen lighting rig built from the LLM-generated environment block. A director can say “change the lighting to sunset” mid-shoot and watch it land within one frame.
Full stack
- Python 3 · FastAPI · Pydantic v2 · httpx/asyncio
- Anthropic SDK · Rodin Gen-2 · FLUX 1.1 Pro · Gemini · Marble
- Next.js 16 · React 19 · Three.js / React Three Fiber
- Server-Sent Events · json-repair fallback parsing
- glTF 2.0 + USD exporters · UE5 Python API · Megascans
- Ollama local fallback · mock backend for offline E2E
Engineering highlights
The parts a code review would find interesting.
01LLM-as-classifier edit routing
Director instructions are tier-classified by Claude into instant (<1s lighting/fog), fast (30–60s retexture), and moderate (3–6min asset regen) — instant edits flow through a file bridge the UE watcher applies within a single Slate tick.
02An 871-line UE5 Python watcher
Slate tick callbacks detect file changes, import GLB/glTF, place StaticMeshActors with Y-up-meters → Z-up-cm conversion, and build the lighting rig — directional light with color-temperature-to-RGB conversion, sky atmosphere, height fog, volumetric clouds, Lumen GI.
03Exporters written from scratch
Custom glTF 2.0 and USD exporters with regression tests locking Euler-to-quaternion conventions (including gimbal-lock branches), so the web viewer, the glTF export, and the Unreal deploy can never silently diverge.
04Concurrency with a budget
Semaphore-bounded asyncio fan-out runs up to 10 simultaneous Rodin mesh jobs; every heavy pipeline stage is opt-in and independently triggerable for cost control, with a mock backend enabling fully offline end-to-end tests.
05Vendor churn as a feature
The git history records deliberate model swaps — Tripo3D → Hunyuan → Rodin Gen-2, Gemini → FLUX — behind pluggable backends, including a local Ollama fallback. Evaluating and replacing AI vendors is part of the architecture, not a rewrite.