VWM

VWM is a real-time world model renderer for interactive voxel worlds.

The goal is to use an open-source voxel engine as the live simulation source, capture the engine data that normally drives rendering, and feed that data into a neural rendering pipeline that generates photorealistic video with low latency and temporal stability.

Architecture Update: Digital Twin Vision

VWM is not only a neural renderer. It is a real-time digital twin platform built on top of a voxel simulation.

The voxel world is the authoritative simulation. Everything updates the voxel world. Everything renders from the voxel world.

The AI renderer never owns gameplay or simulation state. It only visualizes the existing world with different levels of realism.

Real Sensors
        |
Player Input
        |
AI Agents
        |
Automation
        |
Scripts
        |
-------------------
Unified World State
-------------------
        |
Voxel Renderer
Stylized Renderer
Neural Renderer
VR
AR
Replay

The renderer is simply another view of the same world.

Digital Twin Layer

VWM has a first-class digital twin subsystem for synchronizing real-world devices into the voxel simulation in real time.

Examples include door sensors, window sensors, motion sensors, temperature sensors, humidity sensors, smart lights, cameras, radar, GPS, BLE devices, RFID, OBD-II vehicles, IMUs, weather stations, power meters, solar systems, robotics, industrial PLCs, and custom ESP32 devices.

These update the voxel world exactly like player actions do.

Door Sensor
      |
Sensor Event
      |
World State
      |
Voxel Door Opens
      |
Native Renderer or Neural Renderer

The renderer should never know whether an event originated from a player, script, AI agent, automation rule, or real-world sensor.

Sensor Framework

Every device should expose:

• UUID
• type
• timestamp
• confidence
• current value
• metadata
• position
• coordinate mapping
• update frequency

Sensors should be hot-pluggable. Future transports include MQTT, WebSocket, ZeroMQ, BLE, USB, CAN, Serial, TCP, UDP, and ROS.

Device Registry

VWM stores synchronized devices in a persistent registry. Each device stores unique ID, friendly name, physical location, world location, calibration, connection method, status, last update, owner, and tags.

Current tool:

python3 tools/vwm_device_registry.py add --id front-door --name "Front Door" --connection mqtt --location "Entry"
python3 tools/vwm_device_registry.py list

Coordinate Calibration

Physical coordinates must map into voxel coordinates.

Real Room
      |
Calibration
      |
Voxel Coordinates

This lets physical sensors manipulate exact objects inside the simulation.

Object Identity And Semantic Metadata

Every meaningful world object should become a persistent entity rather than anonymous blocks.

Examples: door, window, desk, vehicle, chair, printer, monitor, camera, robot, tree, wall, lamp.

Objects contain UUID, geometry, components, state, metadata, relationships, and history.

Semantic metadata does not affect deterministic simulation. It improves rendering and AI understanding.

object:
  id: office_door

semantic:
  type: office door
  material: oak
  finish: satin
  scratches: around handle
  brass_handle: true

The voxel engine only needs to know it is a door. The neural renderer can receive richer semantic information for higher-quality rendering.

Hierarchical Prompt Injection

Rendering hints should exist at multiple levels:

World
  |
Region
  |
Building
  |
Room
  |
Object
  |
Component

Example prompt stack: modern suburban neighborhood -> single-family home -> engineering office -> dark walnut standing desk -> ultrawide matte display.

The renderer combines applicable metadata into conditioning without affecting simulation.

Hard Facts Vs Soft Hints

Hard facts are authoritative: door, width, height, material, opens inward, position, physics, collision.

Soft hints are renderer suggestions: cozy, modern, dusty, slightly worn, expensive looking, warm lighting, Scandinavian style.

Simulation uses hard facts. Neural rendering may use both hard facts and soft hints.

Multiple Rendering Modes

The same voxel world should support many visualization pipelines: native voxel, enhanced voxel, low poly, LEGO, clay, pixel art, watercolor, anime, photorealistic, cinematic, VR, AR, and replay.

Changing renderers must never modify simulation state.

World Replay And Knowledge Graph

Replay should record frame packets and sensor/world events. It should reproduce player movement, sensor updates, lighting, automation, AI actions, weather, and time.

The persistent knowledge graph stores objects, relationships, history, events, ownership, metadata, and sensor mappings. This becomes the semantic understanding layer for future AI agents.

Bidirectional Synchronization

VWM should support configurable bidirectional sync:

Real World
     ^
     |
Voxel World
     ^
     |
Automation

Opening a real door can update the voxel world. Opening a voxel garage door can optionally send a command to the real garage. Synchronization policy remains configurable.

Long-Term Goal

The long-term objective is a universal real-time digital twin platform where:

• The voxel world is the authoritative simulation.
• Real-world devices continuously synchronize into the simulation.
• Multiple rendering engines visualize the same world.
• AI rendering uses semantic metadata and hierarchical conditioning to generate temporally stable photorealistic imagery.
• The simulation remains deterministic and renderer-independent.
• New renderers, AI models, sensors, automation systems, and visualization backends can be added without changing the core world model.

The architecture prioritizes modularity, deterministic simulation, renderer independence, extensibility, and long-term compatibility so future AI rendering advances improve every connected world without requiring simulation changes.

Core Idea

Luanti remains the game and simulation layer. It owns world state, player movement, input, entities, blocks, lighting, camera pose, and normal gameplay.

world_engine becomes the first neural rendering layer. It receives gameplay-conditioned frame packets and generates video frames from prompt, input, RGB history, and later depth, lighting, camera motion, normals, and entity state.

The first working loop should be:

Luanti gameplay
  -> RGB/depth/camera/input capture
  -> frame packet bridge
  -> Python conditioning encoder
  -> world_engine inference
  -> generated frames
  -> debug viewer first
  -> Luanti overlay/replacement renderer later

Current Wiring Scaffold

The root-level VWM glue now lives outside the cloned/vendor repos:

• vwm/schema/frame_packet.capnp: target typed packet schema for Luanti, Python, replay, and datasets.
• vwm/schema/digital_twin.capnp: target typed schema for sensors, devices, persistent objects, metadata, and world events.
• vwm/schema/gpu_interop.md: external GPU/shared-buffer handle contract for future zero-copy paths.
• vwm/python/vwm/: Python bridge/runtime package.
• vwm/luanti_integration/: C++ capture and ZeroMQ bridge stubs for the future Luanti patch.
• vwm/runtime.md: runtime bridge notes and commands.
• vwm/experiments.md: experiment repo wiring notes.
• vwm/digital_twin.md: digital twin subsystem notes.
• vwm/volsplat_import.md: VolSplat-to-voxel import path notes.
• vwm/integration_manifest.json: machine-readable map of all runtime and experiment components.
• tools/vwm_check.py: verifies all expected repos and integration files exist.
• tools/vwm_bridge.py: replay or ZeroMQ bridge entry point.

Current smoke-test commands:

python3 tools/vwm_check.py
python3 tools/vwm_bridge.py replay --input /tmp/vwm_packet.jsonl --output /tmp/vwm_generated.jsonl --limit 1
python3 tools/vwm_export_rgb.py --input /tmp/vwm_frames.jsonl --output-dir /tmp/vwm_rgb
python3 tools/vwm_export_depth.py --input /tmp/vwm_frames.jsonl --output-dir /tmp/vwm_depth
python3 tools/vwm_device_registry.py list
python3 tools/vwm_splat_to_voxels.py --input /tmp/gaussians.jsonl --output /tmp/voxels.jsonl

The bridge defaults to dry-run mode. Dry-run mode echoes captured RGB into a GeneratedFramePacket so transport, replay, timing, and Luanti presentation can be built before the full model environment is installed.

Current Luanti-side capture:

vwm_enable = true
vwm_capture_rgb = true
vwm_rgb_capture_mode = screenshot
vwm_capture_depth = true
vwm_record_path = /tmp/vwm_frames.jsonl

With that enabled, Luanti writes one JSON packet per rendered frame after Game::drawScene(). The current packet includes real RGB framebuffer bytes, OpenGL depth buffer bytes, and camera/input metadata. Normals, lighting, and motion-vector payloads are declared but not captured yet.

Repo Roles

luanti/

Luanti is the source engine.

It should eventually provide:

• The live voxel world simulation.
• The native rendered RGB frame.
• Depth buffer output.
• Optional normal, lighting, material, and motion-vector buffers.
• Camera pose, projection, FOV, timestep, and viewport data.
• Player input and gameplay actions.
• Entity and nearby world state for model conditioning.
• The final display hook where generated frames can replace or overlay the normal game frame.

The first Luanti work is not AI. The first job is reliable capture: RGB frame, camera pose, and input with frame IDs and timestamps.

world_engine/

world_engine is the core model inference package.

It should eventually provide:

• Prompt-conditioned frame generation.
• Control-conditioned generation through CtrlInput.
• RGB history ingestion through appended frames.
• Temporal frame generation using previous frames.
• Quantized GPU inference experiments.
• A clean Python API that the VWM bridge can call.

The first integration should keep conditioning simple: prompt + player input + RGB history. Depth, normals, lighting, entities, and splats come after that loop works.

third_party/imgui/

Dear ImGui is for developer-facing UI inside or beside Luanti.

It should eventually provide:

• AI rendering toggle.
• Native/overlay/replacement render mode controls.
• Prompt selector.
• Buffer previews for RGB, depth, normals, and generated output.
• Latency graphs.
• Dropped-frame counters.
• Model status and VRAM status.

This is not the main application UI. It is a debug and control surface for building the renderer.

third_party/tracy/

Tracy is for profiling.

It should measure:

• Luanti capture time.
• GPU readback or GPU interop cost.
• Bridge serialization and transport time.
• Python packet decode time.
• Conditioning encoder time.
• world_engine inference time.
• Post-processing time.
• Generated frame upload/presentation time.
• Total input-to-photon latency.

Tracy should be added early because the whole project depends on latency visibility.

third_party/libzmq/

ZeroMQ is one possible game-to-AI transport.

Use it if the first bridge needs to be simple and fast to prototype:

• Luanti publishes frame packets.
• Python subscribes or receives packets.
• Python returns generated frame metadata or generated frame handles.
• The bridge can drop old packets when inference falls behind.

ZeroMQ is a good first transport if iteration speed matters more than a strict schema.

third_party/capnproto/

Cap'n Proto is the other bridge option.

Use it if the frame packet schema needs to be strongly typed and stable:

• Define a FramePacket schema.
• Encode camera, input, buffer metadata, entities, and timing.
• Share the same schema between C++ and Python tooling.
• Keep room for recorded datasets and replay files.

The likely path is to prototype with ZeroMQ and move packet structure into Cap'n Proto once the data shape stops changing.

third_party/streamline/

NVIDIA Streamline is for later rendering optimization features.

It may eventually provide:

• DLSS Super Resolution.
• DLSS Frame Generation.
• NVIDIA Reflex.
• NIS upscaling.

This should not block the first world model loop. It belongs after generated frames can already be displayed and timed.

experiments/depth_anything_v2/

Depth Anything V2 is a fallback and validation depth tool.

It can help with:

• Estimating depth from generated frames.
• Comparing AI-estimated depth against Luanti's real depth buffer.
• Testing depth-conditioned post-processing before engine depth capture is stable.
• Building offline dataset analysis tools.

Runtime should prefer true Luanti depth when available. Monocular depth is useful for validation and experiments, not as the primary source of truth.

experiments/raft/

RAFT is for optical flow experiments.

It can help with:

• Estimating motion between RGB frames.
• Comparing learned optical flow with engine camera delta.
• Warping previous generated frames for temporal stability.
• Debugging flicker and motion inconsistency.

Runtime should eventually prefer engine motion vectors or camera-derived motion where possible. RAFT is mainly a research and validation tool.

experiments/rife/

RIFE is for frame interpolation.

It can help with:

• Filling frames while world_engine generates the next batch.
• Smoothing multi-frame model output.
• Testing latency/quality tradeoffs for generated video.
• Producing intermediate frames during async inference.

RIFE belongs after the basic generation loop works. It should not hide broken synchronization or poor frame pacing.

experiments/gaussian_splatting/

Gaussian Splatting is an optional 3D stability layer.

It may eventually help with:

• Building a short-lived 3D scene cache from generated frames and captured camera/depth data.
• Stabilizing geometry across camera motion.
• Rendering a persistent photorealistic scene representation between model updates.
• Testing whether splats can reduce temporal shimmer.

This is not the first renderer. It is a later layer that can sit beside or behind the frame generator once depth/camera capture is solid.

experiments/volsplat/

VolSplat is a voxel-aligned Gaussian reconstruction experiment.

It can help with:

• Turning multi-view captures into voxel-aligned 3D Gaussians.
• Using recorded Luanti RGB/depth/camera packets as reconstruction inputs.
• Converting predicted splat structure back into voxel occupancy candidates.
• Importing reconstructed real or generated spaces into the authoritative voxel world.
• Creating persistent objects for the digital twin layer from reconstructed geometry.

VolSplat should not own simulation state. Its role is import/reconstruction: captured views -> voxel-aligned splats -> voxel/object candidates -> deterministic world state.

experiments/nerfstudio/

Nerfstudio is an offline neural reconstruction workbench.

It can help with:

• Exploring NeRF and Gaussian Splatting workflows.
• Inspecting captured Luanti datasets.
• Testing camera pose and reconstruction quality.
• Comparing VWM's recorded data format against known reconstruction pipelines.

Nerfstudio is for research, dataset analysis, and tool learning. It is not expected to be in the real-time runtime path.

experiments/colmap/

COLMAP is for offline camera pose and reconstruction validation.

It can help with:

• Structure-from-motion checks on recorded frame sequences.
• Dataset conversion for Gaussian Splatting or Nerfstudio.
• Debugging whether camera intrinsics/extrinsics are exported correctly.
• Validating synthetic captures against standard reconstruction tools.

Luanti already knows the true camera pose, so COLMAP is not needed for live runtime. It is useful for validating dataset exports and external reconstruction compatibility.

How The Pieces Fit Together

Runtime Path

luanti/
  captures frame data
  captures input/camera/world state
  shows native or generated frame

third_party/imgui/
  exposes runtime toggles and debug views

third_party/tracy/
  profiles capture, bridge, inference, and presentation

third_party/libzmq/ or third_party/capnproto/
  moves frame packets between C++ and Python

world_engine/
  generates photorealistic frames from conditioning data

third_party/streamline/
  optionally upscales or frame-generates after the base path works

Python AI Path

incoming Luanti FramePacket
  -> packet decoder
  -> buffer normalization
  -> conditioning encoder
  -> prompt/control/RGB history formatting
  -> world_engine.WorldEngine
  -> generated frame batch
  -> temporal buffer
  -> optional post-processing
  -> output back to viewer or Luanti

Experiment Path

recorded Luanti packets
  -> replay runner
  -> Depth Anything V2 for depth validation
  -> RAFT for flow validation
  -> RIFE for interpolation tests
  -> COLMAP/Nerfstudio/Gaussian Splatting/VolSplat for 3D reconstruction tests

The experiment path should consume recordings from the runtime path. That keeps research tools from complicating the first live integration.

Frame Packet Target

The bridge should move one packet per captured game frame.

Each packet should eventually contain:

• frame_id
• timestamp_game
• timestamp_capture
• dt
• rgb
• depth
• normals
• lighting
• motion_vectors
• camera_position
• camera_rotation
• view_matrix
• projection_matrix
• fov
• viewport_width
• viewport_height
• keyboard_state
• mouse_delta
• mouse_buttons
• scroll
• actions
• player_state
• entity_state
• world_state_summary

The first packet can be much smaller:

• frame_id
• timestamp_capture
• rgb
• camera_position
• camera_rotation
• keyboard_state
• mouse_delta
• actions

Development Order

1. Capture RGB, camera pose, and input from Luanti.
2. Add frame IDs and timestamps.
3. Stream packets to Python.
4. Decode packets and display them in a Python debug viewer.
5. Map input into world_engine.CtrlInput.
6. Feed RGB history and prompt into world_engine.
7. Generate frames outside the game first.
8. Add latency profiling with Tracy.
9. Add Luanti debug controls with ImGui.
10. Send generated frames back to Luanti.
11. Overlay generated frames.
12. Replace native frames.
13. Add depth capture.
14. Add lighting, normals, and optional motion vectors.
15. Add temporal buffering and async batching.
16. Add post-processing, upscaling, and optional interpolation.
17. Evaluate Gaussian Splatting, VolSplat, Nerfstudio, and COLMAP on recorded datasets.
18. Use VolSplat on recorded multi-view captures to produce voxel-aligned Gaussians.
19. Convert splat/gaussian outputs into voxel occupancy and persistent world objects.
20. Import approved voxel/object candidates into the authoritative Luanti world.

TODO

• Establish the repo-level integration target.

  • Use luanti/ as the open-source voxel game engine foundation.
  • Use world_engine/ as the core world model inference package.
  • Define the first end-to-end demo as Luanti gameplay data driving world_engine frame generation.

• Build the Luanti data capture layer.

  • Locate the active client render path in luanti/src/client/.
  • Add framebuffer capture for RGB game frames.
  • Add depth buffer extraction from the active renderer.
  • Add optional normal map extraction from shader outputs.
  • Add optional motion vector or camera delta extraction.
  • Capture camera pose, projection, timestep, FOV, and viewport size each frame.
  • Capture player input: keyboard, mouse delta, mouse buttons, scroll, and gameplay actions.
  • Capture entity and world state needed for conditioning: player state, nearby objects, block IDs, lighting, and time of day.

• Build the custom rendering hooks.

  • Identify where Luanti submits the final frame to the display.
  • Add a replace-or-overlay path for generated frames.
  • Add a toggle to switch between native Luanti rendering, AI overlay, and AI replacement.
  • Add frame timing hooks so generated frames can be synchronized with gameplay.

• Build shader and GPU buffer support.

  • Add a shader pack or renderer mode that exposes depth, normals, lighting, and optional material buffers.
  • Support rasterized shader outputs first.
  • Leave a clean extension point for RTX or hardware ray tracing experiments.
  • Normalize captured GPU buffers into stable tensor-friendly formats.
  • Add an external GPU/shared-buffer handle contract for future zero-copy transfer.
  • Implement and validate CUDA/GL IPC import/export on a real GPU build.

• Build the game-to-AI data bridge.

  • Define a frame packet schema for RGB, depth, normals, lighting, camera pose, motion, input, and entity state.
  • Add a low-latency transport between Luanti and Python.
  • Start with a local IPC path suitable for development.
  • Add timestamps, frame IDs, and dropped-frame tracking.
  • Add recording and playback support for deterministic testing.

• Build the conditioning encoder.

  • Convert Luanti frame packets into world_engine inputs.
  • Map captured player input to world_engine.CtrlInput.
  • Encode camera motion, depth, lighting, and entity state as optional conditioning channels.
  • Keep a compatibility path for prompt + control + RGB history only.

• Integrate world_engine.

  • Create a repo-level example that imports world_engine from world_engine/src/.
  • Load a model through WorldEngine.
  • Set scene style through the prompt system.
  • Append real RGB frames as initial history.
  • Generate frames from control input and temporal history.
  • Support multi-frame batches from Waypoint-style models.

• Build temporal stability systems.

  • Add a temporal frame buffer for RGB history and generated-frame history.
  • Track camera delta and input delta across frames.
  • Feed consistent history into the model to reduce flicker.
  • Add frame rejection or fallback when model output is late or unstable.

• Build the neural rendering pipeline.

  • Use the latent or diffusion model path in world_engine as the first frame generator.
  • Add denoise and color correction post-processing.
  • Add optional AI upscaling or DLSS-style upscaling.
  • Add optional frame interpolation.
  • Add async batching for multi-frame outputs.
  • Match generated frame pacing to Luanti gameplay timing.

• Add optional 3D stability layers.

  • Prototype depth reconstruction from captured depth and generated frames.
  • Prototype a scene representation cache for recent geometry.
  • Evaluate Gaussian Splatting as an optional stability layer.
  • Add VolSplat as a voxel-aligned splat reconstruction/import experiment.
  • Add a first Gaussian-point to voxel-occupancy conversion scaffold.
  • Ingest native VolSplat Gaussian outputs.
  • Convert reconstructed splats into Luanti blocks/entities.
  • Add a real-time splat renderer only after the 2D frame path works.

• Optimize GPU execution.

  • Run inference on CUDA first.
  • Evaluate world_engine quantization options for latency and VRAM use.
  • Add TensorRT or other optimized runtime experiments if needed.
  • Add VRAM buffering, tensor reuse, and cache management.
  • Track latency for capture, bridge, inference, post-processing, and presentation separately.

• Build developer tools.

  • Add a debug overlay in Luanti for latency, buffers, capture status, model status, and toggles.
  • Add a Python-side debug view for incoming packets and generated frames.
  • Add logging for frame IDs, timestamps, dropped frames, and generation time.
  • Add a replay runner that feeds recorded packets into world_engine.

• Build dataset and training support.

  • Use recording/playback to generate paired Luanti frame data.
  • Define a paired dataset format for game buffers and target photorealistic frames.
  • Add scripts for dataset validation and visualization.
  • Add optional LoRA or adapter fine-tuning experiments.
  • Keep training code separate from the runtime path until needed.

• Build the digital twin subsystem.

  • Define a sensor/device/object/world-event schema.
  • Add a persistent device registry.
  • Add a generic sensor event data model.
  • Add persistent world object and semantic metadata data models.
  • Add hard-fact vs soft-hint metadata separation.
  • Add hierarchical prompt hint data model.
  • Add sensor-event replay into world events.
  • Wire sensor events into Luanti world-state updates.
  • Add coordinate calibration UI/tools.
  • Add MQTT/WebSocket/BLE/Serial/CAN/ROS transports.
  • Add persistent knowledge graph storage backend.
  • Add bidirectional automation policies.

• Define first milestones.

  • Milestone 1: capture Luanti RGB frames, camera pose, and player input.
  • Milestone 2: stream captured packets to a Python bridge.
  • Milestone 3: feed RGB history and input into world_engine.
  • Milestone 4: display generated frames outside the game for validation.
  • Milestone 5: overlay or replace the Luanti frame with generated output.
  • Milestone 6: add depth and lighting conditioning.
  • Milestone 7: optimize latency and temporal stability for interactive playback.