Multi-Camera 3D Object Tracking via Browser-Based Computer Vision
Project Proposal
Group Members: Vidhatu Patel, Thomas Fasan
Reconstructing the three-dimensional trajectory of a moving object from multiple camera views is a well-studied problem, yet practical deployment remains inaccessible to most users. Existing solutions typically demand dedicated hardware, offline calibration pipelines, or substantial expertise in computer vision. This project addresses that gap by building a browser-native, multi-camera 3D tracking system that requires nothing beyond consumer webcams and a standards-compliant web browser.
The system exposes a web server to which any camera-equipped client can connect and immediately begin contributing to a shared tracking session. All compute-intensive vision workloads run client-side, accelerated with WebAssembly for OpenCV pipelines and WebGPU for YOLO neural pipelines. Cameras are assumed static once positioned, enabling continuous online refinement of both object trajectories and camera extrinsics within a unified factor-graph framework.
For this project we will only consider the case of single-target multi-view tracking. In the future (or if time allows) this could be expanded to multi-target multi-view tracking.
A WebSocket-connected server will accept streams of detections from an arbitrary number of clients. The core CV pipeline runs entirely in the browser using either OpenCV or YOLO. OpenCV is compiled to WebAssembly via Emscripten (an officially supported build target) and dispatched to a Web Worker to avoid blocking the UI thread. For YOLO pipelines, the ONNX runtime will be used for GPU-accelerated inference via WebGPU when available, attempting to fallback to other APIs otherwise. This fallback behavior is not guaranteed to work on all browsers and hardware configurations, but will be attempted when possible.
The pipeline will be user-supplied OpenCV javascript code or compatible YOLO models (user must upload .onnx files). Pipelines should be lightweight enough to operate on all clients in the session (possibly low-end hardware or mobile devices) without degradation. Uploaded pipelines will be shared between all clients in the same session.
This means arbitrary javascript can be shared and evaluated between clients in the same session. To mitigate security risks, only the client that created the session (the ‘host’) is allowed to upload the pipeline. It is assumed that clients in the same session trust the host (reasonable because the use case of this project is many devices in the same physical space contributing to shared tracking). To further mitigate security risks, each client can preview the JS source of shared OpenCV pipelines before running them in a Web Worker with limited access.
The YOLO upload path allows arbitrary ONNX models to be loaded and run. From https://app.opencve.io/cve/?vendor=onnx all known vulnerabilities are related to filesystem loading/downloading or python APIs, which is not an issue since the browser does not allow access to the clients filesystem and we are using JS apis. The ONNX runtime will run in a Web Worker and be configured to access WebGPU, WebGL, or WASM (https://onnxruntime.ai/docs/api/js/interfaces/InferenceSession.ExecutionProviderOptionMap.html) depending on the client’s hardware. Browsers sandbox these APIs, so it is secure. Even so, only the host will be allowed to upload YOLO pipelines.
With these precautions, even a malicious host is unlikely to negatively impact the clients in the same session with anything worse than compute exhaustion or camera capture (which is assumed to happen anyways).
Each connected client performs intrinsic calibration independently using a standard printed ChArUco board. ChArUco boards combine the robustness of ArUco marker detection with the sub-pixel corner accuracy of chessboard patterns, making them well-suited for in-browser calibration under variable lighting. Calibration is triggered once at session start and the resulting intrinsic parameters are transmitted to the server for use in multi-view reconstruction.
Before tracking begins, the user provides a coarse estimate of each camera’s position and orientation relative to a shared world frame via an interactive UI. Precise placement is not required; these estimates serve solely as initialization for downstream optimization. A 3D viewport will display the current pose estimates, allowing the user to adjust them interactively before committing to a session.
The system will support optional user-defined constraints that narrow the solution space and improve reconstruction accuracy. Supported constraint types include ground-plane assumptions (restricting object motion to a known plane) and kinematic priors such as parabolic trajectory models for projectile tracking. Constraints are encoded as factors within the GTSAM factor graph, where they participate naturally in joint optimization alongside measurement factors. For real-time estimation, observations that strongly violate these constraints will be filtered automatically, either by not introducing the constraint factors or all factors corresponding to that observation. Once recording has finished for a session, the user will be able to specify which observations/constraints to exclude for the final full-horizon estimation. This is useful in certain situations like recording many parabolic trajectories in a single session.
Camera extrinsics estimated from user input will inevitably contain error. Rather than treating calibration as a static pre-processing step, the system performs simultaneous optimization of object tracks and camera poses using GTSAM. Live estimation is possible with GTSAM’s iSAM2 solver, with more accurate full-horizon estimation taking place after recording for the session has completed. As more observations accumulate across frames and cameras, both the reconstructed trajectory and the extrinsic parameters are refined simultaneously, exploiting the consistency constraints inherent in multi-view geometry.
The complete system will be validated through a controlled multi-camera tracking experiment using a volleyball as the target object. A volleyball provides a well-defined color profile and follows a near-parabolic trajectory when thrown, making it a natural test case for both the color-based detection pipeline and the kinematic prior. Multiple cameras will be arranged to achieve overlapping fields of view, and the system will be evaluated on its ability to reconstruct a consistent 3D trajectory across all views.
Evaluation criteria include:
- Reprojection error of the reconstructed trajectory back onto each camera view
- Convergence behavior of extrinsic refinement over the duration of a session
- Qualitative consistency of the 3D trajectory visualized in Rerun across multiple test throws
All reconstructed tracks and camera geometry will be streamed to Rerun for real-time 3D visualization. Rerun’s native support for multi-camera scenes and time-series data makes it well-suited for inspecting the evolving state of the tracker, including camera frustums, per-frame detections, and the accumulated 3D trajectory.
- Webcam (any resolution)
- Browser with WebAssembly + WebGPU + Web Workers support (Chrome 113+, Edge 113+). Partial support for browsers with only WebAssembly + Web Workers.
-
Docker (encapsulates the complex build dependencies: GTSAM, OpenCV/Emscripten toolchain, Rerun server). The server itself will run as a docker container, with scripts provided for building+running the container.
-
High performance: GTSAM is compute intensive and memory+compute will scale with the number of observations so the server must have decent hardware specifications. A medium to high end laptop will probably be enough, final hardware requirements will be set according to benchmarks after the project implementation is complete. The server operator can disable live estimation and only conduct post-recording estimation if the server is not fast enough for live estimation.
-
OpenCV (Emscripten WASM build): camera calibration, ChArUco detection, color-based object detection.
-
ONNX runtime (onnxruntime-web): neural object detection, used for running YOLO models. Supports web environments out of the box.
-
Languages: Server uses Rust for main server code and C++ for GTSAM code, with bindings between them (auto generated via rust-bindgen).
-
GTSAM: factor-graph optimization for simultaneous tracking and extrinsic refinement. Has multiple solvers including iSAM2 meant for live estimation.
-
Rerun (rerun.io): real-time 3D visualization of tracks, camera geometry, and session state. Rerun is itself an in-memory database, and can be exported to disk as .rrd files.(https://rerun.io/docs/getting-started/data-out). The rerun python API only supports queries on .rrd files (not live data), but since we are using rust we can use the same rust query API used by the live visualizer. This is officially supported (rust side has public API and user facing docs) but is not finalized yet so a fixed version of rerun will be used. Rerun is async and multi-threaded, so one rerun server will be used for many sessions (each session logged on a different RecordingStream). Runs on server, rerun web visualizer exposed as a route.
-
SQLite: After each session ends its data is saved here for persistence. Interfaced via sqlx rust crate with sqlite driver, as it supports tokio.
-
Tokio: Most server operations are IO-bound, with GTSAM being an exception. Server code will be asynchronous using tokio as the async runtime, with a dedicated worker thread with a GTSAM instance running per-session. Neither of us have worked with async rust before, so this will be an interesting challenge.
-
Axum: Rust library for network requests and routing. Supports tokio.