Skip to main content

​
How the Splat3D pipeline works

When you upload a video or photo set, Splat3D runs it through a sequence of stages that turn pixels into a 3D scene you can fly through. This guide gives you enough context to predict runtimes and pick the right quality tier for your workload.

​
At a glance

​
Stage 1 — Frame extraction

For video, frames are sampled at roughly 1.5 frames per second and capped at 800 frames (MAX_FRAMES in pipeline/training/frame_extract.py). Blurry frames are filtered out. For photo zips, up to 800 images are extracted (a bigger set is thinned evenly) and EXIF orientation is applied so portrait photos render right-side-up. If the zip holds a COLMAP model (images/ plus sparse/<model>/), the SfM stage is skipped and we go straight to training. Such a zip is not thinned: over 800 registered images, the scene fails (MAX_REGISTERED_IMAGES in pipeline/colmap/prebuilt.py). ARKit poses don’t skip it: GLOMAP still runs, and the poses set which way is up (see Supported inputs). Runtime: seconds.

​
Stage 2 — Structure-from-Motion

Splat3D figures out where each photo was taken from (camera poses) and produces a sparse 3D point cloud that anchors the next stage. Without accurate poses, training cannot produce a useful scene. The flow:
  1. Detect distinctive keypoints in each image.
  2. Match keypoints between images. For video, match each frame to its temporal neighbors. For photo zips, match every pair.
  3. Solve a global optimization that places every camera and every 3D point such that the math is consistent across all observations.
Runtime: 4–10 minutes depending on image count. Photo zips with many images take longer because matching is exhaustive (cost grows with the square of the image count).

​
Stage 3 — Gaussian Splat training

Now that we know where every camera is, we fit the scene representation: a few million 3D Gaussians, each with position, size, color, and opacity. Training is iterative:
  1. Render the current Gaussians from a known camera angle.
  2. Compare the result to the actual photo from that angle.
  3. Update the Gaussians to reduce the difference.
  4. Periodically split, clone, or remove Gaussians.
  5. Repeat for thousands of iterations.
The number of iterations and the resolution scale with the quality tier: The times cover every stage, /process to ready, and are approximate: they depend on video length, the number of frames and GPU availability (TIER_TABLE in packages/core/src/tiers.ts). Higher tiers fit finer detail and more reflective surfaces but cost more credits. Start at Standard and move up if you need sharper results.
Need higher resolution, larger captures, or a custom training configuration for archival or research work? Email [email protected] and tell us about your use case.

​
Stage 4 — Cluster cleanup

Trained scenes can have detached “floater” Gaussians — distant artifacts disconnected from the main subject. Splat3D analyzes the scene topology and records the result alongside the model so the viewer can hide or prune floaters. By default this is advisory, not destructive — the analysis is attached to the scene but no Gaussians are removed automatically. Runtime: seconds.

​
Stage 5 — Compression for the web

Trained models (PLY files) range from tens of megabytes to several gigabytes depending on tier and scene complexity. That’s too large to load instantly in a web viewer, so Splat3D compresses the model to a SOG file (1–10 MB) with negligible quality loss. The viewer always loads the SOG. The original PLY remains available as a direct download for users who want the raw training output. Runtime: ~30 seconds.

​
Stage 6 — LOD generation (optional)

For very large scenes, Splat3D can build a level-of-detail structure that streams the scene in chunks based on camera distance, keeping the framerate steady on huge models. Disabled by default; opt in per scene.

​
Practical takeaways

  • Photo uploads scale quadratically with image count due to exhaustive matching. Keep photo zips under ~500 images for reasonable latency.
  • Video uploads are capped at 800 sampled frames — there is no benefit to uploading hour-long videos.
  • Quality tier picks the speed/fidelity trade-off. All tiers run the same SfM stage; they differ in training iterations, resolution, and total runtime. See quality presets for the recommended tier per use case.

​
Further reading