> ## Documentation Index
> Fetch the complete documentation index at: https://docs.splat-3d.com/llms.txt
> Use this file to discover all available pages before exploring further.

# SfM Backends

> Choose the right structure-from-motion engine for your captures.

# SfM Backends

Structure-from-Motion (SfM) is the first stage of the pipeline. It takes your images and estimates where each camera was positioned in 3D space. These camera poses are then used to train the Gaussian Splat model.

Two SfM backends run today:

## Backend comparison

| Backend | Speed | Quality | Best for |
| - | - | - | - |
| `colmap_sift_glomap` | Fast | Excellent | Every upload, ARKit captures included (default) |
| `prebuilt_colmap` | Instant | Yours | A COLMAP model you already built |

> [!NOTE]
>
> `prebuilt_colmap` is selected automatically when your zip contains a
> `sparse/` model — it is not something you request via `sfm.backend`.
> The pipeline also registers `none_arkit`, a pose passthrough, but no API
> input gives it the model it copies, so selecting it fails the scene
> (refunded; `NoneArkitBackend.run` in `sfm_backends/none_arkit.py`).

> [!NOTE]
>
> The API also accepts legacy short-names for backward compatibility:
> `glomap` and `colmap` both resolve to `colmap_sift_glomap`, and `none`
> resolves to `none_arkit`. Prefer the canonical names in new code; the
> `colmap` alias emits a deprecation warning and may be removed.

## `colmap_sift_glomap` (default)

GPU SIFT feature extraction (via [COLMAP](https://colmap.github.io/)) combined with [GLOMAP](https://github.com/colmap/glomap)'s modern global mapper, which solves all camera poses simultaneously rather than incrementally.

```json theme={null}
{
  "sfm": {
    "backend": "colmap_sift_glomap"
  }
}
```

**Advantages:**

* Significantly faster than incremental COLMAP
* GPU SIFT for feature extraction
* Global optimization is more robust for video walkthroughs
* Default for all presets

**When to use:** Almost always. This is the recommended backend for video input and most photo sets. If you don't override `sfm.backend`, this is what runs.

### Options

Set these in `sfm.options`. SfM reads only this object: the top-level `sfm.matcher` is ignored, and `sfm.max_image_size` sets only the training resolution (`SfmRequest.from_api_payload` in `sfm_backends/configs.py`).

| Key | Type | Default | Description |
| - | - | - | - |
| `matcher` | `string` | By input | `exhaustive` or `sequential`. Unset, photos and video up to 200 frames get `exhaustive`, longer video `sequential`. `exhaustive` is for `standard` and above, as with `sfm.matcher` |
| `max_image_size` | `integer` | `1600` | Longest edge, in pixels, for feature extraction: `400`-`4096` |

The backend also accepts tuning keys for COLMAP and GLOMAP (`camera_model`, `max_num_features`, triangulation thresholds and others, in `ColmapSiftGlomapOptions`); they are unsupported and may change. An unknown key or bad value is refused when `/process` dispatches the job: `400 invalid_input` (`Invalid sfm options: ...`), and the charge is refunded (`translatePydantic422` in `api/src/lib/scenes.ts`).

## Camera poses you already have

There are two ways to supply poses. Only a COLMAP model skips SfM.

### Path A: ARKit poses via API (recommended for iOS)

Pass camera poses directly in the `POST /scenes/{id}/process` request body.
The API stores them server-side. No zip required — upload your video as
normal. GLOMAP still solves the camera poses from the frames (`_stage_sfm` in
`pipeline/training/impl.py`); the ARKit poses set which way is up and, for a
video, which frames are cut: up to 150, at the pose timestamps
(`_extract_arkit_timestamp_frames`).

```bash theme={null}
# 1. Create a scene (standard video upload)
curl -X POST https://api.splat-3d.com/v1/scenes \
  -H "Authorization: Bearer s3d_..." \
  -H "Content-Type: application/json" \
  -d '{"title": "My scan", "preset": "standard", "content_type": "video/mp4"}'

# Response includes sceneId + uploadUrl

# 2. Upload video directly to the presigned URL
curl -X PUT "$UPLOAD_URL" \
  --upload-file capture.mp4 \
  -H "Content-Type: video/mp4"

# 3. Process with ARKit poses. poses.json is {"arkit_poses": [...]}, holding
#    5 to 1,000 poses shaped like the one below.
curl -X POST https://api.splat-3d.com/v1/scenes/$SCENE_ID/process \
  -H "Authorization: Bearer s3d_..." \
  -H "Content-Type: application/json" \
  -d @poses.json
```

One pose:

```json theme={null}
{
  "timestamp": 0.033,
  "file_path": "frame_000001.jpg",
  "transform": [1,0,0,0, 0,1,0,0, 0,0,1,0, 0,0,0,1],
  "intrinsics": [1674,0,960, 0,1674,720, 0,0,1],
  "width": 1920,
  "height": 1440
}
```

**Pose format:**

| Field | Type | Description |
| - | - | - |
| `timestamp` | `number` | Capture time in seconds |
| `file_path` | `string` | Path in the zip of the image this pose corresponds to (the file name alone works when it is unique) |
| `transform` | `number[16]` | 4×4 camera-to-world matrix, **column-major** (matches ARKit `simd_float4x4` layout), in a gravity-aligned world with +Y up |
| `intrinsics` | `number[9]` | 3×3 intrinsics, **row-major**: `[fx, 0, cx, 0, fy, cy, 0, 0, 1]` |
| `width` | `integer` | Image width in pixels |
| `height` | `integer` | Image height in pixels |

Minimum 5 poses, maximum 1000. You don't set `sfm.backend`: the API sends
the poses with the job, and the pipeline runs GLOMAP as for any upload.

**The scene's orientation comes from these poses.** The pipeline reads gravity
off the world's +Y axis, so record them in an ARKit session with
`worldAlignment` set to `.gravity` (the default) or `.gravityAndHeading`, not
`.camera`. It needs at least 30 frames or photos paired with a pose and a
median tilt residual (each pair's up against the fitted up) of 3° or less;
otherwise it estimates up from the cameras, which lays portrait captures on
their side (`MIN_PAIRS`, `MAX_MEDIAN_TILT_DEG` in
`pipeline/training/arkit_gravity.py`). Photos are used as stored: EXIF
orientation is ignored when `arkit_poses` is set, so each image's pixels must
be in the camera axes its `transform` describes, as ARKit's `capturedImage` is.

If you're building an iOS app, the [Splat3D Swift SDK](https://docs.splat-3d.com/sdks/overview.md) handles
all of this for you — ARKit capture, pose encoding, upload, and polling in one call.

***

### Path B: COLMAP zip upload

If you've already run COLMAP (or Metashape, RealityCapture, or your own SfM
code that exports COLMAP binaries), bundle your images and sparse model into a
zip. The pipeline detects the `sparse/` model, skips SfM entirely, and starts
training immediately. This is the fastest path to a splat and gives you full
control over the reconstruction.

**Zip structure:**

```
your-upload.zip
├── images/
│   ├── frame_001.jpg
│   ├── frame_002.jpg
│   └── ...
└── sparse/
    └── 0/
        ├── cameras.bin
        ├── images.bin
        └── points3D.bin
```

A single wrapper folder is fine (`my-scene/images/` + `my-scene/sparse/0/`) —
zip the folder or its contents, whichever your tool produces. If several models
are present, `sparse/0` is used; among several `sparse/0` folders, the one with
`images/` beside its `sparse/`.

Two tool outputs also work zipped as they are:

* **Inria 3DGS `convert.py`** — the undistorted `sparse/0` and `images/` are
  used; `input/` and `distorted/` are ignored.
* **nerfstudio `ns-process-data`** — the model in `colmap/sparse/0` is paired
  with the photos in the top-level `images/`; `images_2/` and the other
  downscaled copies are ignored.

```bash theme={null}
# 1. Create a scene — content_type marks it as a photo/zip upload
curl -X POST https://api.splat-3d.com/v1/scenes \
  -H "Authorization: Bearer s3d_..." \
  -H "Content-Type: application/json" \
  -d '{
    "title": "My scene",
    "preset": "standard",
    "content_type": "application/zip"
  }'

# 2. Upload zip to presigned URL
curl -X PUT "$UPLOAD_URL" \
  --upload-file my-scene.zip \
  -H "Content-Type: application/zip"

# 3. Trigger processing (no body needed — the model is detected in the zip)
curl -X POST https://api.splat-3d.com/v1/scenes/$SCENE_ID/process \
  -H "Authorization: Bearer s3d_..."
```

You do **not** need to set `sfm.backend` — detection is automatic. Uploading a
zip with no `sparse/` directory runs the normal GLOMAP path.

#### Requirements

Your model is validated before training starts, so a mismatch fails fast with a
message instead of burning GPU minutes:

| Requirement | Detail |
| - | - |
| Format | COLMAP binary (`.bin`) or text (`.txt`). A text model is converted to binaries before validation; binaries win when a folder has both. |
| Completeness | All three of `cameras.bin`, `images.bin`, `points3D.bin`, or of `cameras.txt`, `images.txt`, `points3D.txt`. An incomplete set of text files is ignored, and the photos go through SfM. |
| Filenames | Every name in `images.bin` must exist under `images/`, matched as a full relative path — `left/frame_001.jpg` in the model needs `images/left/frame_001.jpg` in the zip, not a flattened `frame_001.jpg`. |
| Camera models | `SIMPLE_PINHOLE`, `PINHOLE`, `SIMPLE_RADIAL`, `RADIAL`, `OPENCV`. Fisheye and other models are rejected — undistort to `PINHOLE` first with `colmap image_undistorter`, then zip its output folder: the undistorted `images/` together with its `sparse/` model. The original photos do not match an undistorted model. |
| Photo size | Each photo must be the width × height its camera has in `cameras.bin`, or a uniform resize of it. Photos are read as stored, without applying EXIF rotation, so zip the exact files the model was built from. |
| Registered images | Between 5 and 800. We cannot subsample a model for you — dropping an image would orphan its pose — so trim it yourself before uploading. |
| Sparse points | Between 1,000 and 5,000,000 in `points3D.bin`. Training starts from one Gaussian per point, with no random initialization, so a poses-only export is rejected — triangulate points first with `colmap point_triangulator`, which needs a `--database_path` holding features and matches for the same photos (`colmap feature_extractor`, then a matcher such as `colmap exhaustive_matcher`). Thin a denser cloud with `colmap point_filtering`. |

Photos in the zip that the model never registered are ignored.

**Getting COLMAP binary output:**

```bash theme={null}
# Run COLMAP and export binary sparse reconstruction
colmap automatic_reconstructor \
  --workspace_path ./colmap_output \
  --image_path ./images

# Binary files are written to colmap_output/sparse/0/ by default
```

***

### When to use pre-supplied poses

| Scenario | Recommended path |
| - | - |
| iOS app with LiDAR (iPhone 12 Pro+) | Path A — ARKit poses via API |
| Existing COLMAP reconstruction | Path B — COLMAP zip |
| Custom SLAM / visual odometry | Path B — convert to COLMAP binary format |
| Re-training with different parameters | Path B — reuse existing sparse reconstruction |

## Matching strategies

Feature matching only happens when SfM actually runs, so a COLMAP model zip (`prebuilt_colmap`) ignores these settings. Otherwise you can control the matching strategy in `sfm.options`:

| Matcher | Description | Speed | Best for |
| - | - | - | - |
| `exhaustive` | Compare every image pair | Slowest | Small-medium scenes (under 200 images) |
| `sequential` | Compare adjacent frames only | Fastest | Ordered video frames |

```json theme={null}
{
  "sfm": {
    "options": {
      "matcher": "sequential"
    }
  }
}
```

> [!TIP]
>
> Video of up to 200 frames gets `exhaustive` when you set none: on the videos we tested it registered as many frames as `sequential` or more. Longer video gets `sequential`, which stays fast as frames grow. Photo sets get `exhaustive`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.