> ## Documentation Index
> Fetch the complete documentation index at: https://docs.splat-3d.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Masking

> Click one object in one frame of a video, or one photo of a zip, and get a scene that holds that object alone.

# Masking

Send a `subject` when you create a scene from a video or a zip of photos, and
the scene keeps one object instead of everything the camera saw. You name the
object by clicking it in one frame or one photo. The pipeline finds it in
every frame or photo and trains only it.

Masking was first published as "Object scan"; the request and its rules are
unchanged.

## The request

`subject` is an optional top-level field of `POST /v1/scenes`. It names one
picture of the upload and the clicks on it. A video names the picture by its
moment:

```json theme={null}
{
  "content_type": "video/mp4",
  "preset": "standard",
  "subject": { "time_s": 12.4, "points": [[0.52, 0.61], [0.50, 0.70]] }
}
```

A zip of photos names it by the photo's entry name:

```json theme={null}
{
  "content_type": "application/zip",
  "preset": "standard",
  "subject": { "image": "trip/IMG_0042.JPG", "points": [[0.52, 0.61]] }
}
```

| Field | Rule |
| - | - |
| `time_s` | Video uploads. Seconds into the video of the frame you clicked, as a browser's `video.currentTime` reports it. A number, 0 or more. |
| `image` | Photo zip uploads. The entry name of the photo you clicked, exactly as the zip lists it: folders included, same case. 1 to 1024 characters of valid Unicode, with no U+0000. |
| `points` | 1 to 5 clicks, each `[x, y]` from 0 to 1. Every click is on the same object. |

Send exactly one of `time_s` and `image`. One photo only: the pipeline
follows the object from it into the other photos.

On splat-3d.com, the upload page's picker shows a sample of up to 12 of the
zip's photos, spread evenly from first to last, not every photo.

`null` or no `subject` makes a normal scene. Nothing else in the request
changes, and the price is the preset's.

## Which upload takes which key

| Upload | `time_s` | `image` |
| - | - | - |
| Video | required | refused |
| Zip of photos, including an ARKit photo zip and a COLMAP workspace | refused | required |
| Zip that runs as a video (a SplatKing `wide.mov` export) | refused | refused |
| R3D | refused | refused |

## Coordinates and orientation

Coordinates are fractions of the picture **as a browser shows it**, measured
from the top-left corner. `[0.5, 0.5]` is the center.

* **Video:** rotation applied. A phone video recorded upright plays upright
  in a browser even when its pixels are stored sideways, and the clicks are
  read in the upright frame.
* **Photo:** EXIF orientation applied. A portrait photo stored as landscape
  pixels with an EXIF rotation shows upright in an `<img>`, and the clicks
  are read in the upright photo. If you read the pixels yourself, apply the
  EXIF `Orientation` tag before you measure.

To click a frame in a browser, pause a `<video>` at the moment you want and
divide the click's offset inside the picture by the picture's width and
height. Leave out any letterbox bars that `object-fit: contain` adds. For a
photo, do the same on an `<img>`.

### The photo's name

`image` must match one of the zip's JPEG or PNG entries character for
character. List the names with the tool that made the zip, e.g.
`unzip -Z1 photos.zip`. Entries the pipeline doesn't read as photos can't be
named: other formats, empty files, and hidden or `__MACOSX` files.

Keep entry names to ASCII if you can. A name stored without the zip's UTF-8
flag is read as code page 437, as Python's `zipfile` reads it, so `día.jpg`
may be listed as `día.jpg`.

## Rules

* **The right key for the upload.** The other key, or any `subject` on an
  R3D upload, answers `422 invalid_input`, with `{"field": "subject"}` in
  `details`.
* **Malformed** values answer `400 invalid_input`, one `{field, message}`
  per problem, e.g. `{"field": "subject.points.0.0", ...}` for an `x` past 1,
  or `{"field": "subject", ...}` for both or neither of `time_s` and `image`.
  Unknown keys inside `subject` are refused.
* **Inside the video.** `/process` and `/retrain` refuse a `time_s` past the
  video's end with `422 source_unusable`, field `subject_time_s`, before
  anything is charged. A video whose length can't be read launches.
* **In the zip.** `/process` and `/retrain` refuse an `image` that isn't one
  of the zip's JPEG or PNG entries with `422 source_unusable`, field
  `subject_image`, value `not_in_zip`, before anything is charged. A zip that
  runs as a video gets the same field with value `zip_runs_as_video`. A zip
  whose file list can't be read (ZIP64, a list over 1 MiB) launches, and the
  pipeline looks for the photo.
* **Kept on the scene.** A retrain of a masked scene is masked too,
  with the same picture and clicks.

## Progress

The pipeline estimates the camera poses from the whole picture first, because
the background is what places the camera. It then finds the object in every
frame or photo and trains only it. While it finds the object,
`GET /v1/scenes/{id}` reads `processing_stage: "segmenting"`; the list and
`PATCH` show `training`, as does `status`.

## Failures

These end the scene as `failed`, with `processing_error` set, and refund it as
any failed scene is refunded. A retrain keeps the subject and the upload, so
`/retrain` refuses the first three with `409 retry_blocked`; create a new
scene instead:

| `processing_error` | What to try |
| - | - |
| We couldn't find an object at the point you picked. Try clicking the middle of it. | Click the middle of the object, or add clicks on its other parts. |
| The object you picked fills the whole frame. Try a smaller object, or scan without picking one. | Pick a smaller object, or leave out `subject`. |
| We couldn't find the photo you picked in your upload. Pick a photo again. | Send `image` as one of the zip's JPEG or PNG entry names. Only a zip whose file list couldn't be read before launch gets this far. |
| Masking needs the Spirula trainer, which is switched off right now. | Try again later, or leave out `subject`. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.