Skip to main content

Masking

Send a subject when you create a scene from a video or a zip of photos, and the scene keeps one object instead of everything the camera saw. You name the object by clicking it in one frame or one photo. The pipeline finds it in every frame or photo and trains only it. Masking was first published as “Object scan”; the request and its rules are unchanged.

The request

subject is an optional top-level field of POST /v1/scenes. It names one picture of the upload and the clicks on it. A video names the picture by its moment:
A zip of photos names it by the photo’s entry name:
Send exactly one of time_s and image. One photo only: the pipeline follows the object from it into the other photos. On splat-3d.com, the upload page’s picker shows a sample of up to 12 of the zip’s photos, spread evenly from first to last, not every photo. null or no subject makes a normal scene. Nothing else in the request changes, and the price is the preset’s.

Which upload takes which key

Coordinates and orientation

Coordinates are fractions of the picture as a browser shows it, measured from the top-left corner. [0.5, 0.5] is the center.
  • Video: rotation applied. A phone video recorded upright plays upright in a browser even when its pixels are stored sideways, and the clicks are read in the upright frame.
  • Photo: EXIF orientation applied. A portrait photo stored as landscape pixels with an EXIF rotation shows upright in an <img>, and the clicks are read in the upright photo. If you read the pixels yourself, apply the EXIF Orientation tag before you measure.
To click a frame in a browser, pause a <video> at the moment you want and divide the click’s offset inside the picture by the picture’s width and height. Leave out any letterbox bars that object-fit: contain adds. For a photo, do the same on an <img>.

The photo’s name

image must match one of the zip’s JPEG or PNG entries character for character. List the names with the tool that made the zip, e.g. unzip -Z1 photos.zip. Entries the pipeline doesn’t read as photos can’t be named: other formats, empty files, and hidden or __MACOSX files. Keep entry names to ASCII if you can. A name stored without the zip’s UTF-8 flag is read as code page 437, as Python’s zipfile reads it, so día.jpg may be listed as d├¡a.jpg.

Rules

  • The right key for the upload. The other key, or any subject on an R3D upload, answers 422 invalid_input, with {"field": "subject"} in details.
  • Malformed values answer 400 invalid_input, one {field, message} per problem, e.g. {"field": "subject.points.0.0", ...} for an x past 1, or {"field": "subject", ...} for both or neither of time_s and image. Unknown keys inside subject are refused.
  • Inside the video. /process and /retrain refuse a time_s past the video’s end with 422 source_unusable, field subject_time_s, before anything is charged. A video whose length can’t be read launches.
  • In the zip. /process and /retrain refuse an image that isn’t one of the zip’s JPEG or PNG entries with 422 source_unusable, field subject_image, value not_in_zip, before anything is charged. A zip that runs as a video gets the same field with value zip_runs_as_video. A zip whose file list can’t be read (ZIP64, a list over 1 MiB) launches, and the pipeline looks for the photo.
  • Kept on the scene. A retrain of a masked scene is masked too, with the same picture and clicks.

Progress

The pipeline estimates the camera poses from the whole picture first, because the background is what places the camera. It then finds the object in every frame or photo and trains only it. While it finds the object, GET /v1/scenes/{id} reads processing_stage: "segmenting"; the list and PATCH show training, as does status.

Failures

These end the scene as failed, with processing_error set, and refund it as any failed scene is refunded. A retrain keeps the subject and the upload, so /retrain refuses the first three with 409 retry_blocked; create a new scene instead: