A Physical AI team that buys or collects data needs more from its supplier than fast, verified transport. It needs a durable connection between the bytes that arrived and the experiment about to run: the recordings, the sensor context required to interpret them, the checks that establish their quality, the rights that govern their use, and a receipt showing that all of it landed intact where the training job will read it. We call the unit that carries all five a verified dataset release. This post sets out what one contains, how we move it, and how to test one.
We built this standard the way most infrastructure gets built: by running deliveries by hand until the handoff was the bottleneck. Our team has been shipping versioned datasets into customer storage for fleet learning programs, and every dispute, re-delivery, and loader failure we worked through came back to the same gaps: a manifest that was not frozen, rights that lived in a contract nobody could query, a transfer that finished without proof it arrived, and a training run nobody could trace to an exact release. So we built the release record, the signed receipt, and the delivery ledger described below, and every handoff now runs through them. We have been slower to write about it than to build it. This post is the write-up: what travels with the bytes, which is where the training team's problems actually live. Measured throughput and per-terabyte cost figures for the pipeline will follow in a benchmark report with disclosed conditions.
Six questions a release has to answer
When a handoff has to preserve provenance, rights, validation evidence, and destination integrity, it needs more structure than a file copy. A training team needs to know exactly what arrived, whether it can load it, and how that input relates to the model behavior it wants to change. A buyer should be able to answer six questions from the release record alone, without a call:
- Which approved collection requirements does this release satisfy?
- Which recordings, modalities, labels, and calibration versions are included?
- What passed, what failed, and which measurements are estimated or unavailable?
- What may we use, where may we store it, and which rights restrictions apply?
- Did the complete release arrive intact in our environment?
- Which exact release did this training or evaluation run consume?
Answering those six questions in the record reduces the engineering a buyer spends ingesting a delivery and narrows acceptance review to objective coverage and quality checks. The rest of this post walks the loop that produces such a release, in the order the data moves.
Five handoffs, one record: the approved brief, the captured and validated recordings, the sealed release, the destination and its receipt, and the customer's training and evaluation run, whose results define the next brief. The customer trains and ships the model; the record keeps the data that trained it traceable end to end.
Start from the brief, not the hours
Consider a robot that struggles to load unfamiliar dishwashers. A useful collection brief does not ask for "dishwasher hours." It picks the coverage cells the model is missing: which appliance mechanisms, which loads and objects, which starting states, which viewpoints, and which actions and outcomes. It fixes the capture configuration and the required outputs, and it states the acceptance checks that will be applied before an hour counts.
In our fleet learning program the brief is the specification for a capture campaign, approved by the customer before capture starts and governed by the statement of work. Cells come from a catalog of 175 workflows in 26 families, and every program tracks six diversity axes separately (places and layouts, appliances and mechanisms, objects and loads, natural starting states, people and viewpoints, and actions and outcomes) so a gap on one axis cannot hide behind volume on another. Each program caps the share of accepted hours from any one home and any one collector, with the caps set in the brief. Exact-file checks and video fingerprints look for duplicate and near-duplicate sequences within a delivery and against earlier deliveries, and flagged matches are excluded from accepted hours. The coverage page describes the standard in full.
That brief gives the release its purpose. The coverage report that ships with every delivery compares what arrived against the cells the brief asked for, flags any home, collector, appliance, or action that is overrepresented, and reports hours and sequences per home and collector, so the accepted-hour tally can be checked against the delivered sequences rather than trusted from a spreadsheet.
Keep the sensor context attached
A robotics recording is not a video file. It can carry multi-camera video, depth, LiDAR, IMU and motion, calibration, hand and body poses, language, and annotations, all of which need consistent time and coordinate conventions to be usable together. We ingest MCAP and RRD natively, import LeRobot datasets and ROS bags through the SDK (each LeRobot episode becomes one MCAP sequence), and connect other formats through adapters. Whatever the source, each sequence in a fleet learning release carries one schema: outcome labels, subtask boundaries, quality and speed grades, mistake and recovery segments, rig and calibration identifiers, pseudonymous home and collector identifiers, consent references, and leakage groups by home, object, or operator so that train and test splits do not share a kitchen.
Validity has to stay visible after the data leaves the pipeline that produced it. A release identifies its required fields and preserves the distinction between measured, estimated, and missing values. The presence of a pose file does not establish its accuracy. A checksum establishes that bytes survived transfer; it cannot establish that the calibration was correct. So preflight validation checks each sequence against the campaign specification before a release can move: media decoding, object completeness, schema conformance, timestamp monotonicity and clock alignment across streams, non-placeholder calibration parameters, units and coordinate conventions, and the campaign-specific checks the brief calls for, such as pose scale and drift, delivered field of view, hand visibility, and privacy redaction. The structural subset (indexing, sequence counts, frame counts, and calibration presence) is queryable self-serve as dataset health through the CLI, the Python and TypeScript SDKs, the REST API, and the MCP server's get_dataset_health tool.
Technical validity and behavioral outcome are graded separately and stored separately. A correctly decoded video can still depict a failed task. A successful demonstration can still have unusable geometry. In our reviews, mixing the two grades is a common reason a dataset passes inspection and still trains poorly, so training teams get both dimensions and filter on each.
Seal the release
A release needs an identity that survives renames, reorganizations, and corrections, and a manifest that cannot drift after the fact. The model is the one container registries use, a movable tag beside an immutable digest, applied to a dataset release and extended with rights and origin records. Each release carries two references. A readable reference is for discovery, and its alias advances as new revisions publish. A canonical reference, a dataset UID plus a content digest, is the machine identity, and it stays fixed even if the dataset or the organization is renamed. The rule we give training teams: persist the canonical reference in the training config, not a movable alias.
The manifest behind that reference seals logical paths, roles, byte sizes, source origins, and SHA-256 hashes for every object. A rights declaration records the license terms and the attestation hashes in force at release; it documents what was declared, and the warranties live in the governing agreement. Release eligibility is based on explicit rights records rather than storage metadata. Lineage records explicit parent relationships between revisions, and a revision's content identity cannot be rewritten after creation. Corrections produce a new revision with a stated relationship to the one it replaces, never a silent overwrite. The release record a buyer sees lists object count, total bytes, paths, roles, and hashes, and shows public origin identities while excluding private locators and credentials.
A hash manifest is evidence only if the reader can trust the manifest, so the manifest and the rights declaration are signed, and the signer identity and signing time are part of the record. This is the problem in-toto, SLSA, and Sigstore solved for software supply chains, and the release record borrows their shape rather than reinventing it: an immutable artifact identity, a signed statement about how it was produced, and a verification path that does not require trusting our database. The human-readable form of the record follows the practice set by Datasheets for Datasets and Data Cards, and the machine-readable form is kept simple enough to export.
The sample release record on our datasets page, shown for scale: 83,440 objects totaling 1.8 TB across three linked revisions and two declared origins. It illustrates the record format with sample digests; it is not a customer delivery.
The loader closes the loop on the consumer side. avala.load resolves the canonical reference, fetches objects lazily, and checks each one's declared size and SHA-256 when it stages the object to local cache, so a truncated download or an altered object fails immediately with an explicit checksum error instead of feeding corrupted frames into a run or crashing a dataloader worker hours after launch. Verified objects are not re-hashed on later epochs. Availability checks fail closed: a revision the caller is not eligible to read does not resolve. A public resolver that lets eligible open-download revisions load without an API key ships next.
The release also travels with its own evidence. The manifest, the rights declaration, the coverage report, and the receipt are exported alongside the data as a self-contained archive, so the customer can verify a release in its own storage after the engagement ends and without calling our API. Release and rights records are kept in an audited system that customers can query; customer source data remains the customer's to keep.
Move the bytes without losing the evidence
Delivery separates two concerns. The control plane decides what is authorized, tracks work, and records evidence. The byte path moves approved objects through the selected route. Keeping them apart is what lets a release be re-delivered to a second destination, or a corrected revision be delivered as a delta, without re-deciding what the customer is allowed to receive.
Customers work in the storage and training environment they already approved. Under BYOS, source data stays in the customer's own bucket and region, with reference connectors for Amazon S3 and Google Cloud Storage, Azure Blob Storage through our integrations, a Snowflake connector for warehouse data, and forward-deployed engineering for destinations we do not list. Teams without a strong cloud preference can use Avala Cloud, which stores on Cloudflare R2 and runs processing on Cloudflare compute. R2 charges no egress bandwidth fees, so repeated downloads do not add an R2 bandwidth bill; storage and operations remain billable, Infrequent Access adds retrieval fees, and the customer's own network and compute charges remain.
A delivery job has to survive production networks. It keeps a durable ledger of objects and parts, runs with bounded concurrency, retries with backoff, resumes after interruption, renews expired access, supports cancellation, routes unrecoverable items to a dead-letter path, and aborts abandoned multipart uploads. Completion is idempotent, so a repeated message or a worker restart may redo work but does not record a second delivery, quantity, or charge. That property is required on any at-least-once queue: Cloudflare Queues, for example, documents at-least-once delivery and recommends a unique identifier per message as an idempotency key, and our ledger follows the same rule regardless of which queue carries the message.
Chunk sizes and concurrency are set from tests on each account and provider, because the limits are real and they differ by hop. Cloudflare's Workers limits cap inbound request bodies by Cloudflare account plan (100 MB on Free and Pro, 200 MB on Business, 5 GB self-serve on Enterprise), give each isolate 128 MB of memory shared across concurrent requests, and stop queue consumers at 15 minutes of wall time, so any Worker-based path has to stream and size parts for the hop it is on. Hugging Face positions Storage Buckets as mutable working storage with chunk-level deduplication, US or EU residency at creation, and encryption at rest, and its S3 gateway documents its differences plainly: path-style addressing, a checksum setting for recent AWS clients, no custom object metadata or object-level policies, conditional headers honored on writes but not reads, and no UploadPartCopy. None of that makes Buckets a bad destination. It means each adapter ships with a conformance test, because a feature a provider ignores is a feature our receipt has to supply another way.
Direct transfer from source to destination is the default for customer-owned source data. Staging and content-addressed reuse are options we benchmark against the actual object mix and routes, and chunk-level deduplication is a strong candidate for exact re-deliveries. We do not confuse byte reuse with diversity: deduplication savings say nothing about scene or task uniqueness, and a re-encoded or privacy-scrubbed video may share almost no chunks with its original.
Verify at the destination, in three states
A drained queue is a progress signal, not proof of delivery. Delivery completes when the intended manifest has been reconciled against the destination: object identities, sizes, and checksums confirmed, and anything missing, unexpected, or mismatched reported, together with the destination, the release revision, the verification method, and the completion time.
The checksum detail matters. An S3 ETag is not a reliable whole-file checksum: for multipart uploads it is a digest of per-part digests with a part-count suffix that changes with part size, and for some encryption modes it is not an MD5 at all. On Amazon S3, full-object checksums for multipart uploads exist only for the CRC family, so the receipt validates a full-object CRC64NVME computed in the same source read that produced the manifest's SHA-256, records both values, and states what each proves: the CRC guards against corruption in transit, and the SHA-256 is what binds the object to the sealed manifest. The mechanics are in AWS's Checking object integrity. Where a destination cannot validate integrity headers on write, verification runs through an in-region worker or a customer-approved read-back pass against the manifest's SHA-256 digests, because cross-cloud read-back costs the customer egress and halves effective throughput.
The receipt itself is signed. It binds the manifest digest, the destination identity, the per-object results, the verification method, the completion time, and the verifier identity, so a customer or an auditor can authenticate it later without trusting our database. We keep three events distinct: transport complete, when the expected bytes are verified at the destination; validation complete, when the technical and coverage checks pass; and customer acceptance, when the review defined in the statement of work is satisfied.
Replacements and accepted quantities connect to billing; a transport receipt on its own does not. The handoff test we run is a loader smoke test from the customer's environment: a fresh machine with the release, its canonical reference, and scoped credentials loads the expected sequences, interprets their sensor context, and reproduces the accepted-hour tally, and a missing calibration file or an altered object fails loudly rather than passing silently.
Isolation and lawful reuse
Two kinds of data move through this loop, and they follow different rules. Data supplied by a partner or commissioned by a customer, meaning that customer's footage, briefs, model outputs, annotations, evaluation sets, and exclusive assets, is delivered only to that customer and is never included in another buyer's delivery; any use of it to improve Avala's own labeling models is limited to the terms in the governing agreement. Recordings that Avala commissions itself under its own nonexclusive license are offered to more than one buyer, and each such release says so in its rights declaration and order form. Acquisition cost can be amortized across those buyers; buyer-specific filtering, processing, delivery, support, and rights checks are incremental every time. Neither "everything is pooled" nor "nothing is ever reused" describes both models, so each release states which one applies.
Each buyer gets its own authorized manifest and its own access grants. A storage prefix is an organizational convention, not evidence of isolation, so we test on every release that one buyer's credentials cannot list or retrieve another buyer's objects. Temporary signed links are issued only for objects in the buyer's authorized manifest, expire quickly, and are redacted from logs. Under standard BYOS, source data stays in the customer's bucket and region, and Avala-managed infrastructure and API metadata are hosted in the United States under the transfer safeguards in our privacy policy and data processing terms. Material with unresolved rights or consent evidence is held back from delivery pending review.
Immutable revisions and the right to withdraw consent coexist by design. When a participant withdraws consent, we publish a new revision without the affected sequences, mark the earlier revisions as withdrawn so the resolver stops serving them, and notify each customer holding a copy with the object list to delete. The lineage record keeps the history of what happened without keeping the data.
Close the loop into the next brief
A failed loader check, a coverage gap, and a model evaluation result describe three different problems, and each calls for a different response: repair the export, collect the missing condition, or investigate the model's behavior. Rejection reasons, missing coverage cells, loader failures, and any evaluation feedback the customer authorizes us to see are stored against the release and the approved brief, and they are the raw material for the next capture campaign. Each cycle is recorded as one capability experiment linking the model version, the failure condition, the hypothesis, the coverage cells, the capture campaign, the dataset release, the held-out evaluation cohort, the outcome, and the next allocation. Held-out cohorts are grouped by home, object, and operator and refreshed over time so repeated evaluation does not quietly tune to them. Training and the ship decision belong to the customer, on the customer's code and compute, and a successful transfer is never reported as model improvement.
When this is overkill
For a few terabytes from one supplier to one destination, a versioned bucket, a manifest, a checksum file, and a README are enough, and we say so to teams that ask. The release record earns its place when any of five things is true: more than one supplier or more than one destination, rights that differ across sequences, revisions that are expected rather than exceptional, a model that will be retrained on later releases of the same data, or a customer or regulator who will ask what trained the model. Most fleet learning programs hit all five within the first quarter. For teams building systems under Article 10 of the EU AI Act, the record supplies evidence on data origin, collection purpose, preparation steps, and coverage against the brief; the compliance obligation stays with the system's provider.
A pilot test you can run
A release standard is only worth adopting if it fails the right way, so here is the test we ask pilots to run against a sample release:
- Alter one object in the destination after delivery. The loader must refuse it with a checksum error, and a re-run of reconciliation must list it as mismatched.
- Delete one calibration file. Dataset health and the loader must both fail on the affected sequences rather than loading them without geometry.
- Replay a completed delivery message. No second delivery, accepted quantity, or charge may appear.
- Interrupt a multipart upload and expire the destination credential mid-job. The job must resume, renew access, and finish with a complete receipt.
- Deny writes to one destination while another proceeds. The healthy destination must complete, and the blocked one must report its state without blocking the other.
- Load the release from a fresh machine with only the canonical reference and scoped credentials. The sequence count and accepted-hour tally must match the record.
Each outcome is observable in the receipt and the record, which is the point. Verification you cannot reproduce is a claim, not evidence.
Sizing delivery capacity
The right way to size delivery capacity is from measured bytes per accepted hour for each rig and output configuration, because mono video, stereo, depth, poses, and annotations have very different footprints. For one buyer, destination bytes equal accepted hours delivered multiplied by delivered gigabytes per hour. Across buyers, sum the actual authorized deliveries, since buyers may receive different slices and outputs.
As an illustration only, using decimal units and round numbers rather than a measured Avala workload: 400,000 unique hours a month at 8 GB per delivered hour, a rough footprint for a few compressed camera streams plus poses and metadata, with two buyers each taking the full corpus, writes 6.4 PB a month to destinations. Spread over 30 days that is 8.9 TB per hour continuously, and at 75 percent usable utilization it means provisioning about 11.9 TB per hour before any headroom policy. Shared source storage or staging deduplication does not remove the two destination copies. The cost model accounts for collection once per unique recording and then adds, per buyer, filtering, privacy and format transforms, source reads and egress, staging, requests, compute, destination operations, verification, retries, and support. Low-overhead transfer lowers the cost of every additional delivery; it does not change what the first collection costs, and the two numbers worth tracking are dollars per destination-verified terabyte and dollars per customer-accepted hour.
What you can use today
Managed dataset operations run end to end for customers today: ingest, validation, labeling, quality control, versioned deliveries with sealed manifests, and coverage reports, with collection briefs, gap reports, and evaluation reports delivered through the fleet learning program. Self-serve, you can ingest MCAP and RRD, import LeRobot datasets and ROS bags, export LeRobot v3, check dataset health, create and track exports, and list slices through the CLI (Python 3.12 or later for the CLI and LeRobot extras), the Python and TypeScript SDKs, the REST API, and the hosted MCP server at mcp.avala.ai. Releases resolve by canonical reference and verify per object through avala.load, destinations receive a signed reconciliation receipt, and BYOS keeps source data in your S3 or Google Cloud Storage bucket, or in Avala Cloud on R2. Two pieces ship next: the public resolver for open-download revisions without an API key, and destination selection with delivery tracking in the portal, which today runs through the API and your Avala team.
Where this fits
Verified releases are the delivery end of the argument in Sovereign Physical AI: own the data-to-model loop, rent the rest: the record a model can trace back to, the loop that writes to it, and the test of whether a customer can leave with everything. The per-layer economics of what goes into a release are in our buyer's guide to the robotics data pyramid, and the reason world models and action-conditioned policies move the bottleneck from model architecture to physical ground truth is in From Imitation to Imagination. The SDK methods, MCP tools, and sequence schema named above are documented on the developers page.
Bring us one dataset handoff or one collection objective and we will run the pilot test above against it.
