Structured serialization

From the repository

This page is docs/structured-clone.md, included when the site is built. Edit it there.

An IDL any value crosses the C API as its structured serialization (HTML §2.7, StructuredSerializeInternal), in the byte format below: history.state, pushState’s data, a PerformanceMark’s detail, what IndexedDB stores, and the messages that postMessage sends between event loops (to a worker, a port, a BroadcastChannel or the window). The format is realm-independent and has no pointers, so it also carries values between threads.

A JS value can only be walked by the engine that holds it, so the JS binding’s glue (SC in crates/niso-structured-clone/glue/structured-clone.js) writes and reads the format. The core keeps serialized values as bytes and decodes them where it needs to look inside one (crates/niso-structured-clone): IndexedDB evaluates key paths on them and injects generated keys into them.

Reading it from native code

C, and every language on the C API, decodes a serialization with niso_structured_clone_decode (niso_runtime.h) into a graph it walks: niso_structured_clone_value is the value, and each object a value refers to has a number and entries, read with niso_structured_clone_entry. Every reference to one object has the same number, so shared objects and cycles can be rebuilt. Strings come out as UTF-8, a JS string’s lone surrogates as U+FFFD. Decoding needs no DOM.

kind	what the fields hold
UNDEFINED, NULL	nothing
BOOLEAN, NUMBER	number (0 or 1 for a boolean)
BIGINT	bytes: the magnitude, least significant byte first; negative
STRING	string
BOOLEAN_OBJECT, NUMBER_OBJECT, BIGINT_OBJECT, STRING_OBJECT	as the primitive
DATE	number: ms since the epoch
REGEXP	string: the source; entry "flags"
ARRAY_BUFFER	bytes; number: a resizable buffer’s maximum byte length, else -1
VIEW	string: its type ("Uint8Array", "DataView", …); bytes: the bytes it views; entry "buffer"
MAP	entries: each key and value
SET	entries: each value (keys are undefined)
ERROR	string: its name ("TypeError", …); entries "message", "stack" and "cause", those it has
ARRAY	number: its length; entries: its properties ("0", "1", …)
OBJECT	entries: its properties, in order
PLATFORM	string: its interface ("DOMException", "DOMRect", "ImageData"); entry "data"
TRANSFERRED	number: its index among the message’s ports

A MessageEvent’s data is the binding’s own value, so native code reads niso_message_event_serialized_data and decodes it, or, for data the platform makes itself (a WebSocket’s text and binary frames, which arrive with no serialization), niso_message_event_native_data: the string or ArrayBuffer itself. A message that arrived serialized has no native data, so a string with lone surrogates keeps them. The JVM binding has both as MessageEvent.serializedData() and nativeData().

Format

Integers are little-endian. A serialization is a version byte (1) and one value. Every record that is an object (tags from 0x10 on) is numbered in the order StructuredSerializeInternal adds it to its memory, starting at 0, and a back-reference (0x07) names an earlier one by that number. That is the order records start in the bytes, except that an ArrayBufferView’s buffer is numbered before the view, as the algorithm serializes the buffer first. StructuredDeserialize creates objects in the same order, so it rebuilds the same graph.

value  = 0x00                          undefined
       | 0x01                          null
       | 0x02 | 0x03                   false, true
       | 0x04 f64                      number
       | 0x05 bigint                   BigInt
       | 0x06 string                   string
       | 0x07 u32                      back-reference
       | 0x10 u8                       Boolean object
       | 0x11 f64                      Number object
       | 0x12 bigint                   BigInt object
       | 0x13 string                   String object
       | 0x14 f64                      Date ([[DateValue]])
       | 0x15 string string            RegExp (source, flags)
       | 0x16 u32 bytes                ArrayBuffer (byte length, data)
       | 0x17 u32 u32 bytes            resizable ArrayBuffer (length, max length, data)
       | 0x18 u8 value u32 u32 u32     ArrayBufferView: kind (VIEW_KINDS), its buffer (an
                                       ArrayBuffer or transferred record, or a back-reference
                                       to one), byte offset, byte length, array length; a
                                       length of 0xFFFFFFFF is "auto" (length-tracking)
       | 0x19 (value value)* 0xFF      Map entries
       | 0x1A value* 0xFF              Set entries
       | 0x1B u8 opt-string opt-string opt-value
                                       Error: name (ERROR_NAMES), message, stack, and the
                                       value of an own data property cause
       | 0x1C u32 (string value)* 0xFF Array: length, then its properties
       | 0x1D (string value)* 0xFF     Object properties
       | 0x1E string value             platform object: its interface, then the data
                                       of its serialization steps as one value
       | 0x1F u32                      a transferred object: its index in the transfer list
bigint = u8 sign (1: negative), u32 n, n bytes of magnitude, least significant first
string = u8 encoding, u32 n, n code units: Latin-1 bytes (encoding 0, when every
         code unit is below 0x100) or UTF-16LE (encoding 1); lone surrogates survive
opt-string = 0x00 | 0x01 string
opt-value = 0x00 | 0x01 value

Records that hold other values (0x18 to 0x1E) nest at most 256 deep (MAX_DEPTH); a deeper serialization does not decode, in the core or the glue, as both decoders recurse.

Builds before this limit wrote serializations of any depth, and IndexedDB keeps values as they were written. A stored value that nests deeper than 256 no longer decodes: reading it (a request’s result, a cursor’s value) throws a DataCloneError, and anything the core does that evaluates a key path over it, such as creating an index on its store, fails. Its key still works, so it can be deleted or overwritten.

The serializable platform objects (0x1E) are the ones the glue knows: DOMException (data: [name, message, stack]), the value interfaces DOMRect and ImageData (data: their fields), Blob (data: [bytes as a Uint8Array, type]), File (data: [bytes, type, name, lastModified]) and CryptoKey (data: [type, extractable, algorithm, usages, key material as a Uint8Array]). The glue checks a Blob’s, File’s or CryptoKey’s data when it deserializes one, and throws DataCloneError for data its serialization does not write: a CryptoKey’s key type, usages, algorithm and material must be ones a key of its algorithm has. A realm that is not a secure context cannot deserialize a CryptoKey.

Transferred records (0x1F) are what a transfer list moves rather than copies. A message carries its MessagePorts beside the bytes: a transferred record is the index of one of the message’s ports, which are the MessageEvent’s ports once it arrives. ArrayBuffers in a message are serialized as copies (0x16) and detached once it is sent, since it may go to another thread. In structuredClone, which stays in one realm, a transferred record is the index in the transfer list, of an ArrayBuffer or a MessagePort alike.