Storage drivers

From the repository

This page is docs/storage.md, included when the site is built. Edit it there.

IndexedDB and localStorage keep their data through a storage driver: an interface at the level of databases, object stores, indexes and ordered records, which the core drives and a host can implement. niso ships three drivers, and a host that wants its data elsewhere (its own database, an encrypted file, the platform’s key-value store) plugs in its own. sessionStorage always uses the DOM’s own memory driver, so a session lasts as long as the DOM; a host’s driver never sees it.

The code is in crates/niso-core/src/storage: the Driver trait in mod.rs, the drivers in sqlite.rs, memory.rs and ffi.rs, and the key encoding in key.rs. IndexedDB is in crates/niso-core/src/indexeddb and Web Storage in crates/niso-core/src/api/storage.rs.

The drivers niso ships

• SQLite (sqlite.rs), the default: one SQLite file per database, at <dir>/<origin>/<endpoint>/<hash>.sqlite. The origin is percent-encoded, and the hash is the FNV-1a hash of the database name, which the file also stores, so names of any length fit a file name. A name whose hash another database’s file has goes in <hash>-1.sqlite or the next number free, and finding a name reads every file of its hash, as deleting a database leaves a gap. Writing transactions are SQLite write transactions in WAL mode, on the database’s connection; the "strict" durability hint commits with synchronous = FULL, the others with NORMAL. A read-only transaction is a deferred SQLite transaction on a connection of its own, so all its reads see one snapshot even when another process writes the file. A database has at most 8 of these readers at once and keeps up to 4 idle ones; a read-only transaction past 8 reads through the database’s own connection, consistent within the DOM but without a snapshot against other processes. A reader opens the file the database was opened on and never creates one, and every transaction first checks that the file is still that one, so once another DOM deletes the database (or deletes and makes it again), its transactions fail with an UnknownError.
• Memory (memory.rs): nothing persists. sessionStorage, tests and workers use it, and so does a DOM given an empty storage directory. It holds at most 256 MiB of records’ keys and values (MemoryDriver::with_quota picks another limit).
• C (ffi.rs): an adapter that calls a host’s niso_storage_driver vtable.

By default the SQLite driver keeps its files in $NISO_STORAGE_DIR when it is set, else in a directory for the app in the platform’s per-user application data directory (~/Library/Application Support/niso/<app> on Apple platforms; %APPDATA%\niso\<app> on Windows; $XDG_DATA_HOME/niso/<app> or ~/.local/share/niso/<app> elsewhere), else in the temporary directory. <app> is the executable’s name, lowercased, with anything but letters, digits and - made -: the default is keyed on the executable, so apps with different executables keep their files apart, and apps run by one executable share them. Every Android app runs the same executable and every JVM app runs as java, so their hosts set the directory and the app origin themselves: the iOS host sets Application Support/niso in the app’s container and its bundle id, the Android host Context.getFilesDir()/niso and its package name, and a JVM app calls Dom.setStorageDirectory and Dom.setAppOrigin. niso-run and the tests keep their data in temporary directories.

Before the default directory was per app it was niso/ itself. niso leaves data written there in place and does not move it: the files of every app that used the old default are mixed there by origin, so no app can tell which are its own. An app that needs them copies its origins’ directories into its new directory itself.

Origins

Data is keyed by the document’s origin, and a document with an opaque origin (a data: URL, say) has no storage: localStorage, sessionStorage and indexedDB throw a SecurityError. A native app’s document has no URL, which leaves it at about:blank; such a document has the app’s origin instead, as a browser’s initial about:blank document has its creator’s: https://<app>.app.niso for the app’s name as above, or the origin of the URL the host passes to niso_dom_set_app_origin (Rust: Dom::set_app_origin), which must have a tuple origin. The iOS host uses https://<bundle id>.app.niso and the Android host https://<package name>.app.niso. Only storage sees this origin; location.origin still reports the document’s own.

Choosing where data goes

A host chooses once, before script first uses storage; later calls fail with NISO_INVALID_STATE_ERROR, because data may already be in the old place.

	C (niso_runtime.h)	Rust (niso_core::Dom)
SQLite files under a directory	niso_dom_set_storage_directory(dom, path, len)	set_storage_directory(path)
Memory only	niso_dom_set_storage_directory(dom, "", 0)	set_storage_directory("")
The host’s own driver	niso_dom_set_storage_driver(dom, &driver)	set_storage_driver(Box::new(driver))

niso_dom_set_storage_driver copies the vtable, so it need not outlive the call. Every function but drop must be set, else it fails with NISO_TYPE_ERROR. The DOM owns userdata only once the call returns 0, and then calls drop once with it when it lets go of it; when the call fails (a missing function, or NISO_INVALID_STATE_ERROR because storage is in use), the host still owns userdata and frees it itself.

Workers

A dedicated or shared worker’s IndexedDB is in memory and its own: it does not see its owner’s databases, and its data goes when it closes. Its first indexedDB.open logs a console warning saying so. Like every memory driver, it holds at most 256 MiB of records’ keys and values; a write past that fails with a QuotaExceededError, as a full disk does with SQLite. Sharing an origin’s databases between threads needs one storage agent per origin in the process, which every thread’s requests go through, so that versionchange events, blocked upgrades and deletes reach every connection and no two drivers write one SQLite file; that agent is still to be built.

What a driver stores

A driver keeps, per origin and storage endpoint (0 IndexedDB, 1 localStorage), named databases. A database has a version and object stores; an object store has a name, an optional key path, a key generator flag and number, its records, and indexes; an index has a name, a key path, unique and multiEntry flags, and entries. Names are UTF-16 code units, which a driver stores as opaque bytes.

A driver never sees a JS value:

• Keys arrive in an encoding whose byte order is the order IndexedDB compares keys in (key.rs documents it), so a driver orders and compares them with memcmp. Ranges bound these encodings.
• Values are the bytes of the value’s structured serialization, which the driver stores and returns as they are.
• Key paths are opaque bytes the core wrote and reads back.
• Index entries are (index key, primary key) pairs the core computes; an index scan returns them in key order, then primary key order, with the value of the record each references.

The core does the rest: key paths and key generators, computing index keys, unique constraints, cursors, requests, events and transaction scheduling. localStorage is one database per origin, with one object store whose records are the items, keyed by the encoding of each name as a string key.

Keys nest at most 128 arrays deep: a deeper value is not a valid key (a DataError). Converting a key takes about 2 KB of stack a level with QuickJS in a release build, so 128 levels fit a 512 KB thread; Chromium’s 2000 would need 4 MB, more than an iOS main thread’s 1 MB. Names cross the C API as UTF-8, so a lone surrogate in a database or object store name, or in a localStorage item’s name or value, becomes U+FFFD: two names that differ only there are one name, and two localStorage keys that differ only there are one item.

What the core guarantees a driver

• Every call happens on the DOM’s thread, in the order the core makes it.
• In a database, at most one readwrite or versionchange transaction is running at a time. Read-only transactions run alongside it only on object stores it does not use, and two databases are independent. So a driver can implement a writing transaction as one underlying write transaction per database, and a read-only one as plain reads.
• Reads and writes stay in the transaction’s scope, and schema changes (set_version, creating, deleting and renaming object stores and indexes) happen only in versionchange transactions.
• The core numbers object stores and indexes. An id is unique among a database’s object stores (or indexes), and is reused only after the transaction that used it aborted.
• open is called once per database however many connections script opens, and close when the last one closes; delete only for a database that is not open.

In return, commit must write everything the transaction did atomically, and abort must undo all of it, version and schema included. put replaces the record with the same key, deleting that record’s index entries first; delete_range and clear delete index entries with the records; deleting an object store deletes its records, indexes and their entries.

Limits

• Busy files. Every SQLite connection waits up to 5 seconds for a lock another DOM holds, blocking the DOM’s thread meanwhile, and then fails the transaction with an UnknownError. Workers have storage of their own (see Workers), so only another DOM on the same files can hold a lock: another page of the app, or a widget or app extension sharing the directory.
• Deleting a database another DOM has open. deleteDatabase removes the file even while another DOM has it open (where the platform lets it: Windows does not): that DOM’s transactions on it fail with an UnknownError from then on, until it closes the database and opens it again.
• No versionchange across DOMs. versionchange and blocked events reach only the connections of the DOM that opens or deletes the database; another DOM’s connections, in this process or another, never hear of an upgrade.
• Drivers are not Send. A driver lives on its DOM’s thread and is called only there.

Errors

A Rust driver returns DriverError { quota, message }. A C driver returns 0 (NISO_OK) for success, NISO_QUOTA_EXCEEDED_ERROR when storage is full, or any other status for another failure. Script sees a full disk as a QuotaExceededError and any other failure as an UnknownError: the request fails, and a failure to begin or commit aborts the transaction.

Writing a driver in C

Fill in a niso_storage_driver (in niso_runtime.h, with a comment on each function) and hand it to the DOM before script runs. A sketch of open and the setup:

static int32_t my_open(void* userdata, const char* origin, size_t len, uint8_t endpoint,
                       const uint16_t* name, size_t name_len, uint64_t* db, uint64_t* version,
                       const niso_storage_schema* schema) {
    struct my_store* s = userdata;
    struct my_db* d = my_find_or_create(s, origin, len, endpoint, name, name_len);
    *db = d->id;
    *version = d->version;
    for (size_t i = 0; i < d->stores_len; i++) {
        schema->store(schema->ctx, &d->stores[i].info);
        for (size_t j = 0; j < d->stores[i].indexes_len; j++)
            schema->index(schema->ctx, d->stores[i].info.id, &d->stores[i].indexes[j]);
    }
    return NISO_OK;
}

/* ... the other functions ... */

niso_storage_driver driver = {
    .userdata = my_store_new(),
    .open = my_open,
    /* ... */
    .drop = my_store_free,
};
niso_dom_set_storage_driver(dom, &driver);

Pointers the DOM passes in (origins, names, keys, values, ranges, schema records) are valid only during the call, so a driver copies what it keeps. scan calls visit for each entry in the range, in order or in reverse, until visit returns false; the bytes it passes need only live until visit returns.

Writing a driver in Rust

Implement niso_core::storage::Driver and pass it to Dom::set_storage_driver. MemoryDriver is the shortest complete driver to start from. The exercise test in storage/mod.rs lists what every driver must do and runs it against the drivers niso ships, and a_hosts_driver_keeps_indexeddb_and_local_storage in crates/niso-quickjs/tests/storage.rs plugs a driver in through the C vtable and runs IndexedDB and localStorage on it.