Render stream format

Generated from the comments in crates/niso-core/include/rendering.h, which define the format; the functions around it are in Rendering. This is render stream version 2 (NISO_RENDER_VERSION, which niso_render_version() returns). Rendering explains the model.

Frames

niso_render_frame runs the rendering update (scroll events, then animation frame callbacks, then style and layout) and returns what changed in the rendered box tree since the previous call, as a byte stream valid until the next call into the DOM. The first frame, and the first after niso_render_reset, describes the whole tree. Hosts keep one view per render node and apply the ops in order.

Values are little-endian. Geometry is in CSS pixels; colors are non-premultiplied sRGB packed as 0xRRGGBBAA. Render node ids are the DOM handles of the elements (or anonymous boxes) that generate the boxes.

Op	Name	Fields (first line)
1	FONT	u32 font, u32 collection index
2	CREATE	u64 node
3	DESTROY	u64 node
4	CHILDREN	u64 node, u32 n, n x u64 child
5	FRAME	u64 node, f32 x, y, width, height
6	BOX	u64 node, u32 background, f32 opacity,
7	TEXT	u64 node, u32 runs, then per run:
8	ROOT	u64 node, u32 canvas color, u8 flags, u8 scroll axes,
9	SCROLL	u64 node, u8 scroll axes, f32 scroll width, height,
10	CONTROL	u64 node, u8 kind, u8 state, f32 x, y, width, height,
11	TEXT_FIELD	u64 node, u8 kind, u8 state, u8 keyboard,
12	IMAGE	u64 node, u32 image, f32 x, y, width, height,
13	MEDIA	u64 node, u32 load, u32 n, n x u8 source URL (UTF-8),
14	NOW_PLAYING	u32 n, n x u8 title, then artist, album and artwork URL
15	EFFECTS	u64 node, u32 n, n x shadow (u32 color, f32 x, y, blur,

1 FONT

  1 FONT      u32 font, u32 collection index
              A font later TEXT ops use. niso_render_font returns its file
              bytes, valid for the life of the DOM. Hosts may call it while
              reading the frame: it leaves the frame's bytes alone.

2 CREATE

  2 CREATE    u64 node

3 DESTROY

  3 DESTROY   u64 node

4 CHILDREN

  4 CHILDREN  u64 node, u32 n, n x u64 child
              The node's children in paint order (back to front).

5 FRAME

  5 FRAME     u64 node, f32 x, y, width, height
              The border box, relative to the parent's border box origin.

6 BOX

  6 BOX       u64 node, u32 background, f32 opacity,
              f32 transform a b c d e f (about the border box origin),
              u8 overflow: 0 visible; 1 clip (text and children are
              clipped to the padding box); 2 scroll (clipped, and the host
              scrolls the children in a native scroll view, see SCROLL),
              f32 border widths top right bottom left,
              u32 border colors top right bottom left,
              u8 border styles top right bottom left (0 solid, 1 dashed,
              2 dotted, 3 double, 4 two halves: groove and ridge),
              f32 corner radii top-left x y, top-right x y,
              bottom-right x y, bottom-left x y,
              u32 inner border colors top right bottom left
              A double border is two lines a third of the width each, at
              its edges. A side in two halves is drawn solid in its color
              from the outer edge to the middle of its width, and in its
              inner color from there in. Inset and outset borders come as
              solid ones; their colors and groove's and ridge's are shaded
              already, darker or lighter by side as Chrome shades them. A
              side's inner color is its color unless it is in two halves.

7 TEXT

  7 TEXT      u64 node, u32 runs, then per run:
              u32 font, f32 size, u32 color, f32 synthetic skew degrees,
              u8 synthetic bold, u8 n, n x (u32 axis tag, f32 value),
              u32 glyphs, glyphs x (u32 glyph id, f32 x, f32 baseline y),
              u8 flags (1 right to left), f32 baseline y, f32 advance,
              u32 n, n x u16 text, u32 pieces, pieces x (u32 start, u32 end,
              f32 x)
              Glyph positions are relative to the node's border box origin.
              The text is what the run was shaped in (UTF-16): its shaped
              run on the line, so hosts that draw text instead of glyphs
              (Android's drawTextRun, which gets outline text) shape it the
              same way, kerning across a color change included. The pieces
              are ranges of it to draw, each from pen x on the baseline:
              the glyph run's own text, or one piece per cluster where
              letter-spacing moved clusters apart and per word (spaces
              left out) where justification or word-spacing moved words.
              A run whose text is unknown has none. The advance is the
              glyph run's width, which a host's own shaping of a single
              piece has to match for its positions to be the core's.
              Then u32 lines, per line: u8 style, u8 flags, u32 color,
              f32 x, y, width, thickness, f32 a, b, u32 gaps,
              gaps x (f32 from x, to x)
              The text decorations (underlines, overlines and
              line-throughs), each a line from x to x + width whose middle
              is at y, positioned like the glyphs. Styles: 0 solid; 1
              dashed, dashes a long with gaps b between them, starting
              with a dash at x; 2 dotted, round dots as wide as the line
              every b, the first centered half a thickness in from x,
              none past the end; 3 wavy, a stroke as thick as the line
              that leaves y downward, swinging a below and above it, in
              quadratic curves each b long (controls at 2a). A double line
              comes as two solid ones. Flag bit 0: the line paints over
              the text (a line-through), else under it, after the box's
              own background and before its glyphs. The gaps are ranges
              of x the line leaves out, where it would cross glyphs' ink
              (text-decoration-skip-ink, always auto: stylo parses the
              property only for Gecko); they may overlap.

8 ROOT

  8 ROOT      u64 node, u32 canvas color, u8 flags, u8 scroll axes,
              f32 scroll width, height, f32 scroll x, y,
              u32 n, n x u64 fixed node
              The root of the tree (0 when nothing renders), the color the
              host fills the whole viewport with behind it (the canvas
              background: the root element's, or the body's, transparent
              when neither has one, over the host's base color, white as
              browsers paint it unless the embedder made the host's view
              transparent, like an overlay's), the flags, and the
              viewport's scrolling area and offset: the host scrolls the
              root natively. Fixed nodes are boxes fixed to the viewport,
              in paint order, painted above the root without scrolling;
              their frames are relative to the root's border box origin.
              Flag bit 0, cover: the page's viewport meta asks for
              viewport-fit=cover. The host then makes the viewport the
              whole window, under the status bar, cutouts and the home
              indicator or navigation bar, and gives the page the safe
              area as --niso-safe-area-inset-top, -right, -bottom and
              -left in its platform style (niso_dom_set_platform_style),
              since env(safe-area-inset-*) is 0 in stylo. Otherwise the
              viewport is the safe area and the host leaves them unset.
              The viewport units' variants, once stylo resolves them (it
              gives sv*, lv* and dv* the viewport size for now): all the
              viewport without cover; with it, large is the whole window,
              small the window less the system bars that can appear, and
              dynamic whichever is on screen, differing from large only
              while bars are transient (Android's immersive mode). The
              keyboard changes none of them, as in browsers by default.

9 SCROLL

  9 SCROLL    u64 node, u8 scroll axes, f32 scroll width, height,
              f32 scroll x, y
              A scroll container's scrolling area, measured from its
              padding box origin, and the offset to show. Scroll axes are
              those the user may scroll (bit 0 x, bit 1 y, from overflow
              auto or scroll); script may scroll either, so hosts show the
              offset on both but let gestures move only these. Children's frames
              are unscrolled: the native scroll view moves them. Sent when
              it changes, including when script scrolls.

10 CONTROL

 10 CONTROL   u64 node, u8 kind, u8 state, f32 x, y, width, height,
              f64 value, min, max, step; for kind 5, u32 n, n x (u8 flags,
              u32 n, n x u8 label (UTF-8))
              The node is a form control the host draws natively in the
              given content box, relative to the border box origin (the
              UA stylesheet sizes it as niso_dom_set_control_size says,
              so the control fits): kind 1 a
              switch (input type=checkbox switch), 2 a slider (input
              type=range), 3 a linear progress bar (progress), 4 a push
              button. Unlike the others a button keeps its TEXT and
              children, which the host shows as the button's label, in
              the content box. The button fills the border box, and its
              BOX's fill, border and radii are the native button's own,
              which the host draws it with instead of drawing the box: the
              stock style when they are the stock button's (see
              niso_dom_set_button_style), else a filled one, or a
              borderless one without a fill; 5 a radio group: a fieldset
              or role=radiogroup element whose radio buttons (two or more,
              outside groups nested in it) share a name and that has no
              other form controls. It is one native control with an
              option per radio button, labeled with the text of the label
              around or for it (else its value), flag 1 when that radio
              button is disabled: a row of up to 5 options of up to 15
              characters (the element is a flex container in a row) is a
              segmented control (iOS's segmented Picker, Android's
              connected button group), any other group a list (state
              bit 16: a menu Picker on iOS, a column of radio button rows
              on Android). Its box takes the control's height, see
              niso_dom_set_control_size, and loses its padding and border,
              so the control fills it; page styles on the radio buttons
              and labels never show. A group that holds more than its
              radio buttons and their labels, such as a legend or a hint,
              keeps it (state bit 32): the control takes only their area,
              its children from the first holding a radio button or a
              label for one to the last, which becomes one box of the
              control's height as wide as the group's content (sharing a
              row with what is beside it), and the group's box, its TEXT
              and its other children are drawn as any others, its fill no
              tint. Kind 0 when
              it stops being one (appearance: none, or a type change).
              State bits: 1 checked, 2 disabled, 4 indeterminate, 8 dark
              (the element's used color scheme is dark, as its
              color-scheme and niso_dom_set_color_scheme pick it: the host
              draws the control in its dark appearance), 16 a list, 32 a
              radio group's control that takes only part of its box. The
              value is a switch's checkedness (0 or 1), a slider's value
              between min and max in multiples of step from min (step 0:
              any), a progress bar's value between 0 and max, a radio
              group's selected option between 0 and max (n - 1), or -1
              for none. A control other than a button or a radio group
              with state bit 32 has no TEXT or children. Sent when any of
              it changes.

11 TEXT_FIELD

 11 TEXT_FIELD u64 node, u8 kind, u8 state, u8 keyboard,
              f32 x, y, width, height, f32 font size, u16 weight,
              u8 italic, u8 align, u32 color, u32 caret color,
              u32 n, n x u8 value, u32 n, n x u8 placeholder (UTF-8),
              u32 selection start, end, u8 direction
              The node is a text field the host edits natively: unless
              plain, its own stock field filling the border box (and
              drawn instead of the node's BOX), else a bare editor in the
              given content box. Kind 1 single-line (input types text,
              search, email, url, tel, password, number), 2 multi-line
              (textarea); 0 when it stops being one, with nothing after
              the keyboard byte. State bits: 1 disabled, 2 read-only,
              4 secure (password), 8 plain (the page gave it a box other
              than the stock field's, see niso_dom_set_field_style, which
              the host draws as the BOX says around a bare editor), 16
              focused, 32 caret (the page set caret-color: the caret is
              the caret color, even transparent), 64 dark (as for
              CONTROL). Keyboard: 0 text, 1 email,
              2 url, 3 phone, 4 decimal, 5 search. The text uses the
              system font at this size (CSS px), weight (100-900), slant,
              alignment (0 start, 1 center, 2 end) and color, with a
              caret of the caret color when the caret bit is set, else
              the platform's. The value is
              what the field shows: the host replaces its text with it
              only when it differs from what the host last reported. The
              selection is the field's, in UTF-16 code units of the value
              (start <= end, equal for a caret; direction 0 none, 1
              forward, 2 backward): the host selects it when it differs
              from what the host last reported, so script's select(),
              setSelectionRange() and value changes (which put the caret
              at the end) reach the native field. An edit the page
              refuses (a beforeinput listener canceled it) leaves the
              field as it was, and the next frame sends this op again,
              though nothing in it changed: the host shows the value and
              selection it sends, putting its native field back. Hosts report the
              user's selection with niso_dom_text_edit and
              niso_dom_text_select. (The selection came after the other
              fields: hosts written before it must now read its 9
              bytes.) A text field has no TEXT or
              children. Sent when any of it changes.

12 IMAGE

 12 IMAGE     u64 node, u32 image, f32 x, y, width, height,
              f32 clip x, y, width, height, u8 flags,
              u32 n, n x (u32 x, y, width, height)
              The node's picture (an img element's decoded image, or a
              canvas's bitmap), drawn scaled into the rect, which
              object-fit and object-position place, clipped to the clip
              rect (the content box), both relative to the border box
              origin, above the BOX and below the TEXT and children.
              niso_render_image returns the image's pixels; the same image
              keeps its id. Image u32 max when the node stops having one.
              Flags: 1 opaque (ignore the pixels' alpha: a canvas whose
              context has no alpha), 2 pixelated (image-rendering:
              pixelated or crisp-edges: scale with nearest-neighbor
              filtering, else smoothly), 4 live (a canvas's bitmap: its
              pixels change and its size may, so hosts update their copy
              rather than keep it). The rectangles are a live image's
              pixels that changed since the node's previous IMAGE, all of
              them when the host hasn't had the image at this size; the
              host copies those from niso_render_image. None for other
              images. Sent when any of it changes, including a live
              image's pixels. A canvas no script has drawn into has no
              image (it is transparent), and its fallback content does
              not render.

13 MEDIA

 13 MEDIA     u64 node, u32 load, u32 n, n x u8 source URL (UTF-8),
              u8 flags, f32 volume, f32 playback rate, f64 seek
              An audio element's player, which the host keeps whether or
              not the element renders. It plays the source when flag 1 is
              set and pauses otherwise; flag 2 mutes it and flag 4 loops
              it. A new load number means load the source again from the
              start. Seek is a position in seconds to jump to once, NaN
              for none. An empty source means stop and drop the player.
              Hosts report the player with niso_dom_media_update. Sent
              when any of it changes.

14 NOW_PLAYING

 14 NOW_PLAYING  u32 n, n x u8 title, then artist, album and artwork URL
              the same way, u8 playback state (0 none, 1 paused,
              2 playing), u32 actions, f64 duration, f64 position,
              f64 rate
              What the platform shows as now playing (lock screen,
              notification, control center) and the controls it offers:
              bit n of actions is niso_dom_media_session_action's action
              n. Sent when any of it changes. duration, position and rate
              are the page's position state (setPositionState), NaN
              without one: seconds, the duration infinite for a live
              stream, the position as of this frame, from which hosts
              advance it at the rate while the state is playing. Without
              a seekto action (bit 7) hosts offer no seeking.

15 EFFECTS

 15 EFFECTS   u64 node, u32 n, n x shadow (u32 color, f32 x, y, blur,
              spread, u8 inset), u32 n, n x filter (u8 kind, f32 amount),
              u32 n, n x backdrop filter (the same)
              The box's box-shadow (in paint order, first on top, with
              the blur radius as CSS gives it, twice the Gaussian's
              standard deviation), filter and backdrop-filter. Filter
              kinds: 1 blur (the standard deviation, CSS pixels),
              2 brightness, 3 contrast, 4 grayscale, 5 hue-rotate
              (degrees), 6 invert, 7 opacity, 8 saturate, 9 sepia, each
              with its CSS amount (1 is 100%) and applied in order;
              drop-shadow() and url() are left out. A filter applies to
              the box with its text and children, as one layer; a backdrop
              filter to what is behind the box, inside its border box's
              curve. Sent when any of it changes; all three empty when the
              box stops having them. Boxes that never have any get none.

Versions: ops carry no length, so a host reads every op and field of the stream. Ops and their fields only grow at the end: a new op takes the next number and a new field goes after an op’s last, and niso_render_version goes up with each, so a host updates by reading the additions. A host checks niso_render_version() against the NISO_RENDER_VERSION it was written for and reads no other version’s stream.

Hosts report the user’s scrolling with niso_dom_scroll; the next niso_render_frame fires the scroll events. They report the user’s changes to native controls with niso_dom_control_input and niso_dom_text_edit, and show the state the next CONTROL or TEXT_FIELD op sends, which script may have changed or refused. Native views that take focus report it with niso_dom_focus_input; script’s focus comes back in the focused bit.

Templates

Template targets (home screen widgets, Live Activities and the Dynamic Island, ongoing notifications; later CarPlay and Android Auto) take only native views and lay text out themselves, often in another process that runs no niso. A page renders content for them as surfaces: elements with a data-niso-surface attribute naming the surface, like

A surface is laid out with the page, at the size its CSS gives it, but is never part of the render stream and takes no taps (the UA stylesheet makes it position: fixed at the viewport’s top left, pointer-events: none). The names hosts know:

widget.small, widget.medium home screen widgets, by size class activity iOS Live Activity on the Lock Screen and the expanded Dynamic Island; Android’s ongoing notification. Present = running. island.leading, island.trailing, island.minimal the compact and minimal Dynamic Island

niso_render_surfaces lays the document out and returns the surfaces whose template changed since the previous call, and those that went away (removed, display: none or renamed), valid until the next call into the DOM; the first call after niso_render_reset returns every surface. Hosts call it after each niso_render_frame. It is a list of

name_len:u32 name:utf8[name_len] len:u32 template:u8[len]

sorted by name, where len 0 means the surface is gone. When two surfaces share a name the first in tree order wins.

niso_render_template returns the template of any element, as the document lays it out, in a buffer valid until the next call into the DOM: a server rendering content for push updates calls it on a headless DOM.

A template is the surface’s box tree as the render stream would describe it, reduced to what native template views take: boxes, text as strings, timers, progress bars and images. Little-endian, lengths in CSS pixels (points on iOS, dp on Android), colors non-premultiplied sRGB 0xRRGGBBAA:

template := ‘N’ ‘T’ version:u8 width:f32 height:f32 node node := kind:u8 x:f32 y:f32 w:f32 h:f32 len:u16 body:u8[len] count:u8 node[count] body := flags:u8 [background:u32] flags & 1 [border_width:f32 border_color:u32] flags & 2 (the top edge’s) [radius:f32] flags & 4 (top left) [opacity:f32] flags & 8 kind’s fields flags & 16: clips children

The root node is the surface, at (0, 0) and the template’s size. A node’s frame is its border box in its parent’s; children are in paint order. Boxes that draw nothing are flattened away, their children moved to their parent, so a template stays shallow. Kinds and their fields:

Kind	Name	Fields (first line)
1	BOX	none
2	TEXT	text_style string: an element's inline content, as one string
3	TIMER	text_style start:f64 end:f64: a <time> element with a
4	PROGRESS	value:f32 max:f32 tint:u32 track:u32 flags:u8 start:f64
5	IMAGE	string: an <img>'s src, which targets resolve to an image
6	BUTTON	node:u64 flags:u8: a <button> (flags 1 when disabled), its

Template 1 BOX

  1 BOX       none

Template 2 TEXT

  2 TEXT      text_style string: an element's inline content, as one string
              in the style of its first run (whitespace collapsed, inline
              boxes dropped). A box with text may also have children.

Template 3 TIMER

  3 TIMER     text_style start:f64 end:f64: a <time> element with a
              data-timer-start (counting up from) or data-timer-end
              (counting down to) attribute, in ms since the Unix epoch, 0
              when absent. Targets show the platform's self-updating timer
              (SwiftUI Text(date, style: .timer), Android Chronometer)
              instead of the text the page shows, so the template does not
              change while it runs. A timer must be a box of its own, like
              a block or a flex item, to be one; inline, its text is part
              of its paragraph's. A <time> without these attributes (a
              paused timer) is TEXT.

Template 4 PROGRESS

  4 PROGRESS  value:f32 max:f32 tint:u32 track:u32 flags:u8 start:f64
              end:f64 period:f64: a <progress> element, tint its CSS
              background (0 for the platform's), track 0 for the
              platform's, flags 1 when indeterminate. With data-timer-start
              and data-timer-end (ms since the Unix epoch) the bar fills
              from empty at start to full at end by itself; with
              data-timer-start and data-timer-period (ms) it fills every
              period from start, then starts over; its value is then 0,
              for the target to compute. Times are 0 when absent.

Template 5 IMAGE

  5 IMAGE     string: an <img>'s src, which targets resolve to an image
              they bundle or have stored.

Template 6 BUTTON

  6 BUTTON    node:u64 flags:u8: a <button> (flags 1 when disabled), its
              own box its only child at (0, 0). Targets that can act on
              it (Android's ongoing notification, as its actions) show
              its text and report the user's action with
              niso_dom_activate(node), node meaning something only to the
              DOM that rendered it; targets that cannot leave it out (the
              iOS widgets and Live Activity, which run no page), and
              decoders older than it draw it as a box.

  text_style := x:f32 y:f32 w:f32 h:f32 size:f32 weight:u16 color:u32
                align:u8 font:u8 lines:u8
  string     := len:u16 utf8[len]

text_style’s rect is where the text’s lines sit in the node: the content box’s left and width, the lines’ top and height. size is the font size, weight 1-1000, align 0 start, 1 center, 2 end; font’s low two bits are the design (0 the system font, 1 monospaced: monospace or ui-monospace, 2 rounded: ui-rounded, 3 serif: serif or ui-serif), 4 italic, 8 tabular figures (font-variant-numeric: tabular-nums). The rect and lines are as the core laid the text out at the user’s text size (niso_dom_set_text_scale), but size is the CSS font size: targets draw it at the platform’s own text scale (sp, Dynamic Type), which matches the core’s when the template is shown at the text size it was made at. lines is how many lines the rect holds: all of them, or those that fit whole in a box that clips its overflow (overflow other than visible, as with a max-height). Targets keep to that many and end text that still does not fit with an ellipsis, never shrinking or cutting it.

Compatibility: an app and its widget extension update separately, and a template from a newer app (or a server) can reach an older extension. A decoder rejects a template whose magic or version differs from its own, and shows nothing rather than guessing. Within a version, newer cores only append fields to a body and add kinds; decoders read the fields they know, skip the rest of the body by its length, and draw a node of an unknown kind as a BOX (its children included). A change that cannot be expressed that way is a new version. One change within version 1 could not be: text lines used to be every line the core laid out, which targets kept to by shrinking the text, and are now those a clipping box holds, which targets keep to with an ellipsis. A target from before it shrinks a newer template’s clipped text to fit instead of ending it with an ellipsis.

Edit page

Previous
niso_intl.h: Intl for engines without itNext
Overview