503 lines
21 KiB
Markdown
503 lines
21 KiB
Markdown
# quickjs-ng serialized bytecode — format specification
|
||
|
||
Produced by `qjsc -b`, and by `JS_WriteObject()` with `JS_WRITE_OBJ_BYTECODE`.
|
||
|
||
Applies to **quickjs-ng v0.16.1, `BC_VERSION 27`**. Hand-maintained against
|
||
`quickjs.c`; quickjs-ng raises `BC_VERSION` on any layout change, so re-check
|
||
after bumping `third_party/quickjs-ng`.
|
||
|
||
- **Part 1** — the wire format, as one declaration.
|
||
- **Part 2** — the algorithms it depends on.
|
||
- **Part 3** — the C structures behind it, and the enumerations.
|
||
- **Appendices** — differences from the original QuickJS; how to verify.
|
||
|
||
---
|
||
|
||
## Notation
|
||
|
||
The wire format is written as TypeScript type declarations. Two caveats,
|
||
since TypeScript is being used for its syntax rather than its semantics:
|
||
|
||
- **Field order is the wire order.** TypeScript object types are unordered;
|
||
here the declaration order is normative.
|
||
- **Widths come from the primitive aliases below**, not from `number`.
|
||
|
||
| Alias | Encoding |
|
||
|---|---|
|
||
| `u8` `u16` `u32` `u64` | Unsigned, fixed width, **native byte order** |
|
||
| `leb` | Unsigned LEB128, 7 bits per byte, at most 5 bytes |
|
||
| `sleb` | Signed LEB128, zigzag (§2.4) |
|
||
| `bit` | One bit inside a packed integer |
|
||
|
||
```ts
|
||
type u8 = number; type u16 = number; type u32 = number; type u64 = bigint;
|
||
type leb = number; type sleb = number; type bit = boolean;
|
||
```
|
||
|
||
An optional field written `x?: T` is present only under the condition stated
|
||
in its comment. A union of object types is a discriminated union: the shared
|
||
literal field is a real field on the wire, and its width is given in the
|
||
type's doc comment, since a TypeScript literal type carries no width.
|
||
|
||
Fixed-width integers are written with `memcpy`, so a file carries the
|
||
**producer's** byte order. Every mainstream target is little-endian; files are
|
||
not portable across endianness.
|
||
|
||
---
|
||
|
||
# Part 1 — Wire format
|
||
|
||
```ts
|
||
// ═══ container ═══════════════════════════════════════════════════════════
|
||
|
||
/** Repeats until end of file. The count is not stored. */
|
||
type File = SerializedObject[];
|
||
|
||
type SerializedObject = {
|
||
version: u8; // === 27, else reject
|
||
checksum: u32; // over bytes 5..end of object, §2.1
|
||
atom_count: leb; // limit 1_000_000
|
||
atoms: AtomEntry[]; // length = atom_count
|
||
root: Value; // MODULE, or FUNCTION for a classic script
|
||
};
|
||
|
||
// ═══ primitives ══════════════════════════════════════════════════════════
|
||
|
||
type JSString = {
|
||
header: leb; // (len << 1) | is_wide_char
|
||
data: u8[] | u16[]; // len Latin-1 bytes, or len UTF-16 units
|
||
};
|
||
|
||
/** `type` is a u8. */
|
||
type AtomEntry =
|
||
| { type: 0; raw: u32 } // a predefined atom number
|
||
| { type: 1 | 2 | 3; text: JSString }; // string | global_symbol | symbol
|
||
|
||
/**
|
||
* One leb. Decoding, where i = v >> 1:
|
||
* v & 1 -> tagged integer atom, value i
|
||
* i < first_atom -> predefined atom number i
|
||
* i >= first_atom -> atoms[i - first_atom]
|
||
*/
|
||
type AtomRef = leb;
|
||
|
||
// ═══ values ══════════════════════════════════════════════════════════════
|
||
|
||
/** `tag` is a u8. */
|
||
type Value =
|
||
| { tag: 1 } // NULL
|
||
| { tag: 2 } // UNDEFINED
|
||
| { tag: 3 } // BOOL_FALSE
|
||
| { tag: 4 } // BOOL_TRUE
|
||
| { tag: 5; value: sleb } // INT32
|
||
| { tag: 6; bits: u64 } // FLOAT64, IEEE-754
|
||
| { tag: 7; s: JSString } // STRING
|
||
| { tag: 8; count: leb; props: [AtomRef, Value][] } // OBJECT
|
||
| { tag: 9; count: leb; items: Value[] } // ARRAY
|
||
| { tag: 10; len: leb; mag: u8[] } // BIG_INT, two's complement LE
|
||
| { tag: 11; count: leb; cooked: Value[]; raw: Value } // TEMPLATE_OBJECT
|
||
| { tag: 12; fn: FunctionBytecode } // FUNCTION
|
||
| { tag: 13; module: Module } // MODULE
|
||
| { tag: 14; kind: u8; len: leb; offset: leb; buffer: Value } // TYPED_ARRAY
|
||
| { tag: 15; len: leb; max: leb; data: u8[] } // ARRAY_BUFFER
|
||
| { tag: 16; len: leb; max: leb; ptr: u64 } // SHARED_ARRAY_BUFFER, host pointer
|
||
| { tag: 17; pattern: JSString; lre_bytecode: JSString } // REGEXP
|
||
| { tag: 18; timestamp: Value } // DATE
|
||
| { tag: 19; inner: Value } // OBJECT_VALUE
|
||
| { tag: 20; index: leb } // OBJECT_REFERENCE
|
||
| { tag: 21; count: leb; entries: [Value, Value][] } // MAP
|
||
| { tag: 22; count: leb; items: Value[] } // SET
|
||
| { tag: 23; description: AtomRef }; // SYMBOL
|
||
|
||
// ═══ functions ═══════════════════════════════════════════════════════════
|
||
|
||
type FunctionBytecode = {
|
||
flags: FunctionFlags;
|
||
is_strict_mode: u8; // not part of flags
|
||
func_name: AtomRef;
|
||
arg_count: leb;
|
||
var_count: leb;
|
||
defined_arg_count: leb; // Function.prototype.length
|
||
stack_size: leb; // max operand stack depth
|
||
var_ref_count: leb;
|
||
closure_var_count: leb;
|
||
cpool_count: leb;
|
||
byte_code_len: leb; // in bytes
|
||
vardef_count: leb; // arg_count + var_count, or 0
|
||
vardefs: VarDef[]; // arguments first, then locals
|
||
closure_vars: ClosureVar[]; // length = closure_var_count
|
||
cpool: Value[]; // length = cpool_count; nested functions live
|
||
// here. NOTE: before the code
|
||
byte_code: Instruction[]; // byte_code_len bytes in total
|
||
debug?: DebugBlock; // present iff flags.has_debug
|
||
};
|
||
|
||
/** Packed into one u16, least significant bit first. */
|
||
type FunctionFlags = {
|
||
has_prototype: bit; // bit 0
|
||
has_simple_parameter_list: bit;
|
||
is_derived_class_constructor: bit;
|
||
need_home_object: bit;
|
||
func_kind: FuncKind; // bits 4..5
|
||
new_target_allowed: bit;
|
||
super_call_allowed: bit;
|
||
super_allowed: bit;
|
||
arguments_allowed: bit;
|
||
backtrace_barrier: bit;
|
||
has_debug: bit; // bit 11
|
||
};
|
||
|
||
type VarDef = {
|
||
name: AtomRef;
|
||
scope_level: leb; // 0 = not lexically scoped; §3.3
|
||
scope_next_plus_1: leb; // int32(v) - 1; -1 none, -2 arg-scope end
|
||
flags: VarDefFlags;
|
||
var_ref_idx?: leb; // present iff flags.is_captured
|
||
};
|
||
|
||
/** Packed into one u8, least significant bit first. */
|
||
type VarDefFlags = {
|
||
var_kind: VarKind; // bits 0..3
|
||
is_const: bit; // bit 4
|
||
is_lexical: bit; // bit 5
|
||
is_captured: bit; // bit 6
|
||
};
|
||
|
||
type ClosureVar = {
|
||
name: AtomRef;
|
||
var_idx: leb; // index in the PARENT frame
|
||
flags: ClosureVarFlags;
|
||
};
|
||
|
||
/** Packed into one leb, least significant bit first. */
|
||
type ClosureVarFlags = {
|
||
closure_type: ClosureType; // bits 0..2
|
||
is_const: bit; // bit 3
|
||
is_lexical: bit; // bit 4
|
||
var_kind: VarKind; // bits 5..8
|
||
};
|
||
|
||
type Instruction = {
|
||
opcode: u8; // §2.3 gives its size and operand format
|
||
operands: u8[]; // size - 1 bytes, fixed width per format
|
||
};
|
||
|
||
type DebugBlock = {
|
||
filename: AtomRef;
|
||
line_num: leb; // of the function's first character
|
||
col_num: leb;
|
||
pc2line_len: leb;
|
||
pc2line: u8[]; // §2.2
|
||
source_len: leb; // 0 when stripped
|
||
source: u8[]; // the function's own text
|
||
};
|
||
|
||
// ═══ modules ═════════════════════════════════════════════════════════════
|
||
|
||
type Module = {
|
||
module_name: AtomRef;
|
||
req_count: leb;
|
||
requests: AtomRef[]; // import specifiers
|
||
export_count: leb;
|
||
exports: ExportEntry[];
|
||
star_count: leb;
|
||
star_exports: leb[]; // indexes into requests
|
||
import_count: leb;
|
||
imports: ImportEntry[];
|
||
has_tla: u8; // top-level await
|
||
func_obj: Value; // the body; always a FUNCTION
|
||
};
|
||
|
||
/** `export_type` is a u8. */
|
||
type ExportEntry =
|
||
| { export_type: 0; var_idx: leb; export_name: AtomRef }
|
||
| { export_type: 1; req_module_idx: leb; local_name: AtomRef; export_name: AtomRef };
|
||
|
||
type ImportEntry = {
|
||
var_idx: leb;
|
||
import_name: AtomRef;
|
||
req_module_idx: leb;
|
||
};
|
||
|
||
// ═══ enumerations, §3.5 ══════════════════════════════════════════════════
|
||
|
||
const enum FuncKind { normal = 0, generator = 1, async = 2, async_generator = 3 }
|
||
|
||
const enum VarKind {
|
||
normal = 0, function_decl = 1, new_function_decl = 2, catch_ = 3,
|
||
function_name = 4, private_field = 5, private_method = 6,
|
||
private_getter = 7, private_setter = 8, private_getter_setter = 9,
|
||
using = 10, using_method = 11,
|
||
}
|
||
|
||
const enum ClosureType {
|
||
local = 0, arg = 1, ref = 2, global_ref = 3,
|
||
global_decl = 4, global = 5, module_decl = 6, module_import = 7,
|
||
}
|
||
```
|
||
|
||
## Notes on the above
|
||
|
||
Things the types cannot state.
|
||
|
||
1. **`first_atom` is not in the file.** It is `JS_ATOM_END` (242) when the file
|
||
contains bytecode and 1 otherwise. The engine takes it from the caller's
|
||
`JS_READ_OBJ_BYTECODE` flag, so a reader must be told which applies.
|
||
2. **Atom operands inside `byte_code` are bare `u32` with no shift.** They use
|
||
the same index space as `AtomRef`, but `JS_WriteFunctionBytecode()` writes
|
||
them with `bc_atom_to_idx()` rather than `bc_put_atom()`, so the `v >> 1`
|
||
step does not apply there.
|
||
3. **Functions nest through `cpool`, and only through `cpool`.** A
|
||
`FunctionBytecode` appears in exactly two places: `Module.func_obj`, the
|
||
module body, and `FunctionBytecode.cpool[]`, a nested function. There is no
|
||
list of functions anywhere. Since `cpool` also precedes `byte_code` — so a
|
||
reader can resolve a `push_const` operand the moment it decodes it — a
|
||
nested function's entire encoding sits inside its parent, ahead of the
|
||
parent's first instruction.
|
||
4. **`VarDef` is variable-width** because of the optional `var_ref_idx`.
|
||
Reading or skipping it unconditionally desynchronises the rest of the table.
|
||
5. **Jump operands are signed offsets from the operand field's own position**:
|
||
target = `operand_offset + value`. For the `atom_label_*` formats the base
|
||
is the label field, 4 bytes into the operands.
|
||
6. **`0xffffffff` in `checksum` disables the check.**
|
||
7. **Tags 14–16, 18, 21 and 22 never come from compiled JavaScript**; they
|
||
arise from serializing live objects. A regexp *literal* compiles to two
|
||
`STRING` constants plus a `regexp` opcode, not to tag 17.
|
||
8. **`source` is required by `Function.prototype.toString()`**, not merely
|
||
diagnostic. `qjsc -s` empties it; a second `-s` clears `has_debug` and
|
||
removes `DebugBlock` entirely.
|
||
|
||
---
|
||
|
||
# Part 2 — Algorithms
|
||
|
||
## 2.1 Checksum
|
||
|
||
Over the object's bytes from offset 5 to its end, compared with the `u32` at
|
||
offset 1.
|
||
|
||
```c
|
||
uint32_t bc_csum(const uint8_t *p, size_t n) {
|
||
uint32_t a, b, c, h = 0;
|
||
size_t i;
|
||
for (i = 0; i + 4 < n; i += 4) { /* note: <, not <= */
|
||
h += get_u32(p + i);
|
||
h *= 0x9e370001;
|
||
}
|
||
a = b = c = 0;
|
||
switch (n - i) { /* note: no case 4 */
|
||
case 3: c = p[i+2]; /* fallthrough */
|
||
case 2: b = p[i+1]; /* fallthrough */
|
||
case 1: a = p[i]; /* fallthrough */
|
||
case 0: break;
|
||
}
|
||
h += a | b << 8 | c << 16;
|
||
return h * 0x9e370001;
|
||
}
|
||
```
|
||
|
||
The loop stops while up to four bytes remain and the tail `switch` has no
|
||
`case 4`, so a trailing group of exactly four bytes contributes nothing.
|
||
Reproduce it exactly or valid files fail verification.
|
||
|
||
## 2.2 pc2line decoding
|
||
|
||
Produces checkpoints mapping a bytecode offset to a source position. State
|
||
starts at `pc = 0`, with `line` and `col` from `DebugBlock`. Constants:
|
||
`PC2LINE_BASE = -1`, `PC2LINE_RANGE = 5`, `PC2LINE_OP_FIRST = 1`.
|
||
|
||
```
|
||
while bytes remain:
|
||
op = read u8
|
||
if op == 0: // escape for large deltas
|
||
pc += read leb
|
||
line += read sleb
|
||
else:
|
||
op -= PC2LINE_OP_FIRST
|
||
pc += op / PC2LINE_RANGE
|
||
line += op % PC2LINE_RANGE + PC2LINE_BASE
|
||
col += read sleb // in both branches
|
||
emit checkpoint (pc, line, col)
|
||
```
|
||
|
||
One byte therefore encodes a small `pc` advance together with a line delta in
|
||
the range −1..+3.
|
||
|
||
## 2.3 Opcode resolution
|
||
|
||
Short opcodes fold an operand into the opcode byte, and their numbers overlap
|
||
the compiler's temporary opcodes, which never appear in a file. Resolve as the
|
||
engine does:
|
||
|
||
```c
|
||
#define short_opcode_info(op) \
|
||
opcode_info[(op) >= OP_TEMP_START ? (op) + (OP_TEMP_END - OP_TEMP_START) : (op)]
|
||
```
|
||
|
||
For v0.16.1: `OP_TEMP_START` = 185, temporary block = 19, real opcodes = 252.
|
||
Omitting the shift yields both the wrong instruction and the wrong width.
|
||
Every opcode, its size and its operand format are in
|
||
`tests/report/OPCODES.md`.
|
||
|
||
## 2.4 Signed LEB128
|
||
|
||
Zigzag over unsigned LEB128.
|
||
|
||
```
|
||
encode: leb( (2 * v) ^ -(v >> 31) )
|
||
decode: (u >> 1) ^ -(u & 1)
|
||
```
|
||
|
||
---
|
||
|
||
# Part 3 — Why the format looks like this
|
||
|
||
Facts about the engine that explain choices in Part 1. The C structures
|
||
themselves are in `quickjs.c`; only what bears on the wire is here.
|
||
|
||
## 3.1 Atoms exist only at run time
|
||
|
||
A `JSAtom` is a `uint32_t` index into `JSRuntime.atom_array`, shared by
|
||
everything in the process and dependent on what was interned and when. It is
|
||
meaningless in another process.
|
||
|
||
`AtomEntry[]` therefore corresponds to no structure. `JS_WriteObjectAtoms()`
|
||
creates it: it walks the object, collects every atom referenced, writes them
|
||
as a file-local table, and rewrites each atom field as an index into it.
|
||
Indexes below `first_atom` are engine built-ins needing no entry, which is why
|
||
only atoms the engine did not already know are written out.
|
||
|
||
## 3.2 What is not serialized
|
||
|
||
`JSFunctionBytecode` holds a GC header, a `JSContext *realm`, and a pointer to
|
||
each of its arrays. None of that is written: pointers become inline data, and
|
||
the header and realm are rebuilt on load. `JSModuleDef` is mostly link and
|
||
evaluation state — `module_ns`, `status`, `dfs_index`, `cycle_root`,
|
||
`promise`, `async_parent_modules` — and `Module` stores only the static shape
|
||
the compiler determined.
|
||
|
||
Two fields are wire-only, with no structure behind them:
|
||
|
||
- **`has_debug`** records the writer's `allow_debug` setting, which is how
|
||
`qjsc -s -s` produces a function with no `DebugBlock`.
|
||
- **`vardef_count`** is redundant with `arg_count + var_count`; the writer
|
||
emits it anyway, and 0 when `vardefs` is absent.
|
||
|
||
One field is structure-only: `JSVarDef.is_static_private` is used while
|
||
parsing private class fields and never written, which is why `VarDefFlags`
|
||
carries seven meaningful bits and not eight.
|
||
|
||
## 3.3 The scope chain
|
||
|
||
`scope_level` and `scope_next` are a fragment of the compiler's scope tree,
|
||
serialized for one consumer: direct `eval`.
|
||
|
||
`scope_level` indexes `fd->scopes[]`, an array that is **not** serialized, so
|
||
the number carries no meaning of its own — only equality does. Two variables
|
||
with the same level were declared in the same block.
|
||
|
||
Level 0 is the exception and is universal: it is the function scope, so
|
||
`scope_level == 0` means the variable is not lexically scoped — a `var`, a
|
||
hoisted function declaration, or a pseudo-variable such as `this`. Level 1 is
|
||
`ARG_SCOPE_INDEX`, the argument scope, which is also the body scope unless the
|
||
function has parameter expressions. Only then does the body get a scope of its
|
||
own, at level 2, and only then can an argument appear twice: once in the
|
||
argument half of `vardefs` and once as a lexical copy at level 1.
|
||
|
||
`scope_next` is a singly linked list, `-1` at the end. It indexes the
|
||
**locals**, that is `vardefs[arg_count + scope_next]`, not `vardefs` directly.
|
||
`js_create_function()` rebuilds the chain immediately before the function is
|
||
written: the variables of one block link to each other, and the last of them
|
||
links to the block outside it. Following `scope_next` from any variable
|
||
therefore walks its own block and then the blocks around it, outward.
|
||
|
||
```js
|
||
function h() {
|
||
var v = 1; // level 0 not in the chain
|
||
let w = 2; // level 1 next -1 end
|
||
{ let u = 3; } // level 2 next 1 -> w
|
||
}
|
||
```
|
||
|
||
`OP_eval` and `OP_apply_eval` hold the entry point: a `u16` operand carrying a
|
||
local index biased by +1. At run time `add_closure_variables()` starts there,
|
||
follows `scope_next`, and makes a closure variable out of every variable it
|
||
reaches, which is how evaluated code sees the caller's locals. Nothing else in
|
||
the engine reads either field. A function that contains no direct `eval` still
|
||
carries the chain.
|
||
|
||
Two cases do not follow the rules above:
|
||
|
||
| Where | Value | Meaning |
|
||
|---|---|---|
|
||
| `scope_next` | `-2`, stored as `0xffffffff` | `ARG_SCOPE_END`: stop here, and do not add the arguments. Written only when the function has parameter expressions — a default value or a destructuring pattern. |
|
||
| any entry with `scope_level` 0 | `scope_next` arbitrary | not maintained. The whole argument half of `vardefs` is level 0, and so is every `var`. `JS_DUMP_BYTECODE_FINAL` prints the pair only when `scope_level != 0`, for this reason. |
|
||
|
||
Because of `ARG_SCOPE_END` the field must be decoded as a **signed** 32-bit
|
||
value before the −1, matching `bc_get_leb128_int()`. Read as unsigned, −2
|
||
becomes 4294967294.
|
||
|
||
## 3.4 The capture pairing
|
||
|
||
`VarDef` and `ClosureVar` are the two halves of one capture. A parent marks a
|
||
local `is_captured` and assigns it a `var_ref_idx`; the child lists the same
|
||
variable with `closure_type = local` and a `var_idx` pointing into the
|
||
parent's frame. Either side read alone is incomplete.
|
||
|
||
`ClosureType.local` and `.arg` index the parent's variables and arguments
|
||
respectively; the remaining six index the parent's own closure variables, or
|
||
describe a global or module binding.
|
||
|
||
## 3.5 Enumeration names
|
||
|
||
| TypeScript | C |
|
||
|---|---|
|
||
| `FuncKind` | `JSFunctionKindEnum` — one bit for generator, one for async, so the four values compose |
|
||
| `VarKind` | `JSVarKindEnum` — `catch_` is `JS_VAR_CATCH`, renamed only to avoid the reserved word |
|
||
| `ClosureType` | `JSClosureTypeEnum` |
|
||
| `ExportEntry.export_type` | `JSExportTypeEnum` |
|
||
|
||
`VarKind.using_method` is a hidden local holding the cached dispose method for
|
||
the preceding `using` variable, always allocated immediately after it.
|
||
|
||
## Appendix A — Differences from the original QuickJS
|
||
|
||
Read from Bellard's `quickjs-2026-06-04`, the release current when this was
|
||
written. Both projects are active, so date the comparison before trusting it.
|
||
|
||
| Aspect | Bellard 2026-06-04 | quickjs-ng v0.16.1 |
|
||
|---|---|---|
|
||
| `BC_VERSION` | 5 | 27 |
|
||
| Header | Version byte, then the atom count | Version byte, then a `u32` checksum |
|
||
| Atom entries | String only | Type byte, then a string or a raw number |
|
||
| Function body order | Code, debug, then constant pool | Constant pool, then code, then debug |
|
||
| Byte after flags | `js_mode` | `is_strict_mode` |
|
||
| Tags | 19; no `MAP`, `SET`, `SYMBOL`, `REGEXP` | 23; later tag numbers shifted |
|
||
| Debug block | Filename and `pc2line` only | `line_num` and `col_num` as well |
|
||
| `VarDef` | No `scope_level`. `var_ref_idx` always written. Extra `has_scope` flag bit | `scope_level` present. `var_ref_idx` only when captured |
|
||
| `ClosureVar` flags | `u16` | leb128 |
|
||
| Opcodes | 244. `get_var` and its three siblings take a 3-byte `var_ref` index | 252. The same four take a 5-byte atom |
|
||
|
||
The pool order is the one that matters to a reader. ng moved it ahead of the
|
||
code on purpose — its own comment says *"write constant pool before code so
|
||
code can be disassembled on the fly at read time"* — so a `push_const` operand
|
||
resolves as soon as it is decoded.
|
||
|
||
## Appendix B — Verifying against a file
|
||
|
||
The listing tool prints one row per field, in the order given above:
|
||
|
||
```sh
|
||
cargo run -p qjsdump -- tests/bin/basic.bin
|
||
```
|
||
|
||
The parser is written in nom, one function per type in Part 1, so
|
||
`crates/qjs-bc/src/{parse,atoms,value,function,module}.rs` reads as an
|
||
executable form of this document. Indentation in the listing is depth in
|
||
`cpool`: a nested function and its sections are indented one level under the
|
||
parent whose constant pool holds them, which is also why a parent's `.code`
|
||
appears after, but less indented than, the whole of a child.
|
||
|
||
`tests/report/` holds a listing per fixture and `OPCODES.md`; `make`
|
||
regenerates both from `tests/src/`.
|