Foreign Functions#

Status: designed, not built (2026-10-24). — CONCEPT ("Foreign Functions (Draft)") sketches foreign: a block of C functions, Pointer, CString, a library per platform in project.trb. Of that, the shape of a foreign block is checked today (checkForeignShape, compiler/src/semantics/checker/declaration.trb) and the capability "foreign functions" is derived from source (compiler/src/package/archive.trb); nothing links, loads or calls anything, and Pointer and CString are not declared in std/. This record fixes the types, the checker rules, how each back end reaches C, how the sandbox, the registry and the destructors treat it, and cuts the work into slices (section 10). It is step 1 of #84. The one thing it does not decide is RELEASE.md section 2 ("whether a registry package may declare foreign functions, and whether it may carry C"): that is a published contract, so section 5.3 sets out the options and stays open, owner decision (#84). COMPUTE.md (std/gpu) and FIRST-PARTY.md row 7 (torb/sqlite) are what the design serves; slices 1 to 4 of section 10 do not depend on that answer.

foreign is where TorbScript stops guaranteeing anything, so the design keeps the place small and visible. A C function can write through any address, keep a pointer it was given and call back from any thread. Nothing the type checker knows survives the call. So the rules below do not try to make C safe; they keep what C returns from becoming a value (it stays behind a shared type that closes it), keep what C holds from crossing a task, and make the capability show wherever a package is looked at.

1. What is decided#

• A foreign declaration is a signature and nothing else, and every type in it is spelled. No inference, no defaults, no overloads (the checker already demands the result type). The C ABI of the platform is the contract.
• Both back ends load the library at run time, through one runtime function, on the first call — the compiled binary as well as the VM. CONCEPT says "the compiler links directly"; this record changes that to one behaviour for both (section 4.2). A missing library or symbol is one panic that names the library, the files tried and the symbol.
• The VM reaches C through one more kernel operation, ForeignCall, a row of the kernel's table in runtime/machine.c and a constant of the emitter. It is not a native: the manifest does not change, so no seed refresh (VM.md section 6, "One native, many operations").
• Pointer<Value> and CString are not values in the language's sense. A type that contains one is confined to the task that made it, by the mechanism that already confines a shared type (CONCURRENCY section 1), may not appear in a public signature, and is not Equal, Hash or Encode.
• Foreign memory is owned by a shared type with Close, and close() is the only place that frees it (DESTRUCTORS sections 2 and 6).
• Foreign functions are never granted to a script: not by a Sandbox, not by modules, not by a per-system grant (the-sandbox rule 8, SCRIPTS capability table, OS.md section 8).
• foreign in a published package is not decided here. Whether a registry package may declare foreign functions, and whether it may carry C, is the owner's decision (#84); section 5.3 has the options and a recommendation. Whatever is chosen, the capability stays visible in source, in torb add and in the lock, and is never grantable to a sandboxed script.
• foreign exists on the targets that have a C ABI. A program that reaches one for the JavaScript or PHP target is a compile error on that target, the way a native that does not exist there is (OS.md section 2, availableOn); what foreign might become there stays with #63 (JAVASCRIPT-AND-PHP.md).
• No unsafe keyword and no new reserved word. foreign is the marker, as native is for std/.

2. What exists#

The last row is the reason for section 6.2: the thunk table is generated when torb is built, but a foreign signature is written by whoever writes the script.

3. Types and the ABI#

3.1 What may cross#

Anything else (a String, a List, a closure with captures, a Result) is an error at the declaration: the message names the type and the table.

3.2 Pointer<Value>#

A Pointer is one machine word. It has no arithmetic and no dereference in ordinary code: reading and writing through one are functions of std/foreign (Pointer.read, Pointer.write, Pointer.offset(by:)), each an operation of the kernel with the trust of Machine.load (VM.md section 6: "a wrong register offset is a wrong write"), which is why std/foreign is a package a sandbox cannot import (the std/machine rule of VM.md section 6). Pointer.none is the null pointer; there is no Pointer?, because C's null is a pointer. A function that can answer null documents it, and the binding turns that into Result or Option at the edge.

3.3 CString#

CString(text) copies a String into a counted block with a terminating NUL and answers Result<CString, CStringError>: a text with a NUL inside is refused, because C would read it as shorter. A CString argument is borrowed for the call. A CString a C function answers is a borrowed view of C's memory: toString() copies it (and fails on invalid UTF-8) and nothing frees it, so the binding frees it with the library's own function where the library says to.

3.4 foreign type: the C layout#

A foreign type is a record whose layout is the C compiler's for the target, not the IR's:

1. Field order is declaration order, as in C. The IR never reorders the fields of a foreign type.
2. Each field is aligned to its natural alignment (Int8 1 to Int64/Float64/Pointer 8 on the 64-bit tier 1 targets) and the whole type to its largest field; padding is inserted as C inserts it and is never written to, so a foreign type is not Equal or Hash (F3).
3. Fields are the types of section 3.1 and nested foreign types. No String, no List, no generics: a foreign type has no type parameters and cannot be one. A fixed-size array field is Array<Item, Size> of a section 3.1 item, laid out inline (VM.md section 4: FixedArray).
4. No packed or bit-field layout in this design. A library that needs one is bound through accessor functions of its own or through Pointer.read at byte offsets (section 3.2).
5. The size and alignment are computed once, by the checker, per target from rules 1 to 3 and stored with the type; the C back end additionally emits a _Static_assert of both against sizeof/_Alignof of the C type it declares, so a mismatch is a build failure on the generated C and not a silent wrong offset.

3.5 A Buffer<Item> handed to C#

Buffer<Item> (COLLECTIONS.md, gap 6) is contiguous heap storage. A foreign function takes one only as Pointer<Item> obtained by buffer.pinned { pointer, length => ... }, a function of std/foreign:

1. The pointer is valid for the duration of the closure and for nothing else. The closure is the pin: while it runs the buffer is not moved, resized, freed or written by TorbScript code; the pointer is Pointer<Item>, which F2 confines to the task, so it cannot escape into a field or a channel.
2. A pin of a var window (Window<Item>, COMPUTE.md) is a writable pin; a pin of a value is read-only, and C writing through it is the binding's contract violation, as everywhere in this section.
3. One call in flight. The pin lasts one closure, and a pointer a C library keeps beyond it (a command buffer that reads it later) is the binding's business (section 7); for that the binding copies into C's own block.
4. Item is a section 3.1 type (a sized number or a foreign type); a Buffer<String> has no C layout and is refused with the message of section 3.1.

3.6 Errors: errno, GetLastError and null#

C reports failure out of band. The binding turns it into Result at the edge, with two functions of std/foreign that read the value of the calling thread and are valid only directly after the call:

• Foreign.lastError(): Int32 is errno on POSIX and GetLastError() on Windows. A foreign call does not clear it, so the binding reads it only when the function's own result says it failed (-1, a null pointer).
• A null pointer result is Pointer.none; pointer.orNone() answers Option<Pointer<Value>>, and the binding writes match into Result<Handle, LibraryError> with lastError() as the code.
• The runtime saves the error value right after the foreign call returns and before it releases anything, so the releases of the call (a CString block) cannot overwrite it before lastError() is read.

4. Declarations and how a library is found#

4.1 The block and the manifest#

foreign "sqlite3" {
  fn sqlite3_open(path: CString, database: Pointer<Pointer<Void>>): Int32
  fn sqlite3_close(database: Pointer<Void>): Int32
}

"sqlite3" is a logical name. project.trb of the package maps it per platform (CONCEPT; PROJECT.md section 1 keeps "native and foreign" out of manifest switches, so the mapping is the one thing the manifest says and the block in source is what makes it required):

foreign {
  library "sqlite3", windows: "sqlite3.dll", linux: "libsqlite3.so.0", macos: "libsqlite3.dylib"
}

A logical name with no row for the platform being built is a compile error for that target, not a load failure later; a row with no block that uses it is a warning. The file name is what the system loader gets, so the search path is the system's. An absolute path is not accepted: it would make the manifest machine-specific.

4.2 Loading — one behaviour in both back ends#

Options for the compiled binary: (a) link at build time (-lsqlite3, an import library on Windows); (b) load at run time like the VM must. Decided (b):

• (a) needs a sysroot with every foreign library's import library or shared object on the build machine, which the managed zig cc of RELEASE section 4 does not have for a library the project does not ship, and it breaks torb build --target for another machine;
• (a) fails before main with a loader message, (b) fails at the call with a panic that names what was missing;
• the VM cannot link, so with (a) the two back ends would differ in when a missing library is found, which CONCEPT says must not be observable;
• the runtime already loads with LoadLibraryW; dlopen is the same function on the other systems.

The runtime gets torb_library_open(file) and torb_library_symbol(library, name) (internal, not natives; POSIX in runtime/os/posix.c, Windows in runtime/os/windows.c, OS.md section 7). A library is opened once per process and never unloaded; a symbol is resolved once and cached in a slot (a static in generated C, an entry of the program's foreign table in the VM). A back end may later link directly where it can prove the library is there, which is its private business.

Because loading is lazy and lowering only reaches what the entry function reaches (OS.md section 2), a foreign declaration reached only through an arm of match OperatingSystem.current the target does not take is not emitted, and its library is not opened.

5. The checker, the sandbox and the registry#

5.1 Rules#

F2 is the one rule C cannot give a library: a handle is used by one task on one worker. A library that is not thread safe is therefore safe by default, and two workers may each hold their own handle. A library whose handles must stay on one OS thread (a GUI toolkit) is not served by this design (section 9).

5.2 The sandbox#

A script has no foreign functions, ever. Two checks, because the first is source analysis and the second is the machine:

1. Sandbox.load refuses a script, or a module it imports, that contains a foreign block, with the diagnostic the refused grant of std/os/windows/registry already uses (OS.md section 8);
2. the kernel's ForeignCall refuses to run while a sandbox is open and stops the script, the way Machine.countUp answers false for every count while a sandbox is open (VM.md section 10). The first is the diagnostic, the second is why a bug in the first is not an escape.

5.2.1 --embed-vm binaries and torb test#

• An --embed-vm binary (VM.md section 11) interprets the whole program, so a foreign call there is the VM's ForeignCall (section 6.2) and the library is loaded by the runtime inside the binary, at run time, from the file names the manifest mapping recorded at build time. It needs no compiler where it runs, and also no library bundled: the library must be on the machine that runs it, exactly as for a binary torb build compiles. A script loaded by such a binary is a script and gets no foreign (5.2).
• torb test runs the program's tests under the toolchain's VM or as a compiled binary, as for any program, so the libraries must be installed where the tests run; a test that needs a library that may be absent uses the probe of section 11 item 3 and skips with a stated reason. The repository's own gate is the CI test library of slice 3, which CI builds, so it never depends on a system library.

5.3 Foreign functions in published packages — open, owner decision (#84)#

RELEASE section 2 names this as a 1.0 decision and FIRST-PARTY row 7 (torb/sqlite) cannot be built until it is made. It is a published contract of the registry, so this record does not decide it. Open, owner decision (#84). The options:

Recommendation: option 2. It keeps the registry a store of source that runs and builds nothing, and leaves option 3 open.

What holds under every option, and so is part of this record: foreign is visible in source; the package summary shows foreign functions with the files that declare them (RELEASE section 7.8); torb add and the lock show it; the registry never loads a library (torb doc --check --no-run runs none of a package's code, and the documentation worker of RELEASE section 7.11 never calls a foreign function); the 9 MB native of RELEASE section 7.11 stays refused.

The answer is written by a follow-up PR into this section, into RELEASE.md section 2 and 7.2 and into FIRST-PARTY.md rows 7 and 9. Until then RELEASE.md and FIRST-PARTY.md say "open, see FOREIGN.md 5.3". Slices 5 (the library mapping, which a local project needs as well) and 8 (torb/sqlite) of section 10 are the ones that wait.

6. The two back ends#

6.0 A signature a back end cannot call#

BACKEND gap 22 (a missing intrinsic is a finding, never a link error or a wrong result) applies: a foreign signature that a back end cannot call is a compile error that names the function, the type or arity that is not supported and the back end, located at the declaration, and never a link error, a runtime crash or a wrong call. The check runs in lowering, which knows the target: F1 is the language-wide rule, this is the per-back-end one (the VM stubs of section 6.2 refuse a foreign type by value and an arity above the cap; the JavaScript and PHP targets refuse every foreign function, section 1).

6.1 The C back end#

A call is a typed call through the slot: the generated C declares the function type from the signature, the slot is a static void (*)(void) filled by torb_library_symbol and cast at the call. The ABI is the C compiler's own, so every signature section 3.1 allows is right by construction, struct by value included. A CString is the borrowed pointer of its block; a Pointer is a word.

6.2 The VM#

ForeignCall takes a library index, a symbol index and a signature index of the program's tables and the registers of the operands, loads and caches the symbol, and calls it. The thunks of VM.md section 6 cannot do this: they are one per known native. Two ways to call a function whose signature is only known at run time:

Decided: the stubs, scalars only first (integer-class and float-class arguments up to an arity cap, the result in one register), because the C back end covers the rest and slice 3 can ship without the VM. A spike in slice 4 is the check: if a stub per ABI costs more than a week or has to know struct classification, the decision is revisited with libffi on the table, and this record says so then.

The VM check and the CI gate of VM.md section 8 ("C and the VM agree") extend to it: a small C library is built in CI, its functions are called from the same test program under both back ends, and the outputs are compared.

7. Memory, destructors and callbacks#

• C's memory is freed by C's function, called from close(). The shape is CONCEPT's Database with var fn close() { sqlite3_close(handle) }. close() is synchronous, returns Void, never fails and runs on whatever release is the last (DESTRUCTORS sections 2 and 6), so a library whose close can fail is wrapped with an end() in front, the way Sink is.
• A Pointer never outlives its block unnoticed: a CString made for a call is released after the call, a pointer a C library keeps beyond the call (sqlite3_bind_text with a static destructor) is the binding's business, and the record repeats it in the docs page because the checker cannot see it.
• A fault inside C is not recovered. A signal in C (a segmentation fault, SIGBUS, a Windows structured exception) ends the process with the platform's own exit status; the runtime installs no handler for it, and the recoverable panic of SCRIPTS.md and PANICS.md does not catch it, because a jump out of a faulting C frame promises nothing about the library's state. A foreign function that returns an error code is the only recoverable failure. The memory counts of the leak report do not include C's allocations (section 7, first bullet).
• A panic inside a callback ends the program like a panic with no recovery point. It is not unwound through C frames, because the runtime's recovery is a jump and C code gives no promise about being jumped over.
• Callbacks (slice 7). C back end: a closure without captures is a C function of the right signature. VM: a small fixed pool of native entry points per signature shape that call the interpreter through Machine.install (VM.md section 10, "A call from the runtime into the interpreter"). A callback is only valid on the thread of the foreign call that received it; one from another thread aborts with a message that names the function.

8. What std/gpu and std/render need from this record#

Constraints on the design (COMPUTE.md layer 0 and 2), not a design of those packages. Each is covered or deferred:

9. What this is not#

• Not a way to run C compiled from a package. This record adds no build.rs and no vendored sources; whether a published package may carry C is the open question of section 5.3.
• Not a capability a sandbox can grant, and not a way around std/os: the per-system natives of OS.md stay natives of runtime/, and a user's foreign call to getpid is a user's choice that the package summary shows.
• Not blocking-safe. A foreign call runs on the calling worker and stalls its other tasks for as long as it takes (CONCURRENCY section 1: one event loop per worker). A foreign marker blocking, run on a helper thread, is the likely answer and is open (section 11).
• Not for thread-affine libraries, because of F2: a handle stays on its task's worker, but the worker is chosen at spawn and nothing here pins it to the main thread.
• Not native with a body. NATIVE-BODIES.md is IR for the kernel, reserved for std/; foreign is C for everybody. Pointer's operations are kernel operations written that way once it is built (before that, manifest natives, slice 1).

10. The slices#

Slices 1 to 3 give a working foreign on the compiled path; 4 gives torb run; 5 adds the manifest mapping (a package of the registry additionally needs the answer of section 5.3). No slice needs the compiler to compile itself, but 1 is cheaper after NATIVE-BODIES slice 1.

11. Open, for the owner#

1. A library the machine does not have. Windows has no system SQLite. Options: the download bundles sqlite3.dll (RELEASE section 4), the package documents the install, or C in packages is decided after all (option 3 or 4 of section 5.3). Recommendation: the first for SQLite only, as part of the release, until section 5.3 is settled.

2. A blocking marker on a foreign function that moves the call to a helper thread. Recommendation: yes, as a slice after 4, since SQLite and every network library need it.

3. A probe for an optional library (Foreign.available("sqlite3")). Recommendation: yes, a function of std/foreign, because a panic on the first call is the wrong answer for a feature that may be absent.

4. libffi if the spike fails (section 6.2).

5. CONCEPT's "the compiler links directly" is replaced by section 4.2: a technical choice, decided here; the owner may overrule it on review.

6. Foreign functions in published packages (section 5.3): options 1 to 4, recommendation 2. Tracked in #84; a merge of this record is not the answer.

7. An indirect foreign call (a Pointer<Void> target plus a signature), which function-pointer tables (section 8) need. Recommendation: yes, with slice 6; the C back end casts, the VM needs a stub per signature shape.

8. A main-thread pin for libraries that must run on the process's first thread (windows, graphics contexts). Recommendation: a foreign marker main that runs the call on the thread main runs on, as a slice after 4; std/render waits for it.

12. Where it is written down#

• Concept: CONCEPT.md, "Foreign Functions (Draft)".
• The language page: Foreign functions, rules 1 to 8.
• The question it leaves open: RELEASE.md section 2 and 7.8; the capability table is SCRIPTS.md and docs/tooling/package-capabilities.md.
• Neighbours: NATIVE-BODIES.md, VM.md sections 6 and 10, OS.md sections 2 and 8, DESTRUCTORS.md, CONCURRENCY.md section 1, JAVASCRIPT-AND-PHP.md, PROJECT.md section 1.