Reference

std/encoding/lib

std/encoding/src/lib.trb

Encoding and decoding: what other languages need reflection for (serialization, config mapping, database rows, schemas, --help texts).

const json = Json()
const text = json.encode user
const back = json.decode<User>(text)?

A value is its constructor call. The compiler knows every type's constructor and offers it in three forms: written (Encode), read (Decode) and described without a value (Describe). A format (Json, a database driver, a command line parser) implements Encoder, Decoder or Describer and never sees a type. There is no tree in between: a value is written while the type describes itself, and nothing is lost on the way.

All three forms are generated for a type whose constructor is usable from outside, over exactly the constructor's parameters: a private field with a default is not one of them, so a cache stays out without an annotation. What the compiler generates is ordinary code over the vocabulary, and could be written by hand:

extend User with Encode {
  fn encode<Target: Encoder>(var target: Target) {
    target.record "app/User"
    target.field "name"
    name.encode target
    target.field "tags"
    tags.encode target
    target.finish()
  }
}

The three traits take their target as a bound, not as a trait type: a format hands its own encoder over as a var path and reads the result out of it afterwards, and the derived code is a direct call chain with no table. EncodedValue is what a program holds where it has left the type behind, and Structure is a type's description as a value. docs/design/ENCODING.md is the design.

This package also has the plainer kind of encoding: bytes as text and back. Base64Alphabet, Base32Alphabet and Bytes.hexEncoded/String.hexDecoded/String.base64Decoded are RFC 4648, unrelated to Encode/Decode above - a value's own Encoder.bytes writes raw bytes, and a format reaches for one of these only where the wire itself is text (a JWT, a !!binary YAML scalar, a TOTP secret).

extend Bytes

extend Bytes

The operations on bytes and text that are RFC 4648's, written where they are imported from: a member another package names is found in the module of the package that is imported, so use Buffer.hexEncoded from "std/encoding" needs them here.

fn hexEncoded

fn hexEncoded(uppercase: Bool = false): String

These bytes as hexadecimal, two digits each, most significant nibble first: lower case unless uppercase.

extend String

extend String

The operations on text that read bytes back out of RFC 4648 text: hexadecimal and Base64. They are written here for the same reason as the ones on Bytes: a member is found in the module of the package that is imported.

Examples

print "48 65 6c 6c 6f".hexDecoded()
print "SGVsbG8".base64Decoded()

fn hexDecoded

fn hexDecoded(): Result<Bytes, EncodingError>

The bytes this hexadecimal text stands for, upper and lower case digits both read, mixed or not, with whitespace inside it skipped.

Errors

  • EncodingError for a character that is not a hexadecimal digit, or an odd number of them.

fn base64Decoded

fn base64Decoded(): Result<Bytes, EncodingError>

The bytes this Base64 text stands for, in either alphabet, padded or not, with whitespace inside it skipped - what a wrapped MIME block and a bare token both need read back.

Errors

  • EncodingError for a character outside both alphabets, padding before the end, or a length no encoder writes.

trait Encode

trait Encode

A value writes itself into any format.

Generated for a type whose constructor is usable from outside and whose passable fields are all Encode, and for a capsule through its conversion pair. The target is a bound, so the call is monomorphized per format.

Examples

type Point {
  x: Int
  y: Int
}

print Json().encode(Point(1, 2))

Related

  • Decode - the way back, derived under the same condition.
  • Encoder - what a format implements.

fn encode

fn encode<Target: Encoder>(var target: Target)

Writes this value into target: the type's constructor call as record or variant, field and the field's own encode, and finish; a scalar or a collection calls its one vocabulary method instead. Nothing is returned, the result is read out of the target afterwards.

Examples

type Point {
  x: Int
  y: Int
}

var target = Values()
const point = Point 1, 2
point.encode target
print target.result()

trait Decode

trait Decode

A value reads itself back. Derived under the same condition as Encode, so what is written can always be read.

A field with a default may be missing from the input, and the default is only evaluated then.

Related

  • DecodeError - what a failure says, with the field chain it happened in.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

Reads one value of this type out of source, asking the decoder for exactly what the constructor expects.

Errors

  • DecodeError when the source holds something else than the type asks for, a field it needs is missing, or a narrow number does not fit; the error names the path of fields and positions it happened at.

Examples

var source = ValueDecoder.of EncodedValue.Sequence([EncodedValue.Int(1), EncodedValue.Int(2)])
print List<Int>.decode(source)

trait Describe

trait Describe

A type describes its structure without a value: what schema-driven formats need (a DDL statement, a .proto file, a JSON Schema, a --help text). It is the same constructor again, with every field's doc comment and whether the field has a default.

Related

fn describe

static fn describe<Target: Describer>(var target: Target)

Describes this type into target, without a value: the same constructor as Encode.encode, walked as a shape. A type that contains itself is described once and referred to by name after that.

Examples

var target = Structures()
Int.describe target
print target.result()

type DecodeError

type DecodeError with Show, Error

What went wrong reading a value, with the whole chain of fields and positions it went wrong in: items[2].price.currency: a value is needed. A field is written after a ., a position of a sequence or a tuple in brackets, and an entry of a map after a . with its key.

A derived decode reads past a field that failed and fails once at the end: its error carries every failed field in problems, each with its own path, in the order the fields were read. Its own message and path are those of the first problem, so code that looks at one problem only still finds one.

Examples

const problem = DecodeError("a value is needed").inside("currency").inside("price").at(2).inside("items")
print problem

field message

message: String

What was wrong, without the place.

field path

path: List<String> = []

The fields and positions from the outermost value down to where it went wrong.

field problems

problems: List<DecodeError> = []

Every failed field of a derived decode that read on after the first one, each with its whole path. Empty for an error that stands for one problem only, such as one written by hand.

fn collecting

static fn collecting(problems: List<DecodeError>): Self

One error for the problems of a value, at least one: the first one's message and path, and all of them.

fn inside

fn inside(segment: String): Self

error.inside("email"): an error collects its path on the way out, outermost segment first.

fn each

fn each(): List<DecodeError>

The problems this error stands for: its problems, or the error itself where it stands for one.

fn at

fn at(position: Int): Self

error.at(2): the position of a sequence or a tuple the error happened in, shown as [2].

fn show

fn show(): String

One problem as items[2].name: a text is needed, and several as one such line each.

trait Encoder

trait Encoder

The writing half of a format. The scalar methods are named after the types of the language, so a format learns no second vocabulary; narrow numbers are widened on the way in and checked on the way back.

sequence, map, record and variant open, and finish closes the innermost of them. A record's field is announced with field and its value is written right after. A record that announces no field at all is a wrapper around the one value it holds, and a text format writes that value without anything around it.

Related

fn nothing

var fn nothing()

An absent value: None. A field written as nothing may be left out, and its default applies on the way back.

fn bool

var fn bool(value: Bool)

A truth value.

fn int

var fn int(value: Int64)

A signed integer; every narrower signed or unsigned type arrives here widened to an Int64.

fn unsigned

var fn unsigned(value: UInt64)

An unsigned integer; it has its own method because the upper half of a UInt64 does not fit into an Int64.

fn float

var fn float(value: Float64)

A floating point number; a Float32 arrives widened to a Float64.

fn decimal

var fn decimal(value: Decimal)

An exact decimal number; a format without one has to spell it, as Values does with its text.

fn string

var fn string(value: String)

A text, which also stands for a Char.

fn bytes

var fn bytes(value: Bytes)

Raw bytes, not text: a format chooses how its wire carries them.

fn sequence

var fn sequence(length: Int?)

length is known for a collection and unknown for a pipeline: a binary format writes it in front.

fn map

var fn map(length: Int?)

A map of length entries, each written as its key and then its value.

fn record

var fn record(typeName: String)

typeName is the qualified name of the declaration ("app/orders/Price"): the key a format finds a mapping by.

fn variant

var fn variant(typeName: String, name: String)

One case of a type with cases. Its fields follow as a record's do.

fn field

var fn field(name: String)

The name of the field whose value comes next.

fn finish

var fn finish()

Closes the innermost sequence, map, record or variant.

fn writesPrivateFields

fn writesPrivateFields(): Bool

Whether a derived encode writes the fields that are no parameter of the constructor from outside, too: a private field with a default. A format writes a type's data and answers false; a copy of the whole value answers true (Values(privateFields: true)), such as the value a native binary hands a script in its VM (std/sandbox, docs/design/SCRIPTS.md section 5). A field whose type is not Encode is left out either way.

fn flagsAsNames

fn flagsAsNames(): Bool

Whether a Flags is written as the list of the names of its cases (["Read", "Write"]) and not as its mask (3). A format answers false, the number, unless it is told otherwise (Json(flagsAsNames: true)): how a set of flags looks on the wire is a convention of the format, not a question the type asks.

trait Decoder

trait Decoder

The reading half of a format. Reading is driven by the type: it asks for what it expects, and gets a DecodeError when something else is there. A format with unordered fields buffers one record at a time; a sequence streams.

Pitfalls

hasNext both tests and positions: it answers true while one more item of the open sequence (or one more key or value of the open map) follows, and the next read takes it.

fn nothing

var fn nothing(): Bool

Consumes a "nothing" if one is there. This is how Option decides between None and Some.

fn bool

var fn bool(): Result<Bool, DecodeError>

Reads a truth value, or fails when the input holds something else.

fn int

var fn int(): Result<Int64, DecodeError>

Reads a signed integer as an Int64; a narrower type checks the range itself afterwards.

fn unsigned

var fn unsigned(): Result<UInt64, DecodeError>

Reads an unsigned integer as a UInt64, or fails for a negative or non-numeric input.

fn float

var fn float(): Result<Float64, DecodeError>

Reads a floating point number as a Float64.

fn decimal

var fn decimal(): Result<Decimal, DecodeError>

Reads an exact decimal number, or fails when the input is not one.

fn string

var fn string(): Result<String, DecodeError>

Reads a text.

fn bytes

var fn bytes(): Result<Bytes, DecodeError>

Reads raw bytes.

fn sequence

var fn sequence(): Result<Int?, DecodeError>

Opens a sequence and answers its length where the format writes one in front.

fn map

var fn map(): Result<Int?, DecodeError>

Opens a map. Its entries are read as a key, then a value, while hasNext answers true.

fn hasNext

var fn hasNext(): Result<Bool, DecodeError>

true while one more item of the open sequence or one more key or value of the open map follows.

fn record

var fn record(typeName: String): Result<Void, DecodeError>

Opens a record. A wrapper's record holds one value and no field.

fn variant

var fn variant(typeName: String): Result<String, DecodeError>

Opens a variant and answers which case is there.

fn field

var fn field(name: String): Result<Bool, DecodeError>

Positions on the field name. false when the input has no such field: the field's default applies.

fn finish

var fn finish(): Result<Void, DecodeError>

Closes the innermost sequence, map, record or variant.

fn readsPrivateFields

fn readsPrivateFields(): Bool

Whether a derived decode reads the fields that are no parameter of the constructor from outside, too, where the input has them: the other half of Encoder.writesPrivateFields. A field the input does not have takes its default.

fn flagsAsNames

fn flagsAsNames(): Bool

Whether a Flags is read from the list of the names of its cases and not from its mask: the other half of Encoder.flagsAsNames. With the option set only the names are read; a number is a DecodeError.

trait Describer

trait Describer

The describing half: the same method names as Encoder, without values. record and variant answer a Bool: false means "this type is already known here", which is what makes a type that contains itself terminate - the name has been handed over before the answer, so the target can refer to the type it declined.

Related

fn bool

var fn bool()

The type is a truth value.

fn int

var fn int()

The type is a signed integer; narrow signed and unsigned types describe themselves this way, too.

fn unsigned

var fn unsigned()

The type is an unsigned 64-bit integer.

fn float

var fn float()

The type is a floating point number.

fn decimal

var fn decimal()

The type is an exact decimal number.

fn string

var fn string()

The type is a text; a Char is one, too.

fn bytes

var fn bytes()

The type is a list of raw bytes.

fn optional

var fn optional()

The description that follows is the one that may be absent. finish closes it.

fn sequence

var fn sequence()

A sequence: the one description that follows is its item's. finish closes it.

fn map

var fn map()

A map: the two descriptions that follow are its key's and its value's. finish closes it.

fn record

var fn record(typeName: String): Bool

Opens a record. false: this target knows the type already and does not want the body again.

fn variant

var fn variant(typeName: String): Bool

Opens a type with cases, as record does.

fn variantCase

var fn variantCase(name: String)

One case of the open variant. Its fields follow, finish closes it.

fn field

var fn field(description: FieldDescription)

The next description belongs to this field.

fn finish

var fn finish()

Closes the innermost open description.

trait Format

trait Format<Failure>

What a whole format is, once: bytes in both directions, and the two Stages that make it work on a stream.

Encode/Decode and Encoder/Decoder stay synchronous, and that is the point. A decoder that could wait would have to be written differently for every source, every derived implementation would have to change with it, and the whole standard library would be coloured by it. Streaming happens one level up, at the level it happens in practice: the element.

body.through(Json.items<User>()).checked()          // bytes in, one `User` at a time out, memory = one user
Source.from(users).through(Json.encoded<User>())    // and back

items is a resumable framer: it finds where one element ends in whatever bytes have arrived - across chunk borders - and decodes that element synchronously, through the ordinary Decode.

Failure is the type parameter because the language has no associated types: Json is a Format<JsonError>, and a function that works for any format says fn load<Chosen: Format<Problem>, Problem>(...). The members are static and use the format's default options; a format value with options has its own encode and decode.

fn encodeAll

static fn encodeAll<Value: Encode>(value: Value): Bytes

The whole value, as the bytes that go into a file or onto a wire.

fn decodeAll

static fn decodeAll<Value: Decode>(bytes: Bytes): Result<Value, Failure>

One whole value out of all of its bytes.

fn items

static fn items<Item: Decode>(): Stage<Bytes, Result<Item, Failure>>

Bytes to items. Fallible, so the items are Results - a Stage is synchronous and knows nothing about the failure of the stream around it; source.checked() is where they become one.

fn encoded

static fn encoded<Item: Encode>(): Stage<Item, Bytes>

Items to bytes: the other direction, and the one that never fails.

extend Bool with Encode, Decode, Describe

extend Bool with Encode, Decode, Describe

Bool is the vocabulary's own bool, in all three forms.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Int64 with Encode, Decode, Describe

extend Int64 with Encode, Decode, Describe

Int64 is the vocabulary's own int, in all three forms.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Int8 with Encode, Decode, Describe

extend Int8 with Encode, Decode, Describe

A narrow integer travels as an Int64 and is checked on the way back: a value that does not fit is an error.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Int16 with Encode, Decode, Describe

extend Int16 with Encode, Decode, Describe

A narrow integer travels as an Int64 and is checked on the way back: a value that does not fit is an error.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Int32 with Encode, Decode, Describe

extend Int32 with Encode, Decode, Describe

A narrow integer travels as an Int64 and is checked on the way back: a value that does not fit is an error.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend UInt8 with Encode, Decode, Describe

extend UInt8 with Encode, Decode, Describe

A narrow unsigned integer travels as an Int64, which holds every one of its values, and is checked on the way back.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend UInt16 with Encode, Decode, Describe

extend UInt16 with Encode, Decode, Describe

A narrow unsigned integer travels as an Int64, which holds every one of its values, and is checked on the way back.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend UInt32 with Encode, Decode, Describe

extend UInt32 with Encode, Decode, Describe

A narrow unsigned integer travels as an Int64, which holds every one of its values, and is checked on the way back.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend UInt64 with Encode, Decode, Describe

extend UInt64 with Encode, Decode, Describe

UInt64 is the vocabulary's own unsigned: its upper half does not fit into an Int64.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Float32 with Encode, Describe

extend Float32 with Encode, Describe

A Float32 travels as a Float64, which holds it exactly. It has no Decode yet: one would read the Float64 and narrow it with Float32.tryFrom, which rounds to the nearest Float32 and fails only beyond its range.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Float64 with Encode, Decode, Describe

extend Float64 with Encode, Decode, Describe

Float64 is the vocabulary's own float, in all three forms.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Decimal with Encode, Decode, Describe

extend Decimal with Encode, Decode, Describe

Decimal is the vocabulary's own decimal: an exact number stays exact in a format that can say so.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Char with Encode, Decode, Describe

extend Char with Encode, Decode, Describe

A Char travels as a text of one character: the formats of the language have no character of their own.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend String with Encode, Decode, Describe

extend String with Encode, Decode, Describe

String is the vocabulary's own string, in all three forms.

fn encode

fn encode<Target: Encoder>(var target: Target)

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Option<Value> with Encode

extend<Value: Encode> Option<Value> with Encode

Some is its value, and None is nothing() - which a field may leave out, so its default applies.

fn encode

fn encode<Target: Encoder>(var target: Target)

extend Option<Value> with Decode

extend<Value: Decode> Option<Value> with Decode

A "nothing" is None, and anything else is the value's own decode.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Value?, DecodeError>

extend Option<Value> with Describe

extend<Value: Describe> Option<Value> with Describe

An optional value describes as optional() around the value's own structure.

fn describe

static fn describe<Target: Describer>(var target: Target)

extend List<Item> with Encode

extend<Item: Encode> List<Item> with Encode

A list is a sequence: its length in front, then every item with its own encode.

fn encode

fn encode<Target: Encoder>(var target: Target)

extend List<Item> with Decode

extend<Item: Decode> List<Item> with Decode

A sequence, item by item, each through the item's own decode. An error names the position it happened at, and an item that failed does not stop the ones after it: the list fails at its end, with every item's problems.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

extend List<Item> with Describe

extend<Item: Describe> List<Item> with Describe

A list describes as a sequence around its item's structure: the shape is one item, repeated.

fn describe

static fn describe<Target: Describer>(var target: Target)

extend ArrayList<Item> with Decode

extend<Item: Decode> ArrayList<Item> with Decode

An ArrayList reads back as the list it is written as - a sequence, item by item - so a field of one does not keep the type around it from having a derived decode (Chunk of the bytecode, whose code is one). It writes through the encode of List.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

extend ArrayList<Item> with Describe

extend<Item: Describe> ArrayList<Item> with Describe

An ArrayList describes as the sequence of List.

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Buffer<Item> with Decode

extend<Item: Decode> Buffer<Item> with Decode

A Buffer reads back as the list it is written as, for the reason an ArrayList does: a field of one, Bytes in the first place, does not keep the type around it from having a derived decode.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

extend Buffer<Item> with Describe

extend<Item: Describe> Buffer<Item> with Describe

A Buffer describes as the sequence of List.

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Flags<Case> with Encode

extend<Case: RawValue & Encode> Flags<Case> with Encode

Flags are their mask, a number, unless the target asks for names (Encoder.flagsAsNames): then they are a sequence of the case names, each through the case's own encode, the lowest bit first.

fn encode

fn encode<Target: Encoder>(var target: Target)

extend Flags<Case> with Decode

extend<Case: RawValue & Decode> Flags<Case> with Decode

The way back of Flags' encode: a mask through fromBits, which refuses a bit no case stands for, or - where the source asks for names (Decoder.flagsAsNames) - a sequence of case names, each through the case's own decode. A name that is no case, and a value that is no sequence, are a DecodeError with the place it happened at.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

extend Set<Item> with Encode

extend<Item: Encode> Set<Item> with Encode

A set is a sequence of its items.

fn encode

fn encode<Target: Encoder>(var target: Target)

extend Set<Item> with Decode

extend<Item: Decode & Hash> Set<Item> with Decode

A sequence read into a set: an item that is there twice is there once.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

extend Set<Item> with Describe

extend<Item: Describe> Set<Item> with Describe

A set describes as a sequence of its item.

fn describe

static fn describe<Target: Describer>(var target: Target)

extend Map<Key, Value> with Encode

extend<Key: Encode, Value: Encode> Map<Key, Value> with Encode

A map is a map: its length in front, then every entry as its key and its value.

fn encode

fn encode<Target: Encoder>(var target: Target)

extend Map<Key, Value> with Decode

extend<Key: Decode & Hash & Show, Value: Decode> Map<Key, Value> with Decode

A map, entry by entry: a key, then its value. An error in a value names the key it belongs to, and the entries after it are read all the same, as a list's items are.

fn decode

static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>

extend Map<Key, Value> with Describe

extend<Key: Describe, Value: Describe> Map<Key, Value> with Describe

A map describes as a map around its key's and its value's structure.

fn describe

static fn describe<Target: Describer>(var target: Target)