DE
Referenz
  1. The standard library

std/encoding

Encode, Decode and Describe, the traits a format implements, EncodedValue and Structure, Format for streaming, and RFC 4648's Base64, Base32 and hexadecimal for bytes as text (hexEncoded(), hexDecoded(), base64Decoded() methods).

Diese Seite gibt es noch nicht auf Deutsch, deshalb steht sie hier auf Englisch. Zur englischen Seite

Die generierte Referenz von std/encoding zeigt jede öffentliche Deklaration mit Signatur, Doc-Kommentar und Beispielen.

std/encoding is what other languages need reflection for: serialization, configuration mapping, database rows, schemas. A value is its constructor call, and the compiler offers that call in three forms - written (Encode), read (Decode) and described without a value (Describe). A format implements Encoder, Decoder or Describer and never sees a type, so there is no tree in between. All three are generated for a type whose constructor is usable from outside, over exactly its parameters; the generated code is a straight line of the steps this package declares, and could be written by hand. The traits, DecodeError, EncodedValue and rendered are in scope through the prelude.

This package also has the plainer kind of encoding, unrelated to the three forms above: bytes as text and back, RFC 4648's Base64, Base32 and hexadecimal - for a format whose wire is itself text (a JWT, a !!binary YAML scalar, a TOTP secret), not for a value's own Encoder.bytes. Hexadecimal and Base64 decoding are methods: bytes.hexEncoded(), text.hexDecoded(), text.base64Decoded(). A file names the methods it uses, written with the type they extend: use Buffer.hexEncoded, String.hexDecoded, String.base64Decoded from "std/encoding".

Import#

use Encode, Decode, Describe, DecodeError, Format, rendered from "std/encoding"
use Encoder, Decoder, Describer, EncodedValue, Structure, structureOf, Naming from "std/encoding"
use Base64Alphabet, Base32Alphabet, EncodingError from "std/encoding"
use Buffer.hexEncoded, String.hexDecoded, String.base64Decoded from "std/encoding"
use structureOf from "std/encoding"

type Point {
  x: Int
  y: Int
}

const point = Point x: 1, y: 2
print rendered(point)
print EncodedValue.of(point)
print structureOf<Point>()

Declarations#

Encode, Decode, Describe#

trait Encode {
  fn encode<Target: Encoder>(var target: Target)
}
trait Decode {
  static fn decode<Source: Decoder>(var source: Source): Result<Self, DecodeError>
}
trait Describe {
  static fn describe<Target: Describer>(var target: Target)
}

encode writes self into whatever format target is; decode is a static fn through Type.decode(source) and answers the failure a format's decoder produced; describe walks the structure without a value. The format is a type parameter, so every call is monomorphized per format and direct. A type whose constructor is closed from outside (a capsule) is written, read and described as the source of its conversion pair (the one with String where there are several).

DecodeError#

type DecodeError with Show, Error {
  message: String
  path: List<String> = []
  problems: List<DecodeError> = []
  static fn collecting(problems: List<DecodeError>): Self
  fn inside(segment: String): Self
  fn each(): List<DecodeError>
  fn at(position: Int): Self
  fn show(): String
}

error.inside("email"): a DecodeError collects the field chain on its way out of nested records, sequences and maps, so show() renders "user.email: a text is needed" rather than losing where the field was.

Encoder, Decoder, Describer#

trait Encoder {
  var fn nothing()
  var fn bool(value: Bool)
  var fn int(value: Int64)
  var fn unsigned(value: UInt64)
  var fn float(value: Float64)
  var fn decimal(value: Decimal)
  var fn string(value: String)
  var fn bytes(value: Bytes)
  var fn sequence(length: Int?)
  var fn map(length: Int?)
  var fn record(typeName: String)
  var fn variant(typeName: String, name: String)
  var fn field(name: String)
  var fn finish()
  fn writesPrivateFields(): Bool
  fn flagsAsNames(): Bool
}
trait Decoder {
  var fn nothing(): Bool
  var fn bool(): Result<Bool, DecodeError>
  var fn int(): Result<Int64, DecodeError>
  var fn unsigned(): Result<UInt64, DecodeError>
  var fn float(): Result<Float64, DecodeError>
  var fn decimal(): Result<Decimal, DecodeError>
  var fn string(): Result<String, DecodeError>
  var fn bytes(): Result<Bytes, DecodeError>
  var fn sequence(): Result<Int?, DecodeError>
  var fn map(): Result<Int?, DecodeError>
  var fn hasNext(): Result<Bool, DecodeError>
  var fn record(typeName: String): Result<Void, DecodeError>
  var fn variant(typeName: String): Result<String, DecodeError>
  var fn field(name: String): Result<Bool, DecodeError>
  var fn finish(): Result<Void, DecodeError>
  fn readsPrivateFields(): Bool
  fn flagsAsNames(): Bool
}
trait Describer {
  var fn bool()
  var fn int()
  var fn unsigned()
  var fn float()
  var fn decimal()
  var fn string()
  var fn bytes()
  var fn optional()
  var fn sequence()
  var fn map()
  var fn record(typeName: String): Bool
  var fn variant(typeName: String): Bool
  var fn variantCase(name: String)
  var fn field(description: FieldDescription)
  var fn finish()
}

The methods are named after the types of the language, and a narrow number is widened before it is written, so nothing about a format's own numbers is lost. sequence, map, record and variant open, and finish closes the innermost of them; a record announces each field with field and writes its value right after. A record that announces no field is a wrapper around its one value. Every method is free of type parameters and closures, so a format is a straight-line state machine. typeName is the declaration's qualified name - package, module path, type - which is the key a format finds a mapping for one type by.

Reading is driven by the type: it asks the decoder for what it expects and gets a DecodeError if something else is there. nothing() is how Option decides between None and Some, and field(name) answers false for a field the input does not have, so the field's default applies. Describer.record answers false for a type the target has seen already, which is what makes a type that contains itself describe exactly once.

flagsAsNames() is how a Flags asks its format how to look on the wire. A format answers false unless it is told otherwise, and then a Flags is its mask, a number. With the option set (Json(flagsAsNames: true), Yaml(flagsAsNames: true)) a Flags is the sequence of its case names, lowest bit first, each written by the case's own encode, and read back through the case's own decode: a name that is no case, and a value that is not a sequence, are a DecodeError. The query is an option of the format, like naming, and not something a type decides.

EncodedValue and the shared decoder#

type EncodedValue with Show {
  case Nothing
  case Bool(value: Bool)
  case Int(value: Int64)
  case Unsigned(value: UInt64)
  case Float(value: Float64)
  case Exact(value: Decimal)
  case String(value: String)
  case Bytes(value: Bytes)
  case Sequence(items: List<Self>)
  case Mapping(entries: List<Self>)
  case Record(typeName: String, fields: List<EncodedField>)
  case Variant(typeName: String, name: String, fields: List<EncodedField>)
  static fn of<Value: Encode>(value: Value, privateFields: Bool = false): Self
  fn show(): String
  fn plain(): String
}
extend EncodedValue with Encode
fn rendered<Value: Encode>(value: Value): String
type Values with Encoder {
  privateFields: Bool = false
  fn writesPrivateFields(): Bool
  fn result(): EncodedValue
  var fn nothing()
  var fn bool(value: Bool)
  var fn int(value: Int64)
  var fn unsigned(value: UInt64)
  var fn float(value: Float64)
  var fn decimal(value: Decimal)
  var fn string(value: String)
  var fn bytes(value: Bytes)
  var fn sequence(length: Int?)
  var fn map(length: Int?)
  var fn record(typeName: String)
  var fn variant(typeName: String, name: String)
  var fn field(name: String)
  var fn finish()
}
type ValueDecoder with Decoder {
  static fn of(value: EncodedValue, lenient: Bool = false, naming: Naming = Naming.Unchanged, strict: Bool = false, privateFields: Bool = false, flagsAsNames: Bool = false, exactAsText: Bool = false): Self
  fn readsPrivateFields(): Bool
  fn flagsAsNames(): Bool
  var fn nothing(): Bool
  var fn bool(): Result<Bool, DecodeError>
  var fn int(): Result<Int64, DecodeError>
  var fn unsigned(): Result<UInt64, DecodeError>
  var fn float(): Result<Float64, DecodeError>
  var fn decimal(): Result<Decimal, DecodeError>
  var fn string(): Result<String, DecodeError>
  var fn bytes(): Result<Bytes, DecodeError>
  var fn sequence(): Result<Int?, DecodeError>
  var fn map(): Result<Int?, DecodeError>
  var fn hasNext(): Result<Bool, DecodeError>
  var fn record(typeName: String): Result<Void, DecodeError>
  var fn variant(typeName: String): Result<String, DecodeError>
  var fn field(name: String): Result<Bool, DecodeError>
  var fn finish(): Result<Void, DecodeError>
}

EncodedValue is any Encode value held without its type: a quotation's captures, a field default in a schema, the text of rendered(value). It is built by Values, an ordinary Encoder, and it writes itself into any other one. An exact Decimal is an EncodedValue.Exact with every digit and its scale: 19.90 stays 19.90. ValueDecoder is the Decoder every buffering format shares: a text format with unordered fields parses into an EncodedValue and reads a typed value out of it, with the options lenient (numbers from text), naming, strict, flagsAsNames and exactAsText (a Decimal from a text, for a format without a decimal type, such as TOML) of the format. A Decimal field that meets a text refuses it unless one of lenient and exactAsText is set. JSON and YAML read a number that a Float64 does not print back as an Exact, so 19.90 and a value of 40 digits keep their digits in a document without a type.

Structure#

fn structureOf<Value: Describe>(): Structure
type Structure with Show {
  case Boolean
  case Integer
  case Natural
  case Floating
  case Exact
  case Text
  case Binary
  case Optional(of: Self)
  case Sequence(of: Self)
  case Mapping(key: Self, value: Self)
  case Record(typeName: String, fields: List<StructureField>)
  case Wrapper(typeName: String, of: Self)
  case Variant(typeName: String, cases: List<StructureCase>)
  case Reference(typeName: String)
  fn recordFields(): List<StructureField>
  fn declaredName(): String
  fn show(): String
}
type StructureField {
  description: FieldDescription
  of: Structure
  fn name(): String
  fn isRequired(): Bool
}
type FieldDescription {
  name: String
  documentation: String = ""
  default: FieldDefault = FieldDefault.Required
}
type FieldDefault {
  case Required
  case Computed
  case Constant(value: EncodedValue)
}
type Structures with Describer {
  fn result(): Structure
  var fn bool()
  var fn int()
  var fn unsigned()
  var fn float()
  var fn decimal()
  var fn string()
  var fn bytes()
  var fn optional()
  var fn sequence()
  var fn map()
  var fn record(typeName: String): Bool
  var fn variant(typeName: String): Bool
  var fn variantCase(name: String)
  var fn field(description: FieldDescription)
  var fn finish()
}

structureOf<Order>() is a type's description as a tree, which is what a schema format wants to walk: a record with its fields in declaration order, each with its doc comment and whether it is required, has a default that is a constant, or has one computed on the way in. A type that is already being described further up is a Reference to its name.

Naming#

type Naming {
  case Unchanged
  case SnakeCase
  fn applied(name: String): String
}

How a format spells the field names a type declares. It is an option of a format (Json(naming: .SnakeCase)) and never a word on the type.

The steps a derived form is made of#

encodeField, decodeRequired, decodeHasField, describeField and the rest of src/derived.trb are what the compiler generates a derived encode, decode and describe out of: one call per field, instantiated for the field's type and the format. They are public so that what a derived form does can be read, and so that a hand-written form can be made of the same pieces.

Format#

trait Format<Failure> {
  static fn encodeAll<Value: Encode>(value: Value): Bytes
  static fn decodeAll<Value: Decode>(bytes: Bytes): Result<Value, Failure>
  static fn items<Item: Decode>(): Stage<Bytes, Result<Item, Failure>>
  static fn encoded<Item: Encode>(): Stage<Item, Bytes>
}

What a whole format is, once: bytes in both directions, plus the two Stages (see std/iteration) that make it work on a Source of std/stream - body.through(Json.items<User>()).checked() reads one User at a time out of a byte stream, with memory of one value. Failure is a type parameter rather than an associated type, because the language has no associated types: Json is a Format<JsonError>. The members are static and use the format's default options. Encode/Decode and Encoder/Decoder stay synchronous either way; streaming happens one level up, at the element, through a resumable framer that finds where one element ends while the element itself decodes through the ordinary Decode.

Base64Alphabet, base64Decoded()#

type EncodingError with Show, Error {
  reason: String
  offset: Int
  fn show(): String
}
type Base64Alphabet with Show, Equals, Hash {
  case Standard
  case UrlSafe
  fn encoded(bytes: Bytes, padded: Bool = true): String
}
extend String {
  fn hexDecoded(): Result<Bytes, EncodingError>
  fn base64Decoded(): Result<Bytes, EncodingError>
}
extend String with Encode, Decode, Describe

Base64 (RFC 4648 sections 4 and 5): Base64Alphabet.Standard (+, /) for everywhere nothing says otherwise, and Base64Alphabet.UrlSafe (-, _) for a JSON Web Token's segments and a JSON Web Key's modulus - each with or without the = padding. text.base64Decoded() reads either alphabet, padded or not, in the same call: the two agree on 62 of their 64 characters, so nothing is lost by not asking which one a text was written in. Whitespace inside the text is skipped, the shape a wrapped MIME block needs.

use Base64Alphabet from "std/encoding"
use String.base64Decoded from "std/encoding"

const encoded = Base64Alphabet.UrlSafe.encoded "hi".bytes(), padded: false
print encoded
print encoded.base64Decoded()

Base32Alphabet#

type Base32Alphabet with Show, Equals, Hash {
  case Standard
  case Extended
  fn encoded(bytes: Bytes, padded: Bool = true): String
  fn decoded(text: String): Result<Bytes, EncodingError>
}

Base32 (RFC 4648 sections 6 and 7): Base32Alphabet.Standard (A-Z, 2-7), what NATS NKeys and a TOTP secret are written in, and Base32Alphabet.Extended (0-9, A-V, "base32hex"), which sorts the same as the bytes it stands for. Unlike Base64's two alphabets, these are not read together - A means 0 in one and 10 in the other - so decoding takes the alphabet the text was written in.

use Base32Alphabet from "std/encoding"

print Base32Alphabet.Standard.encoded("foobar".bytes())

Hexadecimal: hexEncoded(), hexDecoded()#

extend Bytes {
  fn hexEncoded(uppercase: Bool = false): String
}

Hexadecimal (RFC 4648 section 8): two digits a byte, most significant nibble first. bytes.hexEncoded() writes lower case unless told otherwise; text.hexDecoded() reads either case, mixed or not. Digest.hex() of std/digest is this in one direction only, fixed to lower case, for a hash's own printing.

use Buffer.hexEncoded, String.hexDecoded from "std/encoding"

print "hi".bytes().hexEncoded()
print "6869".hexDecoded()
Esc
Enter öffnet ein ErgebnisEsc schließt