std/html

Die generierte Referenz von std/html zeigt jede öffentliche Deklaration mit Signatur, Doc-Kommentar und Beispielen.

std/html reads HTML the way the WHATWG HTML standard says a browser does, written in TorbScript. This is the first step of the package: the tokenizer of section 13.2.5 and the character references. HtmlTokenizer turns a text into HtmlToken values (start and end tags with their attributes, comments, a DOCTYPE, text, and the end of the file), and every mistake the standard names on the way is an HtmlParseError in a list beside the tokens. Reading HTML never fails: the standard defines what to do with every byte sequence, so there is no Result and no error that stops the reader. The tree, with the insertion modes that turn tokens into elements, comes in step 2 and builds on these tokens.

The tokenizer is the 80 states of the standard, one function each, over the characters of a String. A carriage return, alone or before a line feed, is read as one line feed, as the standard's preprocessing says. The 2231 named character references of the standard's table are all known, the 106 legacy names are also read without their semicolon, and a numeric reference gets the replacements the standard defines for the C1 controls. The tokenizer reads each character once, and each & costs a bounded number of table lookups, so the time grows with the length of the text. The tokenizer tests of html5lib, vendored in std/html/tests/html5lib, are run against it (see std/html/tests/NOTICE.md).

std/html also writes HTML. Html<Message> is a page or a piece of one, Markup<Message> is the receiver of the blocks that write it, and every element of the standard is a var fn of Markup. What a program writes is escaped where it is written, so the markup of a page can not be broken by the data in it. The writing side is described first, the reading side (the tokenizer) after it.

Writing HTML#

An element is called in one of three shapes, and one declaration gives all of them:

use fragment from "std/html"

const page = fragment {
  p "Fish & chips"
  table class: "orders" {
    tr {
      td "1"
      td class: "amount" {
        text "12 EUR"
      }
    }
  }
}
print page.render()

p "text" writes text, td class: "amount" { ... } writes attributes and children, tr { ... } only children. The text of a call and text "..." are both escaped. A void element (br, hr, img, input, meta, ...) takes no block and has no end tag. The attributes every element has (id, class, title, lang, hidden) are labelled parameters with defaults, an element's own attributes are typed (colspan is an Int, checked a Bool), and the rare ones go through attributes: ["itemprop": "name"].

Two attribute names are reserved words, so input kind: "email" writes type="email" and label target: "mail" writes for="mail". The element var is variable, the attribute default is isDefault, and loop is repeat.

A page is made with fragment (the markup and nothing around it) or document (the doctype, html with a language, a head with the character set, the viewport and the title, and a body):

use Html, Markup, document from "std/html"

type Order {
  number: Int
  customer: String
}

extend<Message> Markup<Message> {
  /** A component is a member of the builder, so it reads like an element where it is used. */
  var fn customerRow(order: Order) {
    tr {
      td "{order.number}"
      td order.customer
    }
  }
}

fn ordersPage(orders: List<Order>): Html<Never> {
  document title: "Orders", lang: "en" {
    h1 "Orders"
    if orders.isEmpty() {
      p "No orders yet."
    }
    table {
      for order in orders {
        customerRow order
      }
    }
  }
}

print ordersPage([Order(number: 1, customer: "Ada <admin>")]).render()

Html.render() joins the pieces once and answers the String. A component is an extend<Message> Markup<Message> when it works for any page. include page writes a page that was built before at the place it is called.

Escaping#

A String is not a Link: a href: "https://..." does not check, so the scheme of a link is always decided by a Uri. attributes: refuses what has a typed parameter. A name that is not an attribute name of the grammar, an event handler (onclick) and a URL attribute (href, src, ...) panic with a message that names the attribute: the name was written by the program, so it is a bug of the program and not an input.

There is no function from a String to Html that neither escapes nor sanitizes. A Markdown renderer writes through the builder, element by element, like every other producer of markup. Markup the author ships in a file is the one exception: Html.trusted(file:) takes an EmbeddedText, a file of the package that the compiler read at build time, and style(sheet:) writes such a file as the text of a style element. A String is neither, so a value never skips the escaping.

The elements and their attributes are generated: tools/html/elements.trb writes std/html/src/elements.trb from tools/html/elements.txt, and the generated file is checked in. Nothing about it is a build step.

Import#

use Html, Markup, Link, fragment, document from "std/html"
use HtmlTokenizer, HtmlTokenizerState, HtmlToken, HtmlAttribute, HtmlParseError from "std/html"

Tag soup is the input the standard was written for. The tokenizer lower-cases names, drops the second of two equal attributes with an error, resolves the references it knows and keeps the ones it does not:

use HtmlTokenizer from "std/html"

const soup = "<P CLASS=intro id='a' id='b'>Fish &amp; chips &copy 2026 &bogus;<br/><!-- note --></p>"
const result = HtmlTokenizer.tokenize soup
for token in result.tokens {
  print token
}
for problem in result.errors {
  print "error at {problem}"
}

tokenize is the whole text at once. HtmlTokenizer.of(text) makes a tokenizer that hands out one token per next(), which is what the tree construction needs, because it changes the state of the tokenizer between two tokens.

Declarations#

Html#

type Html<Message = Never> {
  fn render(): String
  static fn trusted(file: EmbeddedText): Html<Never>
  fn isEmpty(): Bool
}

An Html<Message> holds the pieces a builder block wrote and has no constructor but fragment and document; its Message is the type of the messages the page sends (Never for a page that sends none). render() is the markup as one String, and isEmpty() says whether nothing was written.

The fragment and document functions#

fn fragment(children: (var self: Markup<Never>) => Void): Html<Never>
fn document(title: String, lang: String = "en", children: (var self: Markup<Never>) => Void = {}): Html<Never>

Both answer an Html<Never>. They are not generic yet: the native back end can not infer the Message of a generic function from the type its result is used as, so for now a page that sends messages waits for that.

Link#

trait Link {
  fn target(): String
}

Link is implemented for Uri and UriReference. target() is the text written into the attribute.

HtmlTokenizer#

type HtmlTokenizer {
  var state: HtmlTokenizerState = .Data
  var lastStartTag: String? = None
  var inForeignContent: Bool = false
  protected var errors: List<HtmlParseError> = []
  static fn of(text: String): Self
  static fn tokenize(text: String): (tokens: List<HtmlToken>, errors: List<HtmlParseError>)
  var fn next(): HtmlToken
  var fn tokens(): List<HtmlToken>
}

An HtmlTokenizer starts in the Data state. next() answers the next token, and after the last one, which is always HtmlToken.EndOfFile, it answers that token again. tokens() collects all that are left. errors is the list of the parse errors so far, in the order they happened, and does not stop anything.

What the tree construction sets. After a start tag whose content is not markup the standard switches the tokenizer: state is set to Rcdata for <title> and <textarea>, to Rawtext for <style>, <xmp>, <iframe>, <noembed> and <noframes>, to ScriptData for <script>, and to Plaintext for <plaintext>. lastStartTag is the name of the last start tag the tokenizer handed out, which decides what an end tag in those states is: only </title> ends a <title>. A tokenizer that starts in one of those states, for a fragment, is told the name of the context element in lastStartTag. inForeignContent says whether <![CDATA[ starts a CDATA section, which is text, or is a bogus comment as it is in HTML.

use HtmlTokenizer, HtmlToken from "std/html"

var tokenizer = HtmlTokenizer.of "<b>not bold</b> &amp; </title>"
tokenizer.state = .Rcdata
tokenizer.lastStartTag = Some "title"
print tokenizer.tokens()

HtmlToken and HtmlAttribute#

type HtmlToken with Show, Equals, Hash {
  case Doctype(name: String?, publicId: String?, systemId: String?, forceQuirks: Bool)
  case StartTag(name: String, attributes: List<HtmlAttribute>, selfClosing: Bool)
  case EndTag(name: String)
  case Comment(data: String)
  case Character(data: String)
  case EndOfFile
}
type HtmlAttribute with Show, Equals, Hash {
  name: String
  value: String
}

The tokens of the standard. A Doctype keeps None for a name or an identifier the DOCTYPE does not have, which is not the same as an empty one, and forceQuirks is the standard's flag. A StartTag has its attributes in the order they were written, with the names lower-cased and the values resolved; of two attributes with one name only the first is kept. Character is a run of text, with the character references already resolved, and the characters between two other tokens are one token.

HtmlTokenizerState#

type HtmlTokenizerState with Show, Equals, Hash {
  case Data
  case Rcdata
  case Rawtext
  case ScriptData
  case Plaintext
  case TagOpen
  case EndTagOpen
  case TagName
  case RcdataLessThanSign
  case RcdataEndTagOpen
  case RcdataEndTagName
  case RawtextLessThanSign
  case RawtextEndTagOpen
  case RawtextEndTagName
  case ScriptDataLessThanSign
  case ScriptDataEndTagOpen
  case ScriptDataEndTagName
  case ScriptDataEscapeStart
  case ScriptDataEscapeStartDash
  case ScriptDataEscaped
  case ScriptDataEscapedDash
  case ScriptDataEscapedDashDash
  case ScriptDataEscapedLessThanSign
  case ScriptDataEscapedEndTagOpen
  case ScriptDataEscapedEndTagName
  case ScriptDataDoubleEscapeStart
  case ScriptDataDoubleEscaped
  case ScriptDataDoubleEscapedDash
  case ScriptDataDoubleEscapedDashDash
  case ScriptDataDoubleEscapedLessThanSign
  case ScriptDataDoubleEscapeEnd
  case BeforeAttributeName
  case AttributeName
  case AfterAttributeName
  case BeforeAttributeValue
  case AttributeValueDoubleQuoted
  case AttributeValueSingleQuoted
  case AttributeValueUnquoted
  case AfterAttributeValueQuoted
  case SelfClosingStartTag
  case BogusComment
  case MarkupDeclarationOpen
  case CommentStart
  case CommentStartDash
  case Comment
  case CommentLessThanSign
  case CommentLessThanSignBang
  case CommentLessThanSignBangDash
  case CommentLessThanSignBangDashDash
  case CommentEndDash
  case CommentEnd
  case CommentEndBang
  case Doctype
  case BeforeDoctypeName
  case DoctypeName
  case AfterDoctypeName
  case AfterDoctypePublicKeyword
  case BeforeDoctypePublicIdentifier
  case DoctypePublicIdentifierDoubleQuoted
  case DoctypePublicIdentifierSingleQuoted
  case AfterDoctypePublicIdentifier
  case BetweenDoctypePublicAndSystemIdentifiers
  case AfterDoctypeSystemKeyword
  case BeforeDoctypeSystemIdentifier
  case DoctypeSystemIdentifierDoubleQuoted
  case DoctypeSystemIdentifierSingleQuoted
  case AfterDoctypeSystemIdentifier
  case BogusDoctype
  case CdataSection
  case CdataSectionBracket
  case CdataSectionEnd
  case CharacterReference
  case NamedCharacterReference
  case AmbiguousAmpersand
  case NumericCharacterReference
  case HexadecimalCharacterReferenceStart
  case DecimalCharacterReferenceStart
  case HexadecimalCharacterReference
  case DecimalCharacterReference
  case NumericCharacterReferenceEnd
}

The 80 states of the standard, named as it names them without the word "state" (Data, TagOpen, ScriptDataEscaped, NamedCharacterReference). The states that matter from outside are Data, Rcdata, Rawtext, ScriptData, Plaintext and CdataSection: the tree construction sets one of those, and the tokenizer finds its own way through the others.

HtmlParseError#

type HtmlParseError with Show, Equals, Hash {
  code: String
  line: Int
  column: Int
  fn show(): String
}

An HtmlParseError has the code the standard gives the error (eof-in-tag, duplicate-attribute, missing-semicolon-after-character-reference), and the line and the column of the character the tokenizer was looking at, counted from 1 and in characters. show() writes both:

line 1, column 24: duplicate-attribute
line 1, column 40: missing-semicolon-after-character-reference

Character references#

A reference starts with &. A name is matched against the standard's table with the longest match winning: &notit; is ¬ followed by the text it; and an error, because &not is a legacy name and &notit; is not a name. A name that ends in ; is read everywhere. A legacy name without its semicolon (&amp, &copy, &lt) is read in text, with the error missing-semicolon-after-character-reference; in an attribute value it is kept as it is when the next character is = or a letter or digit, so a query string like ?a=1&copy=2 is not changed. A name nobody knows stays as it was written, and &name; with a semicolon is the error unknown-named-character-reference.

A numeric reference, A or A, is the character with that number. The standard replaces the numbers it cannot use: 0 and the surrogates and every number above 10FFFF become U+FFFD with an error, and the numbers 80 to 9F are read as the Windows-1252 character of that position (€ is the euro sign), with the error control-character-reference. Noncharacters and the other controls are kept and are an error. A reference that is no reference, such as &#; or &#xZ, is the text it was.

What this package does not do#

• It builds no tree when reading. The tokens are the input of the tree construction in step 2, and nothing in this package nests them or closes an element that was left open.
• It has no input but a String: bytes, the encoding sniffing and the <meta charset> rule of the standard are not part of it, and a document that is not text yet has to be decoded first.
• It does not depend on std/markdown: the two packages share the idea of a table of references and nothing else.

Related#

• std/uri - Uri and UriReference, the two types that are a Link.
• std/resource - Resource, the only thing a script can be.
• std/markdown - the other package with a table of references; this one does not import it.
• std/xml - another markup format, read into a tree.