`std/html/tokenizer`

std/html/src/tokenizer.trb

HtmlTokenizer: the tokenization stage of the HTML standard (section 13.2.5), as a state machine over the characters of a document. It reads one transition at a time and hands out the tokens one at a time, so the tree construction can switch its state between two tokens, as the standard asks.

type HtmlTokenizer

type HtmlTokenizer

The tokenizer of the HTML standard. HtmlTokenizer.of makes one for a text, HtmlTokenizer.next answers the next token, and HtmlTokenizer.tokens all that are left, the last one being HtmlToken.EndOfFile. Reading HTML never fails: each recoverable error of the standard is an HtmlParseError in HtmlTokenizer.errors, and the tokenizer goes on exactly as the standard says.

The text is preprocessed as the standard says first: a carriage return, alone or before a line feed, is read as one line feed. The tree construction steers the tokenizer by setting HtmlTokenizer.state after a start tag (to Rcdata for <title> and <textarea>, to Rawtext for <style>, <xmp>, <iframe>, <noembed> and <noframes>, to ScriptData for <script> and to Plaintext for <plaintext>) and by saying whether a CDATA section is allowed with HtmlTokenizer.inForeignContent.

Examples

var tokenizer = HtmlTokenizer.of "<p class=intro>Fish &amp; chips"
for token in tokenizer.tokens() {
  print token
}
print tokenizer.errors

field state

var state: HtmlTokenizerState = .Data

The state the next transition is made in. It starts in Data. The tree construction sets it to Rcdata, Rawtext, ScriptData, Plaintext or CdataSection after a start tag whose content is not markup, and back to Data where it ends.

field lastStartTag

var lastStartTag: String? = None

The name of the last start tag the tokenizer handed out, which decides what an end tag in Rcdata, Rawtext and ScriptData is: only one with this name ends the element. A tokenizer started in one of those states for a fragment is told the name of its context element here.

field inForeignContent

var inForeignContent: Bool = false

Whether the adjusted current node is a foreign element, which is where <![CDATA[ starts a CDATA section.

field errors

protected var errors: List<HtmlParseError> = []

The parse errors so far, in the order they happened.

fn of

static fn of(text: String): Self

A tokenizer for text, in the Data state.

fn tokenize

static fn tokenize(text: String): (tokens: List<HtmlToken>, errors: List<HtmlParseError>)

The tokens of text and the parse errors found on the way, for a document that needs no steering: the state of the tokenizer stays Data.

Examples

const result = HtmlTokenizer.tokenize "<b>bold"
print result.tokens
print result.errors

fn next

var fn next(): HtmlToken

The next token. After the last one, HtmlToken.EndOfFile, the token the document ends with, is answered again.

fn tokens

var fn tokens(): List<HtmlToken>

All the tokens that are left, the last one being HtmlToken.EndOfFile.