`std/html/token`

std/html/src/token.trb

The tokens of the HTML tokenizer: HtmlToken and the HtmlAttribute of a tag.

type HtmlAttribute

type HtmlAttribute with Show, Equals, Hash

One attribute of a tag, in the order the document wrote it.

name: String

The name, with its ASCII capitals made lower case.

value: String

The value with its character references resolved: an empty text for an attribute written without one.

type HtmlToken with Show, Equals, Hash

One token of the HTML standard's tokenizer (section 13.2.5), which the tree construction reads.

Adjacent characters are one Character token, so a&b is a single token holding a&b.

use HtmlTokenizer from "std/html"

const tokens = HtmlTokenizer.tokenize("<p class=a>Hi").tokens
print tokens

case Doctype(name: String?, publicId: String?, systemId: String?, forceQuirks: Bool)

A document type declaration. A name or an identifier the document leaves out is None, and forceQuirks is set where the declaration is malformed.

case StartTag(name: String, attributes: List<HtmlAttribute>, selfClosing: Bool)

A start tag with its attributes in document order, a repeated attribute left out, and its self-closing flag.

case EndTag(name: String)

An end tag. Attributes and a self-closing flag on it are errors and not part of the token.

case Comment(data: String)

A comment.

case Character(data: String)

A run of characters, one or more.

case EndOfFile

The end of the input: the last token, and the tokenizer answers it again if it is asked for more.