std/http

Die generierte Referenz von std/http zeigt jede öffentliche Deklaration mit Signatur, Doc-Kommentar und Beispielen.

std/http is HTTP/1.1 in TorbScript over std/network, and HTTPS over std/tls: a client, a server, and the messages both speak. Every body is a stream (see std/stream): Body is a Source<Bytes, HttpError>, so a body of any size is read chunk by chunk, and the convenience that covers the common case (body.text(), body.json<User>()) sits on top of it with a limit. It needs the network capability inside a sandboxed script, and is not in the prelude.

Import#

use * as http from "std/http"
use Server, Request, Response, Status, Method, Headers, Body, HttpError from "std/http"

use * as http from "std/http"
use Server, Request, Response, Status, HttpError from "std/http"
use SocketAddress, IpAddress from "std/network"

type User {
  id: Int
  name: String
}

fn fetchUser(api: Uri, id: Int): Task<Result<User, HttpError>> {
  var response = http.get(api.joined("users/{id}")).await()?
  response.json<User>().await()
}

fn hello(request: Request): Task<Result<Response, HttpError>> {
  if request.path() == "/hello" {
    return Response.text "hello"
  }
  Response.of Status.notFound
}

fn serveForever(): Task<Result<Void, HttpError>> {
  var server = Server.listen(SocketAddress(IpAddress.loopback, 8080), hello)?
  server.serve().await()
}

Declarations#

get, post, send#

fn get(url: Uri, headers: Headers = Headers()): Task<Result<Response, HttpError>>
fn post(url: Uri, body: Body, headers: Headers = Headers()): Task<Result<Response, HttpError>>
fn send(method: Method, url: Uri, headers: Headers = Headers(), body: Body = Body.empty(), tls: TlsSettings = TlsSettings()): Task<Result<Response, HttpError>>

The client. What it reaches is a Uri, read from text once where the text enters the program (Uri.tryFrom(text)?) or built from one (api.joined("users/{id}"), which encodes each segment and never climbs above api) - there is no URL string to get wrong at the call. A request opens a connection to the host of the URI - straight to an IP literal, through the resolver for a name - writes the request - with Content-Length where the body's length is known, in chunks where it is not - and answers once the head of the response arrived; its body is read from the connection as the program pulls it, and the connection closes with the response. A status that is not a success is a Response too, and no redirect is followed. A timeout is within: http.get(url).within(10.seconds()). A URI this client does not reach - a scheme other than http and https, an empty host, user information (RFC 9110 section 4.2) - is an HttpError that is unsupported; the fragment is never sent. An https URI is the same request over TLS, on port 443 unless the URI names another: the server's certificate is checked for the host the way the platform checks it, or against the roots of tls where send names some, and a certificate that is refused is an HttpError whose cause() is the NetworkError that isCertificateRejected().

Client#

shared type Client {
  maximumConnections: Int = 6
  timeout: Duration? = None
  redirects: RedirectPolicy = RedirectPolicy.SameHost 10
  tls: TlsSettings = TlsSettings()
  resolver: NameResolver? = None
  fn get(url: Uri, headers: Headers = Headers()): Task<Result<Response, HttpError>>
  fn post(url: Uri, body: Body, headers: Headers = Headers()): Task<Result<Response, HttpError>>
  fn send(method: Method, url: Uri, headers: Headers = Headers(), body: Body = Body.empty()): Task<Result<Response, HttpError>>
}
type RedirectPolicy with Show, Equals {
  case Never
  case SameHost(limit: Int)
  case AnyHost(limit: Int)
  fn show(): String
}

The client that keeps its connections: get, post and send are the free functions', over a pool of kept-alive connections per origin (scheme, host and port), at most maximumConnections of them open to one origin - a request beyond waits until a response gives one back. A connection goes back to the pool once its response's body was read to its end, and is closed where the response is released before that. A kept-alive connection the server closed while it waited is noticed when a request finds it closed before any byte of a response, and a request of an idempotent method without a body is sent again over a new one. timeout limits each request up to the head of its response, redirects included; the body is the program's to limit with within. resolver says what finds the addresses of a host name: None is the system's resolver; Some(resolver) asks that NameResolver for A and AAAA and connects to the first address that accepts, in the order answered. The TLS server name stays the host's name, an IP literal is never looked up, and a lookup that fails is the HttpError that carries the NetworkError - never a fallback to the system's resolver.

Redirects follow redirects: a Client follows up to ten to the same host (http to https included), because a program that named a host gets answers from that host; AnyHost follows them anywhere and then drops Authorization, Cookie and Proxy-Authorization where the host changes; Never answers the redirect itself, as the free functions always do. No policy follows https to http, or a 307/308 that would have to send a streamed body again; 303 - and 301/302 after a POST - becomes a GET without a body. More redirects than the limit is an HttpError (tooManyRedirects). A Client is an object: it stays with the task that made it, so the requests of one client run on that task's worker.

use Client, HttpError from "std/http"

/** Two requests to one API over one kept-alive connection. */
fn twoUsers(api: Uri): Task<Result<(String, String), HttpError>> {
  const client = Client maximumConnections: 2
  var first = client.get(api.joined("users/1")).await()?
  const one = first.body.text().await()?
  var second = client.get(api.joined("users/2")).await()?
  (one, second.body.text().await()?)
}

DohResolver#

type DohResolver with NameResolver {
  url: Uri
  tls: TlsSettings = TlsSettings()
  attemptMilliseconds: Int = 5000
  fn lookup(name: DomainName, recordType: RecordType): Task<Result<List<Record>, NetworkError>>
  fn exchange(name: DomainName, recordType: RecordType): Task<Result<Message, NetworkError>>
}

DNS over HTTPS (RFC 8484), in the shape of TlsResolver: the lookups of std/network's Resolver, asked of one name server as a POST of the query to url, which has to be an https URL whose host proves itself to tls's roots (or the platform's). The query goes with the identifier 0 of RFC 8484 section 4.1 and the media type application/dns-message as Content-Type and Accept; the response is the body of a 2xx answer, at most 65535 bytes of it, read with readResponse of std/dns. One connection per lookup. lookup answers the records of the type, CNAME chains followed, and fails with isHostNotFound() for NXDOMAIN, with isNameServerFailure() for SERVFAIL, REFUSED and NOTIMP, for a body that is no DNS response, for a status that is not a success (the status is in the message) and for a URL that is not https - before any connection is opened - with isCertificateRejected() where the server cannot prove the URL's host, and with isTimedOut() where attemptMilliseconds pass. A Client connects through a DohResolver (or any NameResolver) with its resolver option.

use DohResolver from "std/http"
use DomainName, RecordType, Record from "std/dns"
use NetworkError from "std/network"

/** The IPv6 addresses of a name, asked of a DoH server. */
fn addresses(server: Uri, name: DomainName): Task<Result<List<Record>, NetworkError>> {
  const resolver = DohResolver url: server
  resolver.lookup(name, RecordType.Aaaa).await()
}

Server#

type Handler = (request: Request) => Task<Result<Response, HttpError>>
shared type Server with Close {
  static fn listen(address: SocketAddress, handler: Handler, limits: ServerLimits = ServerLimits(), tls: ServerIdentity? = None): Result<Self, HttpError>
  fn localAddress(): SocketAddress
  var fn serve(): Task<Result<Void, HttpError>>
  var fn serveOne(): Task<Result<Void, HttpError>>
  var fn shutdown(grace: Duration = 10.seconds()): Task<Void>
  var fn close()
}
type ServerLimits {
  headBytes: Int = 65536
  headFields: Int = 100
  headMilliseconds: Int = 10000
  idleMilliseconds: Int = 60000
  drainBytes: Int = 65536
}
type Upgrade = (duplex: Duplex) => Task<Result<Void, HttpError>>
shared type Duplex with Close {
  static fn of(connection: Connection): Self
  fn source(chunk: Int = 65536): Source<Bytes, HttpError>
  fn sink(): Sink<Bytes, HttpError>
  fn remote(): SocketAddress?
  fn isEncrypted(): Bool
  var fn close()
}

The server. A handler is a function from a request to a task of a response - a web framework is one of these, built out of routes. serve() accepts until the server is shut down or closed - with an accept loop on every worker, all of them on the one listening socket, so connections spread over the cores - and each connection is a task of its own, on the worker whose loop accepted it, that serves its requests one after another (a persistent connection, and pipelined requests in order). A handler that holds an object keeps every loop on the serving task's worker, and a server with tls runs one loop. A client that goes away while the handler of a request without a body runs - the connection ends or breaks - cancels the handler's task, with every task it started and every using of it closed; bytes that arrive instead are the next request and wait for their turn. shutdown(grace) stops accepting, closes the connections that wait for their next request, lets the requests in progress finish - each answered with Connection: close - and cancels what still runs after grace; then serve() answers Ok. close(), which the release of the server runs, stops at once and cancels every connection. With tls, every connection is HTTPS: the TLS handshake comes first and has to finish within headMilliseconds. The server writes no Server field of its own, so it does not tell a scanner what it is; a handler that wants one sets it. A handler that fails is answered 500 without telling the client why. A request the parser refuses is answered with the status its failure names - 400 for a malformed message, 408 for a head that took longer than headMilliseconds, 431 for one over headBytes or headFields, 501 for a transfer coding that is not chunked, 505 for another version - and the connection is closed. The parser refuses everything request smuggling lives on: a Content-Length beside a Transfer-Encoding, two lengths, a bare CR or LF, a folded line, a missing Host.

A handler can switch the connection to another protocol: Response.upgrade(handler, headers) is a 101 Switching Protocols with the fields the handler gives it - the protocol's own, such as Upgrade and Connection: Upgrade, are the handler's to set - and no Content-Length, no Transfer-Encoding and no Connection: close. Once the server has written it, it calls the Upgrade closure with a Duplex and the connection is the closure's until the closure finishes: source() yields first what the connection had buffered after the request head (a client may send its first bytes with the request) and then what arrives, and ends with None once the peer ended its half; sink().add(bytes) writes all of the bytes and sink().end() ends the writing half (with a TLS close_notify over https). The head, idle and drain limits no longer apply to it, and the connection is never read as HTTP again. A closure that fails or is cancelled closes only its connection and sends nothing. The connection counts as busy: shutdown(grace) waits for it up to grace and then cancels it, and close() cancels it at once. A 101 without an upgrade, or an upgrade with another status, is answered 500. The protocol on top - WebSocket framing is one - is a stage over source() and sink(), and belongs to std/web, not to the transport.

fn echo(duplex: Duplex): Task<Result<Void, HttpError>> {
  var incoming = duplex.source()
  var outgoing = duplex.sink()
  while const Some(chunk) = incoming.next().await()? {
    outgoing.add(chunk).await()?
  }
  outgoing.end().await()
}

fn handle(request: Request): Task<Result<Response, HttpError>> {
  var headers = Headers()
  headers.set "Upgrade", "echo"
  headers.set "Connection", "Upgrade"
  Ok Response.upgrade(echo, headers)
}

Messages#

shared type Request {
  method: Method
  target: String
  uri: Uri
  headers: Headers = Headers()
  var body: Body
  remote: SocketAddress? = None
  version: String = "HTTP/1.1"
  fn path(): String
  fn segments(): List<String>
  fn query(): String?
}
shared type Response {
  status: Status
  var headers: Headers = Headers()
  var body: Body
  var takeover: Upgrade? = None
  static fn of(status: Status): Self
  static fn upgrade(handler: Upgrade, headers: Headers = Headers()): Self
  static fn text(content: String, status: Status = Status.ok): Self
  static fn jsonOf<Value: Encode>(value: Value, status: Status = Status.ok): Self
  var fn json<Value: Decode>(limit: Int = Body.defaultLimit): Task<Result<Value, HttpError>>
}

Both are shared types, because each owns its body, a stream that is read once. A request's target is what was sent (/users/7?details=true, *, or host:443 for CONNECT), and uri is the target URI RFC 9112 section 3.3 rebuilds from it, the Host field and whether the connection is TLS (https://example.test/users/7?details=true); a request whose target and host make no URI is answered 400. path(), segments() and query() read the URI, so a router sees a normalized path whose dot segments are gone.

Method, Status, Headers#

type Method with Show, Equals, Hash {
  case Get
  case Head
  case Post
  case Put
  case Delete
  case Patch
  case Options
  case Trace
  case Connect
  case Query
  case Search
  case Other(name: String)
  static fn of(name: String): Result<Self, HttpError>
  fn name(): String
  fn isSafe(): Bool
  fn isIdempotent(): Bool
  fn carriesContent(): Bool
  fn show(): String
}
type Status with Show, Equals, Hash, Compare {
  static switchingProtocols = Self 101
  static ok = Self 200
  static created = Self 201
  static accepted = Self 202
  static noContent = Self 204
  static movedPermanently = Self 301
  static found = Self 302
  static seeOther = Self 303
  static notModified = Self 304
  static temporaryRedirect = Self 307
  static permanentRedirect = Self 308
  static badRequest = Self 400
  static unauthorized = Self 401
  static forbidden = Self 403
  static notFound = Self 404
  static methodNotAllowed = Self 405
  static notAcceptable = Self 406
  static requestTimeout = Self 408
  static conflict = Self 409
  static gone = Self 410
  static lengthRequired = Self 411
  static contentTooLarge = Self 413
  static uriTooLong = Self 414
  static unsupportedMediaType = Self 415
  static unprocessableContent = Self 422
  static tooManyRequests = Self 429
  static requestHeaderFieldsTooLarge = Self 431
  static internalServerError = Self 500
  static notImplemented = Self 501
  static badGateway = Self 502
  static serviceUnavailable = Self 503
  static gatewayTimeout = Self 504
  static httpVersionNotSupported = Self 505
  static fn of(code: Int): Result<Self, HttpError>
  fn code(): Int
  fn reason(): String
  fn isInformational(): Bool
  fn isSuccess(): Bool
  fn isRedirect(): Bool
  fn isClientError(): Bool
  fn isServerError(): Bool
  fn show(): String
  fn compare(other: Self): Ordering
}
type Headers with Show, Equals {
  fn get(name: String): String?
  fn all(name: String): List<String>
  fn contains(name: String): Bool
  fn length(): Int
  var fn set(name: String, value: String)
  var fn add(name: String, value: String)
  var fn remove(name: String)
  fn entries(): List<(name: String, value: String)>
  fn show(): String
}
extend Headers with From<Map<String, String>>

Method has a case per method of RFC 9110, Query (RFC 10008, the HTTP QUERY method: a query in the request's content, safe and idempotent like GET) and Search (WebDAV, RFC 5323), so a router matches on it, and Other for the rest: Method.of(name) is the way in from text, and it never answers Other for a name a case has. isSafe(), isIdempotent() (RFC 9110 section 9.2) and carriesContent() say what the method promises; the client announces the content of a method that carries some, even an empty one, and sends a request again over a new connection only where its method is idempotent. Status is a capsule over its number with the common codes as constants, because 599 is a status too. Headers keeps every field in the order it arrived, compares names without regard to case, and keeps a name that repeats (Set-Cookie). A value with a line break is written with a space in its place and a name that is not a token is not written at all, so a header built from what a client sent cannot carry a second header in. Content-Length, Transfer-Encoding and Connection belong to the writer, which sets them from the body and the connection.

Body#

shared type Body with Source<Bytes, HttpError> {
  static defaultLimit: Int = 16777216
  var fn next(): Task<Result<Bytes?, HttpError>>
  var fn close()
  fn length(): Int?
  var fn bytes(limit: Int = Self.defaultLimit): Task<Result<Bytes, HttpError>>
  var fn text(limit: Int = Self.defaultLimit): Task<Result<String, HttpError>>
  var fn json<Value: Decode>(limit: Int = Self.defaultLimit): Task<Result<Value, HttpError>>
  var fn lines(): Source<String, HttpError>
  static fn of(var source: Source<Bytes, HttpError>, length: Int? = None): Self
  static fn empty(): Self
  static fn jsonOf<Value: Encode>(value: Value): Self
}
extend Body with From<String>
extend Body with From<Bytes>

The bytes of a message, read once. defaultLimit is 16 MiB: a peer that decides how much memory a program allocates is a denial of service, and streaming is the way past it, not a bigger number. length() is known for a body made from text or bytes and for one that arrived with Content-Length; a body of unknown length is sent in chunks. Body.from("text") and Body.from(bytes) come from From.

HttpError#

type HttpError with Show, Error {
  static fn timeout(): Self
  static fn connectionFailed(message: String): Self
  static fn network(cause: NetworkError): Self
  static fn status(code: Int, message: String): Self
  static fn decodeFailed(cause: JsonError): Self
  static fn tooLarge(limit: Int): Self
  static fn invalidText(cause: Utf8Error): Self
  static fn malformed(message: String): Self
  static fn headTooLarge(limit: Int): Self
  static fn unsupported(message: String): Self
  static fn unsupportedVersion(version: String): Self
  static fn incomplete(): Self
  static fn tooManyRedirects(limit: Int): Self
  fn isTimeout(): Bool
  fn isRetryable(): Bool
  fn statusCode(): Int?
  fn answerStatus(): Int?
  fn show(): String
  fn networkFailure(): NetworkError
  fn cause(): Error?
}
extend HttpError with From<Utf8Error>
extend HttpError with From<JsonError>
extend HttpError with From<NetworkError>

What went wrong, as one wrapper type: a new kind of failure is never a breaking change. answerStatus() is the status a server answers a request with that failed this way while it was read; a failure of the network under a message keeps its NetworkError as cause(), and networkFailure() hands that NetworkError to a caller whose own failure is one (a resolver), a timeout as a timeout and every other failure as a name server's that gave no usable answer.

Related#

• std/network - the TCP streams under HTTP.
• std/tls - the TLS under HTTPS: TlsSettings and ServerIdentity.
• std/stream - Source, which Body is, and Sink.
• std/json - Json().decode, which Response.json and Body.json call.
• The standard library - the other packages.