std/network

The generated reference of std/network has every public construct with its signature, its doc comment and its examples.

std/network is the network below HTTP: name resolution, DNS lookups, TCP, UDP and Unix domain sockets, over the address values of std/ip. Everything that waits for the network answers a Task and can be cancelled; a timeout is within. It runs on the IO core of the runtime - an IO completion port on Windows, epoll on Linux, kqueue on macOS and FreeBSD - which wakes a task when its bytes arrive, so a thousand connections wait on one thread (see docs/design/NETWORK.md). It needs the network capability inside a sandboxed script, and is not in the prelude.

Import#

use TcpListener, TcpStream, UdpSocket, SocketAddress, IpAddress, NetworkError, resolve from "std/network"

use TcpListener, TcpStream, SocketAddress, IpAddress, NetworkError from "std/network"

/** Echoes one connection back to itself, and answers how many bytes it echoed. */
fn echoOnce(listener: TcpListener): Task<Result<Int, NetworkError>> {
  const connection = listener.accept().await()?
  var echoed = 0
  while const Some(chunk) = connection.receive().await()? {
    echoed = echoed + chunk.length()
    connection.send(chunk).await()?
  }
  connection.shutdown()?
  echoed
}

Declarations#

Addresses#

public use AddressError, Ipv4Address, Ipv6Address, IpAddress, SocketAddress from "std/ip"

The address values are std/ip's, re-exported here: a package that only names an address imports std/ip and does not claim the network, and a program that connects imports them from here as it always did. Both names are one type.

resolve#

fn resolve(host: String): Task<Result<List<IpAddress>, NetworkError>>

The addresses a host name stands for. A literal address answers itself and localhost answers the two loopback addresses without asking anybody, so a program that talks to itself needs no resolver; every other name goes to the system's resolver, on a thread of the runtime.

lookup and Resolver#

type Resolver with NameResolver {
  servers: List<SocketAddress> = []
  attemptMilliseconds: Int = 2000
  rounds: Int = 2
  fn lookup(name: DomainName, recordType: RecordType): Task<Result<List<Record>, NetworkError>>
  fn exchange(name: DomainName, recordType: RecordType): Task<Result<Message, NetworkError>>
}
trait NameResolver {
  fn lookup(name: DomainName, recordType: RecordType): Task<Result<List<Record>, NetworkError>>
  fn exchange(name: DomainName, recordType: RecordType): Task<Result<Message, NetworkError>>
}
fn lookup(name: DomainName, recordType: RecordType): Task<Result<List<Record>, NetworkError>>
fn systemNameServers(): Result<List<SocketAddress>, NetworkError>
fn randomQueryIdentifier(): Int

A stub resolver over the messages of std/dns: the records of a type that a name has - MX, TXT, SRV, HTTPS, or addresses - asked of a recursive name server. lookup answers the records, CNAME chains followed, an empty list for a name without such records, and isHostNotFound() for a name that does not exist; exchange answers the whole response. Each attempt is a UDP datagram with a random identifier from a new socket; a truncated answer is asked again over TCP; no answer within attemptMilliseconds, a refusal, or SERVFAIL, REFUSED and NOTIMP hand the question to the next server, for rounds rounds. An empty servers asks the ones the system is configured with - systemNameServers(): the adapters' on Windows, /etc/resolv.conf elsewhere - and the free lookup is Resolver()'s. There is no cache: resolve stays what connecting uses, with the system's cache and hosts file. DNS over TLS is NameResolver is the trait the three resolvers share - Resolver, std/tls's TlsResolver and std/http's DohResolver - so a Client can connect through any of them. TlsResolver draws its identifiers from randomQueryIdentifier() too.

use Resolver, SocketAddress, IpAddress, NetworkError from "std/network"
use DomainName, RecordType from "std/dns"

/** The mail exchanges of `domain`, lowest preference first, as a server on this machine answers them. */
fn mailExchanges(domain: String): Task<Result<List<String>, NetworkError>> {
  const resolver = Resolver servers: [SocketAddress(IpAddress.loopback, 53)]
  const records = resolver.lookup(DomainName.tryFrom(domain)?, RecordType.Mx).await()?
  records.map({ _.data.show() }).toList()
}

TcpListener#

shared type TcpListener with Close {
  static fn listen(address: SocketAddress, backlog: Int = 128): Result<Self, NetworkError>
  fn localAddress(): SocketAddress
  fn accept(): Task<Result<TcpStream, NetworkError>>
  fn acceptor(): TcpAcceptor
  fn stop()
  var fn close()
}
type TcpAcceptor {
  fn accept(): Task<Result<TcpStream, NetworkError>>
}

A socket that listens. listen is synchronous - binding never waits for the network - and port 0 asks the system for a free port, which localAddress() then says. An IPv6 listener hears IPv6 only, on every system alike. accept waits for the next connection; an accept that is cancelled leaves the connection for the next one. stop() closes the socket before the listener is released, and an accept that waits then fails with isClosed() - which is how a server stops accepting while it is shut down. Otherwise the socket closes when the last reference to the listener goes. acceptor() hands out the right to accept as a value - the socket's handle and nothing else - which a task may take to another worker: several accept loops on one listening socket, each connection going to one of them, are how a server serves on every core. The listener still owns the socket, and once it stops every acceptor's accept fails with isClosed().

TcpStream#

shared type TcpStream with Close {
  static fn of(handle: Int): Self
  static fn connect(address: SocketAddress): Task<Result<Self, NetworkError>>
  static fn connectTo(host: String, port: Int): Task<Result<Self, NetworkError>>
  fn localAddress(): Result<SocketAddress, NetworkError>
  fn remoteAddress(): Result<SocketAddress, NetworkError>
  fn receive(maximum: Int = 65536): Task<Result<Bytes?, NetworkError>>
  fn receiveBuffer(maximum: Int = 65536): Task<Result<Buffer<UInt8>?, NetworkError>>
  fn send(bytes: Bytes): Task<Result<Void, NetworkError>>
  fn sendBuffer(bytes: Buffer<UInt8>): Task<Result<Void, NetworkError>>
  fn shutdown(): Result<Void, NetworkError>
  fn setNoDelay(enabled: Bool): Result<Void, NetworkError>
  fn noDelay(): Result<Bool, NetworkError>
  fn setKeepAlive(idle: Duration?, interval: Duration? = None, count: Int? = None): Result<Void, NetworkError>
  fn keepAlive(): Result<Duration?, NetworkError>
  fn source(chunk: Int = 65536): TcpSource
  fn sink(): TcpSink
  var fn close()
}

One connection. receive answers as soon as any bytes arrived, and None once the peer ended its half; nothing is read before somebody asks, so a slow reader slows the peer down. send finishes once the operating system took the bytes. shutdown ends this half: the peer reads the end of the stream and may still answer. connectTo resolves the host and tries its addresses in turn. source() and sink() are the two directions as a Source<Bytes, NetworkError> and a Sink<Bytes, NetworkError> (see std/stream); the socket closes when the last of the stream, its source and its sink is gone.

The options of the connection are methods of the stream. TCP_NODELAY is on, and setNoDelay(false) switches it off; noDelay() reads it. TCP keepalive is off, and it is how a long-lived connection notices a dead peer: setKeepAlive(idle: Some(60.seconds())) turns it on with the idle time before the first probe, interval and count set the time between probes and how many go unanswered before the connection is dropped (the system's own values stay where they are None), and setKeepAlive(None) turns it off again. keepAlive() answers the idle time, or None while it is off. Times are whole seconds, at least 1: a shorter Duration is rounded up, and a Duration of zero or less or a count below 1 is refused with a NetworkError before anything is changed. Only the arguments are checked up front: the times are then set one at a time, so a system failure midway can leave the earlier ones changed. Every one fails with isClosed() on a closed stream, and none needs more than the network capability the stream was made with. Windows before 10 1709 has no keepalive times, and answers the system's failure.

using stream = TcpStream.connectTo("db.example", 5432).await()?
stream.setKeepAlive(Some(60.seconds()), interval: Some(10.seconds()), count: Some(5))?
print stream.keepAlive()?
// prints Some(60s)

UdpSocket#

shared type UdpSocket with Close {
  static fn bind(address: SocketAddress): Result<Self, NetworkError>
  fn localAddress(): SocketAddress
  fn connect(address: SocketAddress): Result<Void, NetworkError>
  fn peerAddress(): SocketAddress?
  fn send(bytes: Bytes, to: SocketAddress): Task<Result<Void, NetworkError>>
  fn sendToPeer(bytes: Bytes): Task<Result<Void, NetworkError>>
  fn receive(): Task<Result<(Bytes, SocketAddress), NetworkError>>
  var fn close()
}

A UDP socket: datagrams, each one whole or not at all, with the address it came from. A datagram is not a stream, so the socket is no Source: receive answers the next datagram and its sender - a datagram without bytes is one too - and send hands one datagram to the operating system, which says nothing of whether it arrives. bind is synchronous, port 0 asks for a free port, and an IPv6 socket binds IPv6 only. connect names one peer without sending anything: from then on only the peer's datagrams arrive, sendToPeer sends to it, send accepts no other address, and a peer whose port is closed makes the next receive fail with isConnectionRefused() - on every system alike, where an unconnected socket never hears of such a refusal. A datagram that arrives while nobody receives waits in the socket, and a cancelled receive leaves it to the next one; a timeout is within, which is how a protocol over UDP asks again.

use UdpSocket, SocketAddress, IpAddress, NetworkError from "std/network"

/** Answers one datagram with its bytes reversed, to whoever sent it. */
fn answerOnce(socket: UdpSocket): Task<Result<Void, NetworkError>> {
  const (bytes, sender) = socket.receive().await()?
  socket.send(bytes.reversed(), to: sender).await()
}

UnixListener and UnixStream#

shared type UnixListener with Close {
  static fn listen(path: String, backlog: Int = 128): Result<Self, NetworkError>
  fn path(): String
  fn accept(): Task<Result<UnixStream, NetworkError>>
  fn stop()
  var fn close()
}
shared type UnixStream with Close {
  static fn of(handle: Int): Self
  static fn connect(path: String): Task<Result<Self, NetworkError>>
  fn receive(maximum: Int = 65536): Task<Result<Bytes?, NetworkError>>
  fn send(bytes: Bytes): Task<Result<Void, NetworkError>>
  fn shutdown(): Result<Void, NetworkError>
  fn source(chunk: Int = 65536): UnixSource
  fn sink(): UnixSink
  var fn close()
}

The TCP types over a path of the file system, for the local sockets of a database, a cache or a container engine: a stream socket, no datagrams. source() and sink() answer a UnixSource (a Source<Bytes, NetworkError>) and a UnixSink (a Sink<Bytes, NetworkError>), as a TcpStream's do. A path is a String. UnixListener.listen is synchronous and creates the socket file; it fails with isAddressInUse() where something is at the path already - the file of a server that did not clean up has to be removed by the program that knows it is stale - and a path that does not fit the system's sun_path (108 bytes on Linux and Windows, 104 on macOS and the BSDs, the terminating zero included), is empty or holds a NUL byte is refused with a NetworkError, never cut short. accept is TCP's: a task, and a cancelled accept leaves the connection to the next one. Closing the listener, with stop() or by releasing it, removes the socket file it bound, and only that one: a file that was put there meanwhile stays. UnixStream.connect fails with isUnreachable() where there is no file at the path and with isConnectionRefused() where there is one and nobody listens. A stream is used as a TcpStream is: receive answers None once the peer ended its half, shutdown ends this half and the peer may still answer, source() and sink() are the two directions (end() of the sink is the half close), and the socket closes when the last of them is gone.

What a Unix socket does not have: no address (an accepted stream has no peer path, and there is no localAddress), no socket options (those of a TcpStream are not here), no acceptor() yet, and no abstract names, datagrams, peer credentials or descriptor passing. A sandbox that grants no Unix socket stops the script at the call; a unix: grant is not there yet.

using listener = UnixListener.listen("/run/app/control.sock")?
const accepting = listener.accept()
using client = UnixStream.connect("/run/app/control.sock").await()?
using server = accepting.await()?
client.send("ping".bytes()).await()?
const chunk = server.receive().await()?
// Some(the four bytes of "ping")

NetworkError#

type NetworkError with Show, Error {
  fn isConnectionRefused(): Bool
  fn isConnectionReset(): Bool
  fn isTimedOut(): Bool
  fn isAddressInUse(): Bool
  fn isHostNotFound(): Bool
  fn isUnreachable(): Bool
  fn isClosed(): Bool
  fn isTlsFailure(): Bool
  fn isCertificateRejected(): Bool
  fn isNameServerFailure(): Bool
  fn isRetryable(): Bool
  fn show(): String
  static fn of(failure: Int): Self
  static fn withWords(failure: Int, words: String): Self
  static fn closed(): Self
  static fn timedOut(message: String): Self
  static fn nameServerFailed(message: String): Self
  static fn hostNotFound(host: String): Self
  static fn invalid(message: String): Self
  static fn invalidAddress(cause: AddressError): Self
  fn about(subject: String): Self
}
extend NetworkError with From<AddressError>
extend NetworkError with From<DnsError>

What went wrong, as one wrapper type: a new kind of failure is never a breaking change. The kind is a question, and show() is the operating system's own words. A cancelled wait is not one of its kinds: cancelling a task ends it, and result() of the task says Cancelled. isTlsFailure() and isCertificateRejected() are the failures of std/tls, which reports through this type too. isNameServerFailure() is a lookup's: a name server answered with a failure or with no DNS at all, or the system names none. A DnsError - a name that is no domain name - converts into a NetworkError, so DomainName.tryFrom(text)? works in a function that fails with one.

Related#

• std/ip - the address values this package connects to.
• std/tls - TLS over a TcpStream.
• std/dns - the messages of the Domain Name System, which a program sends over a UdpSocket.
• std/http - HTTP/1.1 over these streams.
• std/stream - Source, Sink and Bytes.
• std/task - within, the timeout of every network task, and cancel.
• The standard library - the other packages.