std/archive

The generated reference of std/archive has every public construct with its signature, its doc comment and its examples.

std/archive reads and writes tar archives. bytes.untarred() reads what the tar programs in use write - POSIX ustar, pax (POSIX.1-2001) and GNU tar's own format - and entries.tarred() writes ustar, with a pax header where a path does not fit. A package of the registry is a tar inside gzip (std/compression), and so is a release of the toolchain, which torb upgrade unpacks with this package.

Every operation is a method of the list it works on: tarred() and extract() of a List<TarEntry>, untarred() of Bytes (Buffer<UInt8>). A file names the methods it uses, written with the list they extend: use List.tarred from "std/archive".

tarred() puts nothing of the machine into an archive: no owner and no group, and every mode and time the entry's own - 0644 and 0 unless an entry says otherwise - so the same entries are the same bytes on every machine.

Reading and writing are pure. Writing entries out below a directory is extract() of the module std/archive/extract, which refuses every entry that would land outside of it; it is a module of its own because it reaches the file system, and a package of the registry that imports it has the capability files.

Import#

use TarEntry, TarKind, ArchiveError from "std/archive"
use List.tarred, Buffer.untarred from "std/archive"
use ExtractError from "std/archive/extract"
use List.extract from "std/archive/extract"

use TarEntry, TarKind from "std/archive"
use List.tarred, Buffer.untarred from "std/archive"

const entries = [
  TarEntry.directory("tool"),
  TarEntry("tool/run", "#!/bin/sh\n".bytes(), TarKind.File, 0b111_101_101),
  TarEntry.symbolicLink("tool/latest", "run"),
]
match entries.tarred() {
  Ok(archive) => print archive.untarred().map({ found => found.map({ _.path }).toList() })
  Fail(problem) => print problem
}

Declarations#

TarEntry#

type TarEntry {
  path: String
  bytes: Bytes
  kind: TarKind = TarKind.File
  mode: Int = 0b110_100_100
  modified: Int = 0
  static fn directory(path: String, mode: Int = 0b111_101_101): Self
  static fn symbolicLink(path: String, target: String): Self
}

One entry of an archive: its path, with / between the directories and none at the end, its bytes, what it is, its permission bits the way POSIX writes them (0b111_101_101 is 0755, a program; the set-id and sticky bits above them), and the time it was last written in seconds since 1970. TarEntry(path, bytes) is a file of 0644 and time 0. A directory and a link have no bytes.

TarKind#

type TarKind with Show, Equals, Hash {
  case File
  case Directory
  case SymbolicLink(target: String)
  case HardLink(target: String)
}

What an entry is. The target of a symbolic link is read from the directory of the link; the target of a hard link is another path of the same archive.

tarred()#

extend List<TarEntry> {
  fn extract(into: String, stripComponents: Int = 0): Result<Void, ExtractError>
}
extend List<TarEntry> {
  fn tarred(): Result<Bytes, ArchiveError>
}

The entries as an archive, sorted by path - the hard links after the rest, so that what each names is unpacked before it - and closed by two blocks of zeros. A path that fits ustar's fields (100 bytes, or a prefix of 155 and a name of 100 split at a /) is written in them, so such an archive is plain ustar. A longer path, a link target of more than 100 bytes, a size of 8 GiB or more and a time before 1970 or after 2242 go into a pax header (x) in front of the entry instead - pax rather than GNU's long names, because pax is POSIX, every tar of the last twenty years reads it and it has no limit of length. An empty path, two entries of one path and a directory or a link with bytes are an ArchiveError.

untarred()#

extend Bytes {
  fn untarred(): Result<List<TarEntry>, ArchiveError>
}

The entries of an archive, in the order it holds them: files, directories, symbolic links and hard links, each with its mode and time. What the formats put around an entry is read into it and is no entry of its own: GNU's long names (L for a path, K for a link target), ustar's prefix, and pax headers - x for the next entry, g for every one after it, with path, linkpath, size and mtime read and every other key ignored. GNU's base-256 numbers, the old format's directory (a file whose name ends in /) and a checksum of signed bytes are read too. A device, a pipe, a sparse file and a type the reader does not know are refused, and so are a header whose checksum does not match, a pax record that breaks its format and an archive that ends inside an entry.

ArchiveError#

type ArchiveError with Show, Error {
  reason: String
  offset: Int
  fn show(): String
}
extend ArchiveError with From<ReadError>

What is wrong, and the byte of the archive where it was found.

extract()#

Of std/archive/extract. Writes the entries out below the directory into, creating it: every directory, every file with its bytes and its mode, every hard link as a copy of the file it names, and the symbolic links last. stripComponents drops that many leading directories from every path, as tar --strip-components does; an entry left with no path is skipped, and a later entry of a path replaces an earlier one.

It is safe by default. Every entry is checked before the first byte is written, and one that fails stops the whole extraction:

• a path that is absolute, names a drive (any :), or holds .. - with \ counted as a separator, as Windows reads it;
• a path that leads through a symbolic link, of the archive or already below into;
• a symbolic link whose target is absolute, climbs higher than the link's own directory, or climbs with a .. after a name - that .. follows whatever the name is, and the name may be another link (a -> .. beside b -> a/d/a/..), under another spelling too on a system that ignores case, so only a target that climbs first and then descends is sure to stay inside;
• a hard link that does not name a file of the archive written before it.

A symbolic link already where an entry goes is replaced, not followed. The mode loses the set-id and sticky bits and the write bits of group and others, as the usual umask 022 does, and a directory gets its mode last, so a read-only one is filled first. On Windows the mode sets only the read-only attribute, where the owner's write bit is clear.

ExtractError#

public type ExtractError with Show, Error {
  case Refused(path: String, reason: String)
  case Failed(problem: IoError)
}

An entry extract() refused, with its path in the archive and why, or a failure of the file system, with what it wrote before it left in place.

Pitfalls#

untarred() keeps a path as the archive says it. An entry named ../../etc/passwd or /tmp/x comes back with exactly that path, and a link with exactly its target. Only extract() decides where a file may go; whoever writes entries out another way checks every path first, as the package manager does.

Windows allows a symbolic link only in developer mode or with the privilege to. Without either, extract() of an archive that has one fails with Operation not permitted after everything else is written.

The whole archive is in memory. untarred() reads bytes and answers a list; an archive of a few hundred megabytes needs that much memory twice.

Related#

• std/compression - the gzip around the archive.
• std/fs - what extract() writes with, and Permissions, the mode as a type.
• std/digest - SHA-256, which the tree hash of a package is made of.
• The standard library - the other packages.