std/compression

Die generierte Referenz von std/compression zeigt jede öffentliche Deklaration mit Signatur, Doc-Kommentar und Beispielen.

std/compression reads and writes DEFLATE (RFC 1951), the gzip members around it (RFC 1952), and the block formats of Snappy and LZ4. DEFLATE and gzip are what a package archive of the registry is compressed with, and what an HTTP body with Content-Encoding: gzip is; the two block formats are what MongoDB's wire compression (Snappy) and Kafka (Snappy, LZ4) put around a message. Everything is ordinary TorbScript over Bytes, with no native. Every operation is a method of Bytes (Buffer<UInt8>): bytes.gzipped(), packed.gunzipped(). A file names the methods it uses, written with the list they extend: use Buffer.gzipped from "std/compression".

Import#

use Buffer.gzipped, Buffer.gunzipped, Buffer.deflated, Buffer.inflated, Buffer.crc32 from "std/compression"
use Buffer.snappyCompressed, Buffer.snappyDecompressed from "std/compression"
use Buffer.lz4BlockCompressed, Buffer.lz4BlockDecompressed from "std/compression"
use CompressionError from "std/compression"

use Buffer.gzipped, Buffer.gunzipped from "std/compression"

const bytes = "hello, hello, hello".bytes()
print(bytes.gzipped().gunzipped() == Ok(bytes))

use Buffer.snappyCompressed, Buffer.snappyDecompressed from "std/compression"
use Buffer.lz4BlockCompressed, Buffer.lz4BlockDecompressed from "std/compression"

const bytes = "hello, hello, hello, hello, hello".bytes()
print(bytes.snappyCompressed().snappyDecompressed() == Ok(bytes))
print(bytes.lz4BlockCompressed().lz4BlockDecompressed(100) == Ok(bytes))

Declarations#

gzipped(), gunzipped()#

extend Bytes {
  fn inflated(): Result<Bytes, CompressionError>
  fn deflated(): Bytes
  fn gzipped(): Bytes
  fn gunzipped(): Result<Bytes, CompressionError>
  fn snappyCompressed(): Bytes
  fn snappyDecompressed(): Result<Bytes, CompressionError>
  fn lz4BlockCompressed(): Bytes
  fn lz4BlockDecompressed(maximumSize: Int): Result<Bytes, CompressionError>
  fn crc32(): UInt32
}

bytes.gzipped() writes one member whose header says nothing about the machine or the moment - no file name, no time, the operating system "unknown" - so the same bytes give the same member everywhere. packed.gunzipped() reads every member of its bytes one after the other, skips the optional header fields, and checks each member's CRC-32 and length.

deflated(), inflated()#

Raw DEFLATE streams. Inflating reads every stream the format allows: stored blocks, the fixed codes and dynamic Huffman codes, decoded through a table of the next ten bits of the stream; a 30 MB release tarball gunzips in about 0.6 seconds in a release binary, about twice what gzip -d takes. Deflating is deterministic and simple: one block of the fixed codes, with a greedy search for matches along hash chains over the last 32 KiB. It compresses text to roughly half where zlib gets to a third, and every inflater reads it.

snappyCompressed(), snappyDecompressed()#

One Snappy block, as the format description of the Snappy project gives it: the length of what the block holds as a varint, then literals and copies with one-, two- and four-byte offsets. snappyDecompressed() reads every valid block and refuses a truncated length or element, a copy of offset 0 or one that reaches back before the start of the output, and elements that do not add up to the length in front. That length is checked before anything is allocated for it: a five-byte input that claims 4 GiB fails at once. snappyCompressed() is greedy, keeps one position per hash of four bytes and looks back as far as the input goes, so a match further than 64 KiB is written as a copy of four-byte offsets; the same bytes give the same block on every machine. A block is not framed and has no checksum: those are the Snappy frame format, which is not here.

lz4BlockCompressed(), lz4BlockDecompressed(maximumSize)#

One LZ4 block (lz4_Block_format.md of the LZ4 project): sequences of a token, literals, a two-byte offset and the rest of a match length. lz4BlockCompressed() keeps the end of block rules - the last match starts at least 12 bytes before the end and the last five bytes are literals - looks back at most 64 KiB, and is deterministic. A block does not hold the size of what it decodes to, so lz4BlockDecompressed(maximumSize) is told the largest output it accepts and fails when the output would grow past it, on a truncated sequence, on an offset of 0 or beyond the start of the output, and on a block that does not end with a sequence of literals. The LZ4 frame format, with its checksums, is not here.

crc32()#

The CRC-32 of ISO 3309 and ITU-T V.42, the checksum of gzip, zip and PNG. "123456789".bytes().crc32() is 0xCBF43926.

CompressionError#

type CompressionError with Show, Error {
  reason: String
  offset: Int
  fn show(): String
}
extend CompressionError with From<ReadError>

What is wrong with the compressed input, and the byte offset where it was found: a truncated stream, a code no symbol has, a distance back past the start of the output, a checksum that does not match, a Snappy block whose elements do not add up to its length.

Pitfalls#

Inflating and gunzipping limit nothing. A small gzip member can inflate to gigabytes. A caller that takes input from outside checks the size of what it got back, as the package manager does before it writes an archive to disk. A Snappy block states its length, which bounds the output, but a block may state a large one; an LZ4 block states none, which is why lz4BlockDecompressed takes the limit.

Related#

• std/archive - the tar a package archive is, inside the gzip.
• std/digest - SHA-256.
• The standard library - the other packages.