Reference

std/compression/lib

std/compression/src/lib.trb

Compressed bytes: DEFLATE (RFC 1951) with Bytes.deflated and Bytes.inflated, gzip (RFC 1952) around it with Bytes.gzipped and Bytes.gunzipped, and the Bytes.crc32 gzip checks a member with; and the two byte-oriented block formats of the Snappy and LZ4 projects, Bytes.snappyCompressed with Bytes.snappyDecompressed and Bytes.lz4BlockCompressed with Bytes.lz4BlockDecompressed. All of them are methods of Bytes, imported by the name of the list they extend: use Buffer.gzipped from "std/compression". What a package archive of the registry is compressed with (docs/design/RELEASE.md section 7.2), and what an HTTP body with Content-Encoding: gzip is.

Ordinary TorbScript over Bytes, with no native. Inflating reads every stream the format allows - stored, fixed and dynamic blocks, and gzip members one after the other. Deflating is deterministic and simple: one block of the fixed codes, with a greedy match search over the last 32 KiB. It compresses text to roughly half, where zlib's dynamic codes get to a third; what it writes is the same for the same bytes on every machine, which is what a reproducible archive needs, and any inflater reads it.

Snappy and LZ4 trade ratio for speed: literals and copies of whole bytes, no entropy coding and no checksum, which is what the wire formats of MongoDB (Snappy) and Kafka (Snappy, LZ4) put around a message. Both write one block - the frame formats come beside them - and both compressors are greedy and deterministic. A Snappy block holds the length of what it decompresses to; an LZ4 block does not, so Bytes.lz4BlockDecompressed is told the largest output it accepts.

Examples

use Buffer.gzipped, Buffer.gunzipped from "std/compression"
use Buffer.snappyCompressed, Buffer.snappyDecompressed from "std/compression"

const bytes = "hello, hello, hello".bytes()
const packed = bytes.gzipped()
print(packed.gunzipped() == Ok(bytes))
print(bytes.snappyCompressed().snappyDecompressed() == Ok(bytes))

extend Bytes

extend Bytes

The operations of the module, written where they are imported from: a member another package names is found in the module of the package that is imported, so use Buffer.gzipped from "std/compression" needs them here.

fn inflated

fn inflated(): Result<Bytes, CompressionError>

Everything these bytes hold as a raw DEFLATE stream, which has to end with its last block.

fn deflated

fn deflated(): Bytes

These bytes as a raw DEFLATE stream: one final block of the fixed Huffman codes.

fn gzipped

fn gzipped(): Bytes

These bytes as one gzip member, with nothing in its header that depends on the machine or the moment: no file name, no time, the operating system "unknown". The same bytes give the same member everywhere.

fn gunzipped

fn gunzipped(): Result<Bytes, CompressionError>

What every gzip member of these bytes holds, one after the other. Each member's checksum and length are checked.

Errors

  • CompressionError where a header is not gzip's, a method is not DEFLATE, the DEFLATE stream is broken, or a checksum or a length does not match what was inflated.

fn snappyCompressed

fn snappyCompressed(): Bytes

These bytes as one Snappy block: the length of the bytes, then literals and copies. The same bytes give the same block everywhere.

Panics

When there are more than 4294967295 bytes, which is the most a Snappy block can say it holds.

fn snappyDecompressed

fn snappyDecompressed(): Result<Bytes, CompressionError>

The bytes of one Snappy block, which is the whole input. The length at its start has to be what the elements produce, and is checked before anything is allocated for it.

Errors

  • CompressionError where the input ends inside the length or an element, a copy has the offset 0 or reaches back before the start of the output, or the elements do not add up to the length the block starts with.

fn lz4BlockCompressed

fn lz4BlockCompressed(): Bytes

These bytes as one LZ4 block, with the end of block rules of the format kept: the last match starts at least 12 bytes before the end, and the last five bytes are literals. The same bytes give the same block everywhere.

fn lz4BlockDecompressed

fn lz4BlockDecompressed(maximumSize: Int): Result<Bytes, CompressionError>

The bytes of one LZ4 block, which is the whole input and holds no size of its own: maximumSize is the largest output the caller accepts, and nothing is written past it.

Errors

  • CompressionError where the input ends inside a sequence, a match has the offset 0 or reaches back before the start of the output, the block does not end with literals, or the output would be larger than maximumSize.

fn crc32

fn crc32(): UInt32

The CRC-32 of ISO 3309 and ITU-T V.42 - the reflected polynomial 0xEDB88320 - of these bytes.