0
Fork 0
mirror of https://codeberg.org/ziglang/zig synced 2026-11-18 07:05:05 +00:00
zig/lib/std
Igor Anić d645114f7e add deflate implemented from first principles
Zig deflate compression/decompression implementation. It supports compression and decompression of gzip, zlib and raw deflate format.

Fixes #18062.

This PR replaces current compress/gzip and compress/zlib packages. Deflate package is renamed to flate. Flate is common name for deflate/inflate where deflate is compression and inflate decompression.

There are breaking change. Methods signatures are changed because of removal of the allocator, and I also unified API for all three namespaces (flate, gzip, zlib).

Currently I put old packages under v1 namespace they are still available as compress/v1/gzip, compress/v1/zlib, compress/v1/deflate. Idea is to give users of the current API little time to postpone analyzing what they had to change. Although that rises question when it is safe to remove that v1 namespace.

Here is current API in the compress package:

```Zig
// deflate
    fn compressor(allocator, writer, options) !Compressor(@TypeOf(writer))
    fn Compressor(comptime WriterType) type

    fn decompressor(allocator, reader, null) !Decompressor(@TypeOf(reader))
    fn Decompressor(comptime ReaderType: type) type

// gzip
    fn compress(allocator, writer, options) !Compress(@TypeOf(writer))
    fn Compress(comptime WriterType: type) type

    fn decompress(allocator, reader) !Decompress(@TypeOf(reader))
    fn Decompress(comptime ReaderType: type) type

// zlib
    fn compressStream(allocator, writer, options) !CompressStream(@TypeOf(writer))
    fn CompressStream(comptime WriterType: type) type

    fn decompressStream(allocator, reader) !DecompressStream(@TypeOf(reader))
    fn DecompressStream(comptime ReaderType: type) type

// xz
   fn decompress(allocator: Allocator, reader: anytype) !Decompress(@TypeOf(reader))
   fn Decompress(comptime ReaderType: type) type

// lzma
    fn decompress(allocator, reader) !Decompress(@TypeOf(reader))
    fn Decompress(comptime ReaderType: type) type

// lzma2
    fn decompress(allocator, reader, writer !void

// zstandard:
    fn DecompressStream(ReaderType, options) type
    fn decompressStream(allocator, reader) DecompressStream(@TypeOf(reader), .{})
    struct decompress
```

The proposed naming convention:
 - Compressor/Decompressor for functions which return type, like Reader/Writer/GeneralPurposeAllocator
 - compressor/compressor for functions which are initializers for that type, like reader/writer/allocator
 - compress/decompress for one shot operations, accepts reader/writer pair, like read/write/alloc

```Zig
/// Compress from reader and write compressed data to the writer.
fn compress(reader: anytype, writer: anytype, options: Options) !void

/// Create Compressor which outputs the writer.
fn compressor(writer: anytype, options: Options) !Compressor(@TypeOf(writer))

/// Compressor type
fn Compressor(comptime WriterType: type) type

/// Decompress from reader and write plain data to the writer.
fn decompress(reader: anytype, writer: anytype) !void

/// Create Decompressor which reads from reader.
fn decompressor(reader: anytype) Decompressor(@TypeOf(reader)

/// Decompressor type
fn Decompressor(comptime ReaderType: type) type

```

Comparing this implementation with the one we currently have in Zig's standard library (std).
Std is roughly 1.2-1.4 times slower in decompression, and 1.1-1.2 times slower in compression. Compressed sizes are pretty much same in both cases.
More resutls in [this](https://github.com/ianic/flate) repo.

This library uses static allocations for all structures, doesn't require allocator. That makes sense especially for deflate where all structures, internal buffers are allocated to the full size. Little less for inflate where we std version uses less memory by not preallocating to theoretical max size array which are usually not fully used.

For deflate this library allocates 395K while std 779K.
For inflate this library allocates 74.5K while std around 36K.

Inflate difference is because we here use 64K history instead of 32K in std.

If merged existing usage of compress gzip/zlib/deflate need some changes. Here is example with necessary changes in comments:

```Zig

const std = @import("std");

// To get this file:
// wget -nc -O war_and_peace.txt https://www.gutenberg.org/ebooks/2600.txt.utf-8
const data = @embedFile("war_and_peace.txt");

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    defer std.debug.assert(gpa.deinit() == .ok);
    const allocator = gpa.allocator();

    try oldDeflate(allocator);
    try new(std.compress.flate, allocator);

    try oldZlib(allocator);
    try new(std.compress.zlib, allocator);

    try oldGzip(allocator);
    try new(std.compress.gzip, allocator);
}

pub fn new(comptime pkg: type, allocator: std.mem.Allocator) !void {
    var buf = std.ArrayList(u8).init(allocator);
    defer buf.deinit();

    // Compressor
    var cmp = try pkg.compressor(buf.writer(), .{});
    _ = try cmp.write(data);
    try cmp.finish();

    var fbs = std.io.fixedBufferStream(buf.items);
    // Decompressor
    var dcp = pkg.decompressor(fbs.reader());

    const plain = try dcp.reader().readAllAlloc(allocator, std.math.maxInt(usize));
    defer allocator.free(plain);
    try std.testing.expectEqualSlices(u8, data, plain);
}

pub fn oldDeflate(allocator: std.mem.Allocator) !void {
    const deflate = std.compress.v1.deflate;

    // Compressor
    var buf = std.ArrayList(u8).init(allocator);
    defer buf.deinit();
    // Remove allocator
    // Rename deflate -> flate
    var cmp = try deflate.compressor(allocator, buf.writer(), .{});
    _ = try cmp.write(data);
    try cmp.close(); // Rename to finish
    cmp.deinit(); // Remove

    // Decompressor
    var fbs = std.io.fixedBufferStream(buf.items);
    // Remove allocator and last param
    // Rename deflate -> flate
    // Remove try
    var dcp = try deflate.decompressor(allocator, fbs.reader(), null);
    defer dcp.deinit(); // Remove

    const plain = try dcp.reader().readAllAlloc(allocator, std.math.maxInt(usize));
    defer allocator.free(plain);
    try std.testing.expectEqualSlices(u8, data, plain);
}

pub fn oldZlib(allocator: std.mem.Allocator) !void {
    const zlib = std.compress.v1.zlib;

    var buf = std.ArrayList(u8).init(allocator);
    defer buf.deinit();

    // Compressor
    // Rename compressStream => compressor
    // Remove allocator
    var cmp = try zlib.compressStream(allocator, buf.writer(), .{});
    _ = try cmp.write(data);
    try cmp.finish();
    cmp.deinit(); // Remove

    var fbs = std.io.fixedBufferStream(buf.items);
    // Decompressor
    // decompressStream => decompressor
    // Remove allocator
    // Remove try
    var dcp = try zlib.decompressStream(allocator, fbs.reader());
    defer dcp.deinit(); // Remove

    const plain = try dcp.reader().readAllAlloc(allocator, std.math.maxInt(usize));
    defer allocator.free(plain);
    try std.testing.expectEqualSlices(u8, data, plain);
}

pub fn oldGzip(allocator: std.mem.Allocator) !void {
    const gzip = std.compress.v1.gzip;

    var buf = std.ArrayList(u8).init(allocator);
    defer buf.deinit();

    // Compressor
    // Rename compress => compressor
    // Remove allocator
    var cmp = try gzip.compress(allocator, buf.writer(), .{});
    _ = try cmp.write(data);
    try cmp.close(); // Rename to finisho
    cmp.deinit(); // Remove

    var fbs = std.io.fixedBufferStream(buf.items);
    // Decompressor
    // Rename decompress => decompressor
    // Remove allocator
    // Remove try
    var dcp = try gzip.decompress(allocator, fbs.reader());
    defer dcp.deinit(); // Remove

    const plain = try dcp.reader().readAllAlloc(allocator, std.math.maxInt(usize));
    defer allocator.free(plain);
    try std.testing.expectEqualSlices(u8, data, plain);
}

```
History 2024-02-14 18:28:20 +01:00
..
Build Merge pull request #18712 from Vexu/std.options 2024-02-09 13:38:42 -08:00
c std.os.speed_t: add type safety 2024-02-12 21:49:09 -07:00
compress add deflate implemented from first principles 2024-02-14 18:28:20 +01:00
crypto x86_64: implement @byteSwap of big integers 2024-02-12 05:25:07 +01:00
dwarf openbsd: add proper support for aarch64 2024-01-18 15:45:42 -08:00
fmt lib: correct unnecessary uses of 'var' 2023-11-19 09:55:07 +00:00
fs some API work on std.c, std.os, std.os.wasi 2024-02-11 13:38:55 -07:00
hash Replace std.rand references with std.Random 2024-02-08 15:21:35 +01:00
heap Replace std.rand references with std.Random 2024-02-08 15:21:35 +01:00
http add deflate implemented from first principles 2024-02-14 18:28:20 +01:00
io Replace std.rand references with std.Random 2024-02-08 15:21:35 +01:00
json Revert "Merge pull request #18410 from dweiller/by-length-slice-bug" 2024-01-13 23:21:44 -07:00
math x86_64: implement @clz and @ctz of big integers 2024-02-12 05:25:07 +01:00
mem mem: explicit dupe and dupeZ error on Allocator 2023-09-07 21:56:57 +03:00
meta AstGen: add error for redundant comptime var in comptime scope (#18242) 2024-01-09 20:09:39 -05:00
net remove std.io.Mode 2024-02-01 15:22:36 +02:00
os std.posix.termios: bring V back 2024-02-13 20:10:32 -08:00
Random x86_64: implement shifts of big integers 2024-02-12 05:25:07 +01:00
sort lib: correct unnecessary uses of 'var' 2023-11-19 09:55:07 +00:00
tar tar: rename reader to iterator 2024-01-13 19:37:33 -07:00
Target std.Target.Query: avoid using builtin.target.abi 2024-02-02 20:43:01 -07:00
testing std: enable FailingAllocator to fail on resize 2023-09-06 19:06:32 +03:00
Thread Add timedWait to std.Thread.Semaphore (#18805) 2024-02-13 11:51:42 -06:00
time all: migrate code to new cast builtin syntax 2023-06-24 16:56:39 -07:00
tz Support legacy TZ format, expose header struct to a potential writer 2022-01-01 12:47:08 +00:00
unicode all: migrate code to new cast builtin syntax 2023-06-24 16:56:39 -07:00
valgrind valgrind.memcheck: fix makeMem*() 2023-10-03 02:51:01 -07:00
zig remove std.io.Mode 2024-02-01 15:22:36 +02:00
array_hash_map.zig std: remove meta.trait 2023-11-22 13:24:27 -05:00
array_list.zig std.ArrayList: pedantic fixups to previous commit 2024-01-19 16:24:51 -07:00
ascii.zig migration: std.math.{min, min3, max, max3} -> @min & @max 2023-06-16 13:44:09 -07:00
atomic.zig rework std.atomic 2023-11-22 19:08:55 -07:00
base64.zig lib: correct unnecessary uses of 'var' 2023-11-19 09:55:07 +00:00
bit_set.zig Merge pull request #17852 from ziglang/zig-reduce 2023-11-04 14:25:50 -04:00
BitStack.zig std.json: Unify stringify and writeStream (#16405) 2023-07-21 19:56:46 -04:00
bounded_array.zig Smaller memory footprint for BoundedArray (#16299) 2023-07-03 13:58:03 -04:00
buf_map.zig lib: correct unnecessary uses of 'var' 2023-11-19 09:55:07 +00:00
buf_set.zig lib: correct unnecessary uses of 'var' 2023-11-19 09:55:07 +00:00
Build.zig std.Build: use a runtime panic for wrong dependency API 2024-02-02 20:43:01 -07:00
builtin.zig Merge pull request #18875 from ziglang/macho-zo-dwarf 2024-02-09 23:12:04 +01:00
c.zig std.posix.termios: bring V back 2024-02-13 20:10:32 -08:00
child_process.zig some API work on std.c, std.os, std.os.wasi 2024-02-11 13:38:55 -07:00
coff.zig lib: correct unnecessary uses of 'var' 2023-11-19 09:55:07 +00:00
compress.zig add deflate implemented from first principles 2024-02-14 18:28:20 +01:00
comptime_string_map.zig do not enforce function parameters to be marked comptime if only called at comptime 2024-01-19 15:31:18 -08:00
crypto.zig x86_64: implement 128-bit builtins 2023-10-23 22:42:18 -04:00
debug.zig add deflate implemented from first principles 2024-02-14 18:28:20 +01:00
dwarf.zig lib: correct unnecessary uses of 'var' 2023-11-19 09:55:07 +00:00
dynamic_library.zig some API work on std.c, std.os, std.os.wasi 2024-02-11 13:38:55 -07:00
elf.zig elf: init output COMDAT group sections 2023-11-09 17:41:14 +01:00
enums.zig Work around stage1 not yet returning null-terminated @typeInfo strings 2024-01-07 16:21:08 +01:00
fifo.zig std: eliminate some uses of usingnamespace 2024-02-01 20:30:42 +00:00
fmt.zig x86_64: fix assert location 2024-02-12 05:25:07 +01:00
fs.zig remove std.io.Mode 2024-02-01 15:22:36 +02:00
gpu.zig std: add gpu namespace 2024-02-05 11:55:14 +03:30
hash.zig XXH3 Implementation for Zig STD (#17530) 2023-10-17 13:55:01 -05:00
hash_map.zig Replace std.rand references with std.Random 2024-02-08 15:21:35 +01:00
heap.zig std: eliminate some uses of usingnamespace 2024-02-01 20:30:42 +00:00
http.zig Remove some @as coercions from assertions 2024-01-03 21:20:48 +01:00
Ini.zig add std.Ini for basic .ini file parsing 2023-01-11 15:39:48 -08:00
io.zig Merge pull request #18712 from Vexu/std.options 2024-02-09 13:38:42 -08:00
json.zig x86_64: pass more tests 2023-10-25 04:28:30 -04:00
leb128.zig Dwarf: improve x86_64 backend debug info 2023-11-12 03:21:52 -05:00
linked_list.zig Revert "std.SinglyLinkedList: add sort function" 2023-11-24 22:33:50 -07:00
log.zig Buffer the logging function 2024-02-09 14:02:57 -08:00
macho.zig std.macho: add segment flags 2024-01-24 12:48:12 +01:00
math.zig std.math: add wrap function (#18154) 2024-01-14 20:45:39 -05:00
mem.zig x86_64: implement @clz and @ctz of big integers 2024-02-12 05:25:07 +01:00
meta.zig std.fmt: fix formatting slices of structs with custom formatting 2024-02-03 21:47:55 +00:00
multi_array_list.zig Correct expected/actual parameter order of some assertions 2024-01-03 21:20:49 +01:00
net.zig remove std.io.Mode 2024-02-01 15:22:36 +02:00
once.zig x86_64: fix std test failures 2023-11-03 23:18:21 -04:00
os.zig std.posix.termios: bring V back 2024-02-13 20:10:32 -08:00
packed_int_array.zig std.builtin.Endian: make the tags lower case 2023-10-31 21:37:35 -04:00
pdb.zig remove std.io.Mode 2024-02-01 15:22:36 +02:00
priority_dequeue.zig Replace std.rand references with std.Random 2024-02-08 15:21:35 +01:00
priority_queue.zig lib: correct unnecessary uses of 'var' 2023-11-19 09:55:07 +00:00
process.zig doc: fix typo in getEnvMap 2024-02-07 12:41:40 +02:00
Progress.zig lib: correct unnecessary uses of 'var' 2023-11-19 09:55:07 +00:00
Random.zig Fixup filename casing 2024-02-08 15:39:28 +01:00
RingBuffer.zig std.RingBuffer.zig: Fix typo in readLastAssumeLength 2023-11-21 13:55:09 +02:00
segmented_list.zig x86_64: implement enough to pass unicode tests 2023-10-23 22:42:18 -04:00
SemanticVersion.zig std.zig.system.NativeTargetInfo: fix glibc version parsing 2023-10-23 06:00:53 -04:00
simd.zig Deprecate suggestVectorSize in favor of suggestVectorLength 2024-01-01 16:18:57 +01:00
sort.zig Replace std.rand references with std.Random 2024-02-08 15:21:35 +01:00
start.zig std.start: remove event loop integration 2024-02-01 15:22:36 +02:00
start_windows_tls.zig all: migrate code to new cast builtin syntax 2023-06-24 16:56:39 -07:00
std.zig some API work on std.c, std.os, std.os.wasi 2024-02-11 13:38:55 -07:00
tar.zig Permits tar directory path without trailing slash 2024-01-15 18:31:32 +02:00
Target.zig spirv: basic shader support 2024-02-05 11:55:14 +03:30
testing.zig std.testing.expectEqualSlices: some improvements 2024-01-08 00:13:21 -07:00
Thread.zig std.os.linux.MAP: use a packed struct 2024-02-06 21:12:11 -07:00
time.zig some API work on std.c, std.os, std.os.wasi 2024-02-11 13:38:55 -07:00
treap.zig Replace std.rand references with std.Random 2024-02-08 15:21:35 +01:00
tz.zig std.builtin.Endian: make the tags lower case 2023-10-31 21:37:35 -04:00
unicode.zig Fix fmt UTF-8 characters as fill (#18533) 2024-01-13 22:47:03 -05:00
Uri.zig std.Uri: change specifier for printing with scheme to semicolon ; 2024-01-18 09:13:04 +02:00
valgrind.zig Fix simple doc mistakes. (#17624) 2023-10-21 21:24:55 +00:00
wasm.zig fix typo in wasm atomic opcodes doc comment 2024-02-04 00:25:05 +01:00
zig.zig Zir: store extra source hashes required for incremental 2024-02-04 18:38:39 +00:00