Pure Go Brotli encoder and decoder

Related tags

Network brotli
Overview
Comments
  • Panic in BestSpeed compression (version 1.0.1)

    Panic in BestSpeed compression (version 1.0.1)

    pelase check the attched zip js file: marked.js.zip 1.0.0 is ok!

    /usr/local/Cellar/go/1.16/libexec/src/runtime/panic.go:971 +0x499
    github.com/andybalholm/brotli.compressFragmentFastImpl(0xc0002c1380, 0x31, 0x40, 0x31, 0x0, 0xc0002f8a68, 0x400, 0x400, 0x9, 0xc0002faa88, ...)
        /pkg/mod/github.com/andybalholm/[email protected]/compress_fragment.go:382 +0x23fb
    github.com/andybalholm/brotli.compressFragmentFast(0xc0002c1380, 0x31, 0x40, 0x31, 0xc0002f8a00, 0xc0002f8a68, 0x400, 0x400, 0x200, 0xc0002faa88, ...)
        /pkg/mod/github.com/andybalholm/[email protected]/compress_fragment.go:673 +0x418
    github.com/andybalholm/brotli.encoderCompressStreamFast(0xc0002f8800, 0x0, 0xc000816448, 0xc000816458, 0x16c4980)
        /pkg/mod/github.com/andybalholm/[email protected]/encode.go:971 +0x314
    github.com/andybalholm/brotli.encoderCompressStream(0xc0002f8800, 0x0, 0xc000816448, 0xc000816458, 0x10a5680)
        /pkg/mod/github.com/andybalholm/[email protected]/encode.go:1101 +0x369
    github.com/andybalholm/brotli.(*Writer).writeChunk(0xc0002f8800, 0xc0002c1380, 0x31, 0x40, 0x0, 0x1669a20, 0x0, 0x385f0b8)
        /pkg/mod/github.com/andybalholm/[email protected]/writer.go:77 +0xba
    github.com/andybalholm/brotli.(*Writer).Write(0xc0002f8800, 0xc0002c1380, 0x31, 0x40, 0xc0002f8800, 0x0, 0x3860b18)
        /pkg/mod/github.com/andybalholm/[email protected]/writer.go:111 +0x52
    github.com/ije/rex.(*responseWriter).Write(0xc0001de400, 0xc0002c1380, 0x31, 0x40, 0x163e6e0, 0x16bce00, 0x1c00000000000001)
    
    opened by ije 8
  • Performance: ensureRingBuffer allocating huge amount of memory

    Performance: ensureRingBuffer allocating huge amount of memory

    brotli.ensureRingBuffer is allocating more than 17 gb of memory and it significantly impact the performance.

       Total:     17.86GB    17.86GB (flat, cum) 68.44%
       1307            .          .           	var old_ringbuffer []byte = s.ringbuffer 
       1308            .          .           	if s.ringbuffer_size == s.new_ringbuffer_size { 
       1309            .          .           		return true 
       1310            .          .           	} 
       1311            .          .            
       1312      17.86GB    17.86GB           	s.ringbuffer = make([]byte, uint(s.new_ringbuffer_size)+uint(kRingBufferWriteAheadSlack)) 
       1313            .          .           	if s.ringbuffer == nil { 
       1314            .          .           		/* Restore previous value. */ 
       1315            .          .           		s.ringbuffer = old_ringbuffer 
       1316            .          .            
       1317            .          .           		return false 
    

    I suggest reusing buffer with sync.Pool.

    opened by cloudwindy 4
  • The compiled binary contains lot of test data

    The compiled binary contains lot of test data

    Issue description

    On static analysis of the binary, generated after compiling our go fiber app we found out a lot of text and links in the final binary, which turned out to be test data in the package.

    Please check this issue:

    https://github.com/gofiber/fiber/issues/1782

    opened by ahmad-kemsan 4
  • avoid initialising fields twice

    avoid initialising fields twice

    If you wouldn't mind looking over this, I've removed some redundant assignments for when the writer is initialised (they're set to 0 and false when the struct is created and it doesn't appear to be in a sync.Pool).

    There may also be the possibility of reusing the same quality settings instead of overwriting them on each reset?

    opened by Azareal 4
  • Serialize dictionnary state

    Serialize dictionnary state

    I want to use brotli for stream compression.

    I have a use case when i need to interrupt ( stop ) the stream compression for one week before continuing. For this I need to save the dictionary created already. I want to get a serialized state of the dynamic dictionary once the compression is stopped. So that, after, I will be able to de serialize the state and continue the process of compression using the dictionary created.

    Is this feature/api available within this library ?

    If no, I want to add it to this library, so from where should I start ? Can you give me some hints ?

    Are you interested in this use case to add it to the library ?

    opened by IhsenCharfi 3
  • Flush() does not completely process all data following a large Write()

    Flush() does not completely process all data following a large Write()

    Thanks for this library! It is great to have a pure go brotli encoder.

    I am seeing the Flush() calls not completely process all data following a call to a large Write(). Modifying your unit test TestEncoderFlush to use an input size of 32766 instead of 1000 demonstrates the behavior.

    func TestEncoderFlush(t *testing.T) {
      input := make([]byte, 32766) // MODIFIED
      rand.Read(input)
      out := bytes.Buffer{}
      e := NewWriterOptions(&out, WriterOptions{Quality: 5})
      in := bytes.NewReader(input)
      _, err := io.Copy(e, in)
      if err != nil {
        t.Fatalf("Copy Error: %v", err)
      }
      if err := e.Flush(); err != nil {
        t.Fatalf("Flush(): %v", err)
      }
      if out.Len() == 0 {
        t.Fatalf("0 bytes written after Flush()")
      }
      decompressed := make([]byte, 32766) // MODIFIED
      reader := NewReader(bytes.NewReader(out.Bytes()))
      n, err := reader.Read(decompressed)
      if n != len(decompressed) || err != nil {
        t.Errorf("Expected <%v, nil>, but <%v, %v>", len(decompressed), n, err)
      }
      if !bytes.Equal(decompressed, input) {
        t.Errorf(""+
          "Decompress after flush: %v\n"+
          "%q\n"+
          "want:\n%q",
          err, decompressed, input)
      }
      if err := e.Close(); err != nil {
        t.Errorf("Close(): %v", err)
      }
    }
    

    results in:

        brotli_test.go:241: Expected <32766, nil>, but <32765, <nil>>
        brotli_test.go:244: Decompress after flush: <nil>
        ...
    

    Is it expected that large Writes() are supported? Is the unit test somehow using the API incorrectly to ensure all data is both written and flushed?

    opened by hochhaus 2
  • Compression performance deteriorates for large files

    Compression performance deteriorates for large files

    Hi Andy,

    I understand this is a golang port, but raising this here anyway. I am observing the following benchmark results from encoding and decoding a random character sequence. As the []byte size grows to 32MB the execution time of the encoder slows down to a whole second. This is many orders of magnitude slower than similar algorithms in go and negatively affected by input size.

    I was wondering if you can confirm this and/or have any input on how to improve performance?

    Results

    BenchmarkBrotliEncode128B-16              144399              7416 ns/op
    BenchmarkBrotliDecode128B-16              398236              3124 ns/op
    BenchmarkBrotliEncode1KB-16                74959             15411 ns/op
    BenchmarkBrotliDecode1KB-16               354914              4664 ns/op
    BenchmarkBrotliEncode64KB-16                2673            415396 ns/op
    BenchmarkBrotliDecode64KB-16              143876             23852 ns/op
    BenchmarkBrotliEncode128KB-16               1393            795543 ns/op
    BenchmarkBrotliDecode128KB-16              46455             28896 ns/op
    BenchmarkBrotliEncode1MB-16                  181           5975186 ns/op 5ms
    BenchmarkBrotliDecode1MB-16                14785             75049 ns/op
    BenchmarkBrotliEncode2MB-16                   98          11964768 ns/op 11ms
    BenchmarkBrotliDecode2MB-16                 2452            849694 ns/op
    BenchmarkBrotliEncode4MB-16                   45          25300261 ns/op
    BenchmarkBrotliDecode4MB-16                 5822           1166313 ns/op
    BenchmarkBrotliEncode8MB-16                   19          60138005 ns/op
    BenchmarkBrotliDecode8MB-16                 3024            720402 ns/op
    BenchmarkBrotliEncode16MB-16                   5         205008482 ns/op
    BenchmarkBrotliDecode16MB-16                 104          14655937 ns/op
    BenchmarkBrotliEncode32MB-16                   1        1146873803 ns/op 1s!!!
    BenchmarkBrotliDecode32MB-16                  10         103324444 ns/op
    

    Tests

    Source code: https://github.com/simonmittag/j8a/blob/166/brotli_test.go

    opened by simonmittag 2
  • Tag new release v1.1.0 with latest optimizations

    Tag new release v1.1.0 with latest optimizations

    @andybalholm the diff since the last release seems significant with a bunch of improvements and optimizations. Would you considering creating a v1.1.0 release?

    • https://github.com/andybalholm/brotli/compare/v1.0.0...master
    opened by jaimem88 2
  • Can you please do a simple encode and decode instructions for usage of this brotli?

    Can you please do a simple encode and decode instructions for usage of this brotli?

    Can you please do a simple encode and decode instructions for usage of this brotli?

    I've only found how to encode it. How do I decode it? https://gist.github.com/miguelmota/a0a3f18d2c6caee52508f1c393927ec6

    Pls provide the instructions on your front page of github. thx

    opened by gitmko0 2
  • Status of this library

    Status of this library

    Hi, this looks neat! What is the status of this library? I'd like to add this to archiver and possibly to Caddy next chance I get, if it's ready to use.

    (Edit: disregard the license question, duh, for some reason I didn't see it on first pass)

    opened by mholt 2
  • Tag v1.0.4

    Tag v1.0.4

    There's a bug fix provided in ec682fbe0a16a840ca30e24f9aa8f62c65515f64 that affects brotli.Reader.Reset. Would you be willing to release v1.0.4 so that a tagged release has this change? Thanks!

    opened by dsnet 1
  • Shared dictionary support?

    Shared dictionary support?

    In theory google/brotli has support for shared dictionaries. In practice I've not seen any example anywhere.

    Does this package support shared dictionaries? Could you maybe provide an example?

    opened by espoal 4
  • bytewise difference between this package and the CLI

    bytewise difference between this package and the CLI

    Hi,

    I created a bunch of compressed files with the help of your package (version 1.0.4). I can decompress them with your library, however I am failing to decompress them with the CLI (neither with version 1.0.7, nor with 1.0.9). Interestingly, your library is able to decompress the brotli files which were created with the CLI.

    The brotli files that are produced by both your library and CLI have the same file size, however they differ in content(when I do a byte-wise comparison).

    This is how I compress my files with you library:

    bw := brotli.NewWriterLevel(&buf, brotli.BestCompression)
    

    Using the CLI:

    $ brotli foo.html -o foo.html.br
    

    Decompressing them with your library:

    br := brotli.NewReader(os.Stdin)
    _, err := io.Copy(os.Stdout, br)
    

    Decompressing with the CLI:

    cat foo.html.br | brotli --decompress
    

    Am I doing something wrong here?

    opened by laszbalo 1
  • feat: upgrade Go version

    feat: upgrade Go version

    Reason: 1.12 was released more than 3 years ago, and since then, many changes/fixes were applied.

    The project may benefit from these changes if we consider facts like security and performance.

    Test execution using Go 1.18 - click to expand

    Obs: tests running in a Docker image (golang:1.18):

    docker run -ti --rm -v $(pwd):/code -w /code golang:1.18 sh
    # go test -v --cover ./...
    === RUN   TestEncoderNoWrite
    --- PASS: TestEncoderNoWrite (0.00s)
    === RUN   TestEncoderEmptyWrite
    --- PASS: TestEncoderEmptyWrite (0.00s)
    === RUN   TestWriter
    --- PASS: TestWriter (0.02s)
    === RUN   TestIssue22
    --- PASS: TestIssue22 (6.89s)
    === RUN   TestEncoderStreams
    --- PASS: TestEncoderStreams (0.13s)
    === RUN   TestEncoderLargeInput
    --- PASS: TestEncoderLargeInput (2.46s)
    === RUN   TestEncoderFlush
    --- PASS: TestEncoderFlush (0.00s)
    === RUN   TestDecoderStreaming
    === RUN   TestDecoderStreaming/Segment0
    === RUN   TestDecoderStreaming/Segment1
    === RUN   TestDecoderStreaming/Segment2
    --- PASS: TestDecoderStreaming (0.00s)
        --- PASS: TestDecoderStreaming/Segment0 (0.00s)
        --- PASS: TestDecoderStreaming/Segment1 (0.00s)
        --- PASS: TestDecoderStreaming/Segment2 (0.00s)
    === RUN   TestReader
    --- PASS: TestReader (0.00s)
    === RUN   TestDecode
    --- PASS: TestDecode (0.00s)
    === RUN   TestQuality
    --- PASS: TestQuality (0.08s)
    === RUN   TestDecodeFuzz
    --- PASS: TestDecodeFuzz (0.00s)
    === RUN   TestDecodeTrailingData
    --- PASS: TestDecodeTrailingData (0.00s)
    === RUN   TestEncodeDecode
        brotli_test.go:441: case "" x 0
        brotli_test.go:441: case "A" x 1
        brotli_test.go:441: case "<html><body><H1>Hello world</H1></body></html>" x 10
        brotli_test.go:441: case "<html><body><H1>Hello world</H1></body></html>" x 1000
    --- PASS: TestEncodeDecode (0.00s)
    === RUN   ExampleWriter_Reset
    --- PASS: ExampleWriter_Reset (0.00s)
    PASS
    coverage: 86.8% of statements
    ok      github.com/andybalholm/brotli   9.587s  coverage: 86.8% of statements
    # go version
    go version go1.18.2 linux/amd64
    
    opened by devdrops 1
  • CVE-2020-8927 vulnerability

    CVE-2020-8927 vulnerability

    In google/brotli latest release (v1.0.9), they are mentioning that they fixed the vulnerability issue: https://nvd.nist.gov/vuln/detail/CVE-2020-8927: SECURITY: decoder: fix integer overflow when input chunk is larger than 2GiB (CVE-2020-8927)

    I am wondering is it mitigated here as well? in the latest release v1.0.4?

    Thanks!

    opened by elsappp 1
  • Process: integration into google/brotli

    Process: integration into google/brotli

    Hello.

    I wonder, if this work could be integrated into google/brotli.

    As I understand, this repo is generated by c2go project. But:

    • it is unclear, if it required ad-hoc configuration (mentioned in c2go readme)
    • also it generated code license might be a tricky question; I'll investigate soon

    But the first things go first - I'm asking for authors blessing to do that.

    opened by eustas 2
Owner
Andy Balholm
Andy Balholm
concurrent caching proxy and decoder library for collections of PMTiles

go-pmtiles A caching proxy for the serverless PMTiles archive format. Resolves several of the limitations of PMTiles by running a minimalistic, single

Protomaps 36 Nov 14, 2022
Go decoder for EU Digital COVID Certificate (EUDCC) QR code data

Go Corona QR Code Decoder This repository contains a decoder for EU Digital COVID Certificate (EUDCC) QR code data, written in Go. If you got vaccinat

Michael Stapelberg 172 Nov 9, 2022
Distributed RTC System by pure Go and Flutter

ION is a distributed real-time communication system, the goal is to chat anydevice, anytime, anywhere! Distributed Real-time Communication System ION

Pion 3.6k Nov 20, 2022
🚀 gnet is a high-performance, lightweight, non-blocking, event-driven networking framework written in pure Go./ gnet 是一个高性能、轻量级、非阻塞的事件驱动 Go 网络框架。

English | ???? 中文 ?? Introduction gnet is an event-driven networking framework that is fast and lightweight. It makes direct epoll and kqueue syscalls

Andy Pan 7.1k Nov 21, 2022
Pure-Go library for cross-platform local peer discovery using UDP multicast :woman: :repeat: :woman:

peerdiscovery Pure-go library for cross-platform thread-safe local peer discovery using UDP multicast. I needed to use peer discovery for croc and eve

Zack 568 Nov 26, 2022
A QUIC implementation in pure go

A QUIC implementation in pure Go quic-go is an implementation of the QUIC protocol in Go. It implements the IETF QUIC draft-29 and draft-32. Version c

Lucas Clemente 7.5k Nov 27, 2022
Pure Go implementation of the WebRTC API

Pion WebRTC A pure Go implementation of the WebRTC API New Release Pion WebRTC v3.0.0 has been released! See the release notes to learn about new feat

Pion 10.3k Nov 19, 2022
A Windows named pipe implementation written in pure Go.

npipe Package npipe provides a pure Go wrapper around Windows named pipes. Windows named pipe documentation: http://msdn.microsoft.com/en-us/library/w

Nate Finch 254 Nov 16, 2022
Pure Go implementation of the WebRTC API

Pure Go implementation of the WebRTC API

Pion 10.3k Nov 26, 2022
A pure Unix shell script implementing ACME client protocol

An ACME Shell script: acme.sh An ACME protocol client written purely in Shell (Unix shell) language. Full ACME protocol implementation. Support ACME v

acme.sh 29.3k Nov 21, 2022
Pure-Go HBase client

Golang HBase client This is a pure Go client for HBase. Current status: beta. Supported Versions HBase >= 1.0 Installation go get github.com/tsuna/goh

Benoit Sigoure 659 Nov 18, 2022
Snugger is a light weight but fast network recon scanner that is written from pure golang

Snugger is a light weight but fast network recon scanner that is written from pure golang. with this scann you can ARP your network, port scan hosts and host lists, as well as scan for BSSId

RE43P3R 2 May 19, 2022
A golang library about socks5, supports all socks5 commands. That Provides server and client and easy to use. Compatible with socks4 and socks4a.

socks5 This is a Golang implementation of the Socks5 protocol library. To see in this SOCKS Protocol Version 5. This library is also compatible with S

chenhao zhang 40 Nov 22, 2022
Transparent TLS and HTTP proxy serve and operate on all 65535 ports, with domain regex whitelist and rest api control

goshkan Transparent TLS and HTTP proxy serve & operating on all 65535 ports, with domain regex whitelist and rest api control tls and http on same por

Sina Ghaderi 11 Nov 5, 2022
Tapestry is an underlying distributed object location and retrieval system (DOLR) which can be used to store and locate objects. This distributed system provides an interface for storing and retrieving key-value pairs.

Tapestry This project implements Tapestry, an underlying distributed object location and retrieval system (DOLR) which can be used to store and locate

Han Cai 1 Mar 16, 2022
A Go package for sending and receiving ethernet frames. Currently supporting Linux, Freebsd, and OS X.

ether ether is a go package for sending and receiving ethernet frames. Currently supported platform: BPF based OS X FreeBSD AF_PACKET based Linux Docu

Song Gao 78 Sep 27, 2022
Package ethernet implements marshaling and unmarshaling of IEEE 802.3 Ethernet II frames and IEEE 802.1Q VLAN tags. MIT Licensed.

ethernet Package ethernet implements marshaling and unmarshaling of IEEE 802.3 Ethernet II frames and IEEE 802.1Q VLAN tags. MIT Licensed. For more in

Matt Layher 246 Nov 9, 2022
A Stable & Secure Tunnel based on KCP with N:M multiplexing and FEC. Available for ARM, MIPS, 386 and AMD64。KCPプロトコルに基づく安全なトンネル。KCP 프로토콜을 기반으로 하는 보안 터널입니다。

Disclaimer: kcptun maintains a single website — github.com/xtaci/kcptun. Any websites other than github.com/xtaci/kcptun are not endorsed by xtaci. Re

xtaci 13.1k Nov 20, 2022
httpstream provides HTTP handlers for simultaneous streaming uploads and downloads of objects, as well as persistence and a standalone server.

httpfstream httpfstream provides HTTP handlers for simultaneous streaming uploads and downloads of files, as well as persistence and a standalone serv

Sourcegraph 16 May 1, 2021