Pure Go Brotli encoder and decoder

pelase check the attched zip js file: marked.js.zip 1.0.0 is ok!

/usr/local/Cellar/go/1.16/libexec/src/runtime/panic.go:971 +0x499
github.com/andybalholm/brotli.compressFragmentFastImpl(0xc0002c1380, 0x31, 0x40, 0x31, 0x0, 0xc0002f8a68, 0x400, 0x400, 0x9, 0xc0002faa88, ...)
    /pkg/mod/github.com/andybalholm/[email protected]/compress_fragment.go:382 +0x23fb
github.com/andybalholm/brotli.compressFragmentFast(0xc0002c1380, 0x31, 0x40, 0x31, 0xc0002f8a00, 0xc0002f8a68, 0x400, 0x400, 0x200, 0xc0002faa88, ...)
    /pkg/mod/github.com/andybalholm/[email protected]/compress_fragment.go:673 +0x418
github.com/andybalholm/brotli.encoderCompressStreamFast(0xc0002f8800, 0x0, 0xc000816448, 0xc000816458, 0x16c4980)
    /pkg/mod/github.com/andybalholm/[email protected]/encode.go:971 +0x314
github.com/andybalholm/brotli.encoderCompressStream(0xc0002f8800, 0x0, 0xc000816448, 0xc000816458, 0x10a5680)
    /pkg/mod/github.com/andybalholm/[email protected]/encode.go:1101 +0x369
github.com/andybalholm/brotli.(*Writer).writeChunk(0xc0002f8800, 0xc0002c1380, 0x31, 0x40, 0x0, 0x1669a20, 0x0, 0x385f0b8)
    /pkg/mod/github.com/andybalholm/[email protected]/writer.go:77 +0xba
github.com/andybalholm/brotli.(*Writer).Write(0xc0002f8800, 0xc0002c1380, 0x31, 0x40, 0xc0002f8800, 0x0, 0x3860b18)
    /pkg/mod/github.com/andybalholm/[email protected]/writer.go:111 +0x52
github.com/ije/rex.(*responseWriter).Write(0xc0001de400, 0xc0002c1380, 0x31, 0x40, 0x163e6e0, 0x16bce00, 0x1c00000000000001)

brotli.ensureRingBuffer is allocating more than 17 gb of memory and it significantly impact the performance.

   Total:     17.86GB    17.86GB (flat, cum) 68.44%
   1307            .          .           	var old_ringbuffer []byte = s.ringbuffer 
   1308            .          .           	if s.ringbuffer_size == s.new_ringbuffer_size { 
   1309            .          .           		return true 
   1310            .          .           	} 
   1311            .          .            
   1312      17.86GB    17.86GB           	s.ringbuffer = make([]byte, uint(s.new_ringbuffer_size)+uint(kRingBufferWriteAheadSlack)) 
   1313            .          .           	if s.ringbuffer == nil { 
   1314            .          .           		/* Restore previous value. */ 
   1315            .          .           		s.ringbuffer = old_ringbuffer 
   1316            .          .            
   1317            .          .           		return false 

I suggest reusing buffer with sync.Pool.

Issue description

On static analysis of the binary, generated after compiling our go fiber app we found out a lot of text and links in the final binary, which turned out to be test data in the package.

Please check this issue:

https://github.com/gofiber/fiber/issues/1782

If you wouldn't mind looking over this, I've removed some redundant assignments for when the writer is initialised (they're set to 0 and false when the struct is created and it doesn't appear to be in a sync.Pool).

There may also be the possibility of reusing the same quality settings instead of overwriting them on each reset?

I want to use brotli for stream compression.

I have a use case when i need to interrupt ( stop ) the stream compression for one week before continuing. For this I need to save the dictionary created already. I want to get a serialized state of the dynamic dictionary once the compression is stopped. So that, after, I will be able to de serialize the state and continue the process of compression using the dictionary created.

Is this feature/api available within this library ?

If no, I want to add it to this library, so from where should I start ? Can you give me some hints ?

Are you interested in this use case to add it to the library ?

Thanks for this library! It is great to have a pure go brotli encoder.

I am seeing the Flush() calls not completely process all data following a call to a large Write(). Modifying your unit test TestEncoderFlush to use an input size of 32766 instead of 1000 demonstrates the behavior.

func TestEncoderFlush(t *testing.T) {
  input := make([]byte, 32766) // MODIFIED
  rand.Read(input)
  out := bytes.Buffer{}
  e := NewWriterOptions(&out, WriterOptions{Quality: 5})
  in := bytes.NewReader(input)
  _, err := io.Copy(e, in)
  if err != nil {
    t.Fatalf("Copy Error: %v", err)
  }
  if err := e.Flush(); err != nil {
    t.Fatalf("Flush(): %v", err)
  }
  if out.Len() == 0 {
    t.Fatalf("0 bytes written after Flush()")
  }
  decompressed := make([]byte, 32766) // MODIFIED
  reader := NewReader(bytes.NewReader(out.Bytes()))
  n, err := reader.Read(decompressed)
  if n != len(decompressed) || err != nil {
    t.Errorf("Expected <%v, nil>, but <%v, %v>", len(decompressed), n, err)
  }
  if !bytes.Equal(decompressed, input) {
    t.Errorf(""+
      "Decompress after flush: %v\n"+
      "%q\n"+
      "want:\n%q",
      err, decompressed, input)
  }
  if err := e.Close(); err != nil {
    t.Errorf("Close(): %v", err)
  }
}

results in:

    brotli_test.go:241: Expected <32766, nil>, but <32765, <nil>>
    brotli_test.go:244: Decompress after flush: <nil>
    ...

Is it expected that large Writes() are supported? Is the unit test somehow using the API incorrectly to ensure all data is both written and flushed?

Hi Andy,

I understand this is a golang port, but raising this here anyway. I am observing the following benchmark results from encoding and decoding a random character sequence. As the []byte size grows to 32MB the execution time of the encoder slows down to a whole second. This is many orders of magnitude slower than similar algorithms in go and negatively affected by input size.

I was wondering if you can confirm this and/or have any input on how to improve performance?

Results

BenchmarkBrotliEncode128B-16              144399              7416 ns/op
BenchmarkBrotliDecode128B-16              398236              3124 ns/op
BenchmarkBrotliEncode1KB-16                74959             15411 ns/op
BenchmarkBrotliDecode1KB-16               354914              4664 ns/op
BenchmarkBrotliEncode64KB-16                2673            415396 ns/op
BenchmarkBrotliDecode64KB-16              143876             23852 ns/op
BenchmarkBrotliEncode128KB-16               1393            795543 ns/op
BenchmarkBrotliDecode128KB-16              46455             28896 ns/op
BenchmarkBrotliEncode1MB-16                  181           5975186 ns/op 5ms
BenchmarkBrotliDecode1MB-16                14785             75049 ns/op
BenchmarkBrotliEncode2MB-16                   98          11964768 ns/op 11ms
BenchmarkBrotliDecode2MB-16                 2452            849694 ns/op
BenchmarkBrotliEncode4MB-16                   45          25300261 ns/op
BenchmarkBrotliDecode4MB-16                 5822           1166313 ns/op
BenchmarkBrotliEncode8MB-16                   19          60138005 ns/op
BenchmarkBrotliDecode8MB-16                 3024            720402 ns/op
BenchmarkBrotliEncode16MB-16                   5         205008482 ns/op
BenchmarkBrotliDecode16MB-16                 104          14655937 ns/op
BenchmarkBrotliEncode32MB-16                   1        1146873803 ns/op 1s!!!
BenchmarkBrotliDecode32MB-16                  10         103324444 ns/op

Tests

Source code: https://github.com/simonmittag/j8a/blob/166/brotli_test.go

@andybalholm the diff since the last release seems significant with a bunch of improvements and optimizations. Would you considering creating a v1.1.0 release?

• https://github.com/andybalholm/brotli/compare/v1.0.0...master

Can you please do a simple encode and decode instructions for usage of this brotli?

I've only found how to encode it. How do I decode it? https://gist.github.com/miguelmota/a0a3f18d2c6caee52508f1c393927ec6

Pls provide the instructions on your front page of github. thx

Hi, this looks neat! What is the status of this library? I'd like to add this to archiver and possibly to Caddy next chance I get, if it's ready to use.

(Edit: disregard the license question, duh, for some reason I didn't see it on first pass)

There's a bug fix provided in ec682fbe0a16a840ca30e24f9aa8f62c65515f64 that affects brotli.Reader.Reset. Would you be willing to release v1.0.4 so that a tagged release has this change? Thanks!

In theory google/brotli has support for shared dictionaries. In practice I've not seen any example anywhere.

Does this package support shared dictionaries? Could you maybe provide an example?

Hi,

I created a bunch of compressed files with the help of your package (version 1.0.4). I can decompress them with your library, however I am failing to decompress them with the CLI (neither with version 1.0.7, nor with 1.0.9). Interestingly, your library is able to decompress the brotli files which were created with the CLI.

The brotli files that are produced by both your library and CLI have the same file size, however they differ in content(when I do a byte-wise comparison).

This is how I compress my files with you library:

bw := brotli.NewWriterLevel(&buf, brotli.BestCompression)

Using the CLI:

$ brotli foo.html -o foo.html.br

Decompressing them with your library:

br := brotli.NewReader(os.Stdin)
_, err := io.Copy(os.Stdout, br)

Decompressing with the CLI:

cat foo.html.br | brotli --decompress

Am I doing something wrong here?

Reason: 1.12 was released more than 3 years ago, and since then, many changes/fixes were applied.

The project may benefit from these changes if we consider facts like security and performance.

docker run -ti --rm -v $(pwd):/code -w /code golang:1.18 sh
# go test -v --cover ./...
=== RUN   TestEncoderNoWrite
--- PASS: TestEncoderNoWrite (0.00s)
=== RUN   TestEncoderEmptyWrite
--- PASS: TestEncoderEmptyWrite (0.00s)
=== RUN   TestWriter
--- PASS: TestWriter (0.02s)
=== RUN   TestIssue22
--- PASS: TestIssue22 (6.89s)
=== RUN   TestEncoderStreams
--- PASS: TestEncoderStreams (0.13s)
=== RUN   TestEncoderLargeInput
--- PASS: TestEncoderLargeInput (2.46s)
=== RUN   TestEncoderFlush
--- PASS: TestEncoderFlush (0.00s)
=== RUN   TestDecoderStreaming
=== RUN   TestDecoderStreaming/Segment0
=== RUN   TestDecoderStreaming/Segment1
=== RUN   TestDecoderStreaming/Segment2
--- PASS: TestDecoderStreaming (0.00s)
    --- PASS: TestDecoderStreaming/Segment0 (0.00s)
    --- PASS: TestDecoderStreaming/Segment1 (0.00s)
    --- PASS: TestDecoderStreaming/Segment2 (0.00s)
=== RUN   TestReader
--- PASS: TestReader (0.00s)
=== RUN   TestDecode
--- PASS: TestDecode (0.00s)
=== RUN   TestQuality
--- PASS: TestQuality (0.08s)
=== RUN   TestDecodeFuzz
--- PASS: TestDecodeFuzz (0.00s)
=== RUN   TestDecodeTrailingData
--- PASS: TestDecodeTrailingData (0.00s)
=== RUN   TestEncodeDecode
    brotli_test.go:441: case "" x 0
    brotli_test.go:441: case "A" x 1
    brotli_test.go:441: case "<html><body><H1>Hello world</H1></body></html>" x 10
    brotli_test.go:441: case "<html><body><H1>Hello world</H1></body></html>" x 1000
--- PASS: TestEncodeDecode (0.00s)
=== RUN   ExampleWriter_Reset
--- PASS: ExampleWriter_Reset (0.00s)
PASS
coverage: 86.8% of statements
ok      github.com/andybalholm/brotli   9.587s  coverage: 86.8% of statements
# go version
go version go1.18.2 linux/amd64

In google/brotli latest release (v1.0.9), they are mentioning that they fixed the vulnerability issue: https://nvd.nist.gov/vuln/detail/CVE-2020-8927: SECURITY: decoder: fix integer overflow when input chunk is larger than 2GiB (CVE-2020-8927)

I am wondering is it mitigated here as well? in the latest release v1.0.4?

Thanks!

Hello.

I wonder, if this work could be integrated into google/brotli.

As I understand, this repo is generated by c2go project. But:

• it is unclear, if it required ad-hoc configuration (mentioned in c2go readme)
• also it generated code license might be a tricky question; I'll investigate soon

But the first things go first - I'm asking for authors blessing to do that.