Native LZO implementation in Go

Related tags

Compression go-lzo
Overview

go-lzo

Build status Coverage Status

Native LZO1X implementation in Golang

This code has been written using the original LZO1X source code as a reference, to study and understand the algorithms. Both the LZO1X-1 and LZO1X-999 algorithms are implemented. These are the most popular of the whole LZO suite of algorithms.

Being a straightforward port of the original source code, it shares the same license (GPLv2) as I can't possibly claim any copyright on it.

I plan to eventually reimplement LZO1X-1 from scratch. At that point, I will be also changing license.

Benchmarks

These are the benchmarks obtained running the testsuite over the Canterbury corpus for the available compressor levels:

Compressor Level Original Compressed Factor Time Speed
LZO1X-1 - 18521760 8957481 51.6% 0.16s 109MiB/s
LZO1X-999 1 18521760 8217347 55.6% 1.38s 13MiB/s
LZO1X-999 2 18521760 7724879 58.3% 1.50s 12MiB/s
LZO1X-999 3 18521760 7384377 60.1% 1.68s 10MiB/s
LZO1X-999 4 18521760 7266674 60.8% 1.69s 10MiB/s
LZO1X-999 5 18521760 6979879 62.3% 2.75s 6.4MiB/s
LZO1X-999 6 18521760 6938593 62.5% 4.53s 3.9MiB/s
LZO1X-999 7 18521760 6905362 62.7% 6.94s 2.5MiB/s
LZO1X-999 8 18521760 6713477 63.8% 20.96s 863KiB/s
LZO1X-999 9 18521760 6712069 63.8% 22.82s 792KiB/s
Comments
  • Update Decompress1X handling of runtime error for Go1.13.

    Update Decompress1X handling of runtime error for Go1.13.

    Upcoming Go 1.13 change https://golang.org/cl/161477 updates the runtime error strings for index out of range errors.

    This change simply makes the comparison a bit more loose to handle both pre-1.13 and 1.13.

    opened by cybrcodr 3
  • Go1.13 golang.org/cl/161477 will break Decompress1X runtime error handling

    Go1.13 golang.org/cl/161477 will break Decompress1X runtime error handling

    Upcoming Go 1.13 change https://golang.org/cl/161477 updates the runtime error strings for index out of range errors, which breaks the assumption that Decompress1X func has on it at https://github.com/rasky/go-lzo/blob/master/decompress.go#L160.

    $ go version
    $ go version devel +431b5c69ca Tue Apr 16 21:54:01 2019 +0000 darwin/amd64
    $ go test
    --- FAIL: TestDecompInlen (0.00s)
    panic: runtime error: index out of range [0] with length 0 [recovered]
            panic: runtime error: index out of range [0] with length 0 [recovered]
            panic: runtime error: index out of range [0] with length 0
    
    goroutine 20 [running]:
    testing.tRunner.func1(0xc0000e8100)
            /Users/herbie/work/go113/go/src/testing/testing.go:830 +0x3a3
    panic(0x115b440, 0xc00010a020)
            /Users/herbie/work/go113/go/src/runtime/panic.go:619 +0x1b2
    github.com/rasky/go-lzo.Decompress1X.func1(0xc000106e70)
            /Users/herbie/work/google3-go/lzo/decompress.go:165 +0xfa
    panic(0x115b440, 0xc00010a020)
            /Users/herbie/work/go113/go/src/runtime/panic.go:619 +0x1b2
    github.com/rasky/go-lzo.(*reader).ReadU16(...)
            /Users/herbie/work/google3-go/lzo/decompress.go:94
    github.com/rasky/go-lzo.Decompress1X(0x1199c40, 0xc0000c2040, 0x31, 0x0, 0xc0000d6090, 0xf, 0x10, 0x0, 0x0)
            /Users/herbie/work/google3-go/lzo/decompress.go:234 +0x944
    github.com/rasky/go-lzo.TestDecompInlen(0xc0000e8100)
            /Users/herbie/work/google3-go/lzo/compress_test.go:126 +0x20c
    testing.tRunner(0xc0000e8100, 0x1177800)
            /Users/herbie/work/go113/go/src/testing/testing.go:865 +0xbf
    created by testing.(*T).Run
            /Users/herbie/work/go113/go/src/testing/testing.go:916 +0x350
    exit status 2
    FAIL    github.com/rasky/go-lzo 0.024s
    

    I'll send a PR to update the check.

    opened by cybrcodr 3
  • Proposal to dual license the project

    Proposal to dual license the project

    Can this project be dual licensed? perhaps in Apache 2.0 or MIT such that Apache 2.0 projects can link against this library?

    Current LICENSE.gpl is too restrictive and incompatible with Apache 2.0 released projects.

    opened by harshavardhana 2
  • Can't decompress LZO compressed in Python

    Can't decompress LZO compressed in Python

    Hi Giovanni,

    I have a Python application that compresses data using python-lzo and then decompresses when needed in another part of application.

    I'm currently porting decompressing part of the application to golang and for some reason when I'm trying to use your library it always throws LookBehindUnderrun error at me. Here is the code I'm using:

    import (
        "bytes"
        "fmt"
    
        "github.com/rasky/go-lzo"
    )
    
    outData, err = lzo.Decompress1X(bytes.NewReader(inData), 0, 0)
    if err != nil {
        fmt.Println(err)
    }
    

    Any ideas ?

    Thanks in advance.

    opened by maxtepkeev 1
  • Possible infinite loop with single byte input

    Possible infinite loop with single byte input

    Hello,

    I just ran go-fuzz on the decompressor and it found this possibly triggering an infinite loop:

    https://github.com/pmezard/go-lzo/blob/fuzz-it/crashers/5ba93c9db0cff93f52b521d7420e43f6eda2784f

    with the fuzzer here:

    https://github.com/pmezard/go-lzo/blob/fuzz-it/fuzz.go

    (I committed the corpus/ to the branch as well if you are interested).

    I do not know if you implemented it just as an experiment or if you care about such reports.

    opened by pmezard 1
  • compression of files

    compression of files

    Hi, I am using go language. And i want to compress files which are retrieved by ls command. I want to use rasky/go-lzo package. please guide me how can i do this?

    opened by poojakarale 0
  • Add compress/decompress tests for lzop(1) compatibility

    Add compress/decompress tests for lzop(1) compatibility

    Make existing decompress tests a bit more idiomatic by having them check returned error values.

    Add two tests which compare uncompressed and compressed output from lzop(1).

    Currently, we can now uncompress data compressed with lzop -1.

    Make debug prints conditioned on a variable (Verbose), not commented out, and expose both Verbose and the print function (Debug) so users of this package can set them. Debug defaults to an empty function.

    For the loops where we do not want to continually call an empty function, Verbose is used to control whether we call Debug at all.

    Still kind of a WIP, comments appreciated. I actually have a need for this package in u-root.

    Signed-off-by: Ronald G Minnich [email protected]

    opened by rminnich 1
Owner
Giovanni Bajo
Developer Relation @italia / @teamdigitale. CTO of @develersrl and @greenapes; Python, Go, C++. Interested in algorithms & security.
Giovanni Bajo
A simple zip compactor app written in golang to help you life. Usage with native GUI and CLI.

Usage Install go install github.com/gustavonobreza/zip-compactor Run in GUI (Can select many files) zip-compactor Run in CLI (Can select just one file

Gustavo Alexandre Nobre Mesquita 1 Nov 12, 2021
Parallel implementation of Gzip for modern multi-core machines written in Go

gzip Parallel implementation of gzip for modern multi-core machines written in Go Usage: gzip [OPTION]... [FILE] Compress or uncompress FILE (by defau

Pedro Albanese 0 Nov 16, 2021
A simple zip implementation in Go

A simple zip implementation in Go

Phil Eaton 4 Sep 19, 2022
Go wrapper for LZO compression library

This is a cgo wrapper around the LZO real-time compression library. LZO is available at http://www.oberhumer.com/opensource/lzo/ lzo.go is the go pack

Damian Gryski 13 Mar 4, 2022
An open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developersAn open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developers

Developer-oriented Continuous Delivery Product ⁣ English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use? How to

null 0 Oct 19, 2021
Go implementation of systemd Journal's native API for logging

journald Package journald offers Go implementation of systemd Journal's native API for logging. Key features are: based on a connection-less socket wo

Grigory Zubankov 31 Sep 2, 2022
A native Go clean room implementation of the Porter Stemming algorithm.

Go Porter Stemmer A native Go clean room implementation of the Porter Stemming Algorithm. This algorithm is of interest to people doing Machine Learni

Charles Iliya Krempeaux 179 Oct 15, 2021
Full-native go implementation of Telegram API

MTProto Full-native implementation of MTProto protocol on Golang! english русский 简体中文 Features Full native implementation All code, from sending requ

Xelaj technologies 920 Sep 22, 2022
Constant Database native golang implementation

CDB golang implementation cdb is a fast, reliable, simple package for creating and reading constant databases see docs for more details Advantages Ite

Alexander Petrov 32 Jul 15, 2022
go implementation of fissions web-native file system

wnfs-go go language implementation of the fission web-native file system, using the typescript implementation as a reference. Development Status: Work

Qri.io 13 Mar 28, 2022
Package rsync contains a native Go rsync implementation.

gokrazy rsync Package rsync contains a native Go rsync implementation. ⚠ Beware: very fresh. Might eat your data. You have been warned! ⚠ The only com

null 297 Sep 14, 2022
Go language implementation of a blockchain based on the BDLS BFT protocol. The implementation was adapted from Ethereum and Sperax implementation

BDLS protocol based PoS Blockchain Most functionalities of this client is similar to the Ethereum golang implementation. If you do not find your quest

Yongge Wang 0 Jan 1, 2022
Sequence-based Go-native audio mixer for music apps

Mix https://github.com/go-mix/mix Sequence-based Go-native audio mixer for music apps See demo/demo.go: package main import ( "fmt" "os" "time"

null 162 Sep 15, 2022
A "native" ogg vorbis decoder for Go (uses inline stb_vorbis)

vorbis This Go package provides a "native" ogg vorbis decoder, but still requires cgo, as it uses inline code from stb_vorbis. Someday, it won't. The

Steve McCoy 30 May 27, 2022
An opinionated configuration loading framework for Containerized and Cloud-Native applications.

Opinionated configuration loading framework for Containerized and 12-Factor compliant applications. Read configurations from Environment Variables, an

Sherif Abdel-Naby 81 Sep 25, 2022
Drone is a Container-Native, Continuous Delivery Platform

Drone is a Continuous Delivery system built on container technology. Drone uses a simple YAML configuration file, a superset of docker-compose, to def

drone.io 25.8k Sep 22, 2022
Go native library for fast point tracking and K-Nearest queries

Geo Index Geo Index library Overview Splits the earth surface in a grid. At each cell we can store data, such as list of points, count of points, etc.

Hailo Network IP Ltd 339 Sep 15, 2022
CockroachDB - the open source, cloud-native distributed SQL database.

CockroachDB is a cloud-native SQL database for building global, scalable cloud services that survive disasters. What is CockroachDB? Docs Quickstart C

CockroachDB 25.6k Sep 17, 2022
Native GraphQL Database with graph backend

The Only Native GraphQL Database With A Graph Backend. Dgraph is a horizontally scalable and distributed GraphQL database with a graph backend. It pro

Dgraph 18.5k Sep 18, 2022
Micro is a platform for cloud native development

Micro Overview Micro addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provid

Micro 11.3k Sep 23, 2022