Using brotli compression to embed static files in Go.

Overview

🥦 Broccoli

go get -u aletheia.icu/broccoli

GoDoc Travis Go Report Card codecov.io

Broccoli uses brotli compression to embed a virtual file system of static files inside Go executables.

A few reasons to pick broccoli over the alternatives:

  • ⚡️ The average is 13-25% smaller binary size due to use of superior compression algorithm, brotli.
  • 💾 Broccoli supports bundling of multiple source directories, only relies on go generate command-line interface and doesn't require configuration files.
  • 🔑 Optional decompression is something you may want; when it's enabled, files are decompressed only when they are read the first time.
  • 🚙 You might want to target wasm/js architecture.
  • 📰 There is -gitignore option to ignore files, already ignored by your existing .gitignore files.

Performance

Admittedly, there are already many packages providing similar functionality out there in the wild. Tim Shannon did an overall pretty good overview of them in Choosing A Library to Embed Static Assets in Go, but it should be outdated by at least two years, so although we subscribe to the analysis, we cannot guarantee whether if it's up–to–date. Most if not all of the packages mentioned in the article, rely on gzip compression and most of them, unfortunately are not compatible with wasm/js architecture, due to some quirk that has to do with their use of http package. This, among other things, was the driving force behind the creation of broccoli.

The most feature-complete library from the comparison table seems to be fileb0x.

How does broccoli compare to flexb0x?

Feature fileb0x broccoli
compression gzip brotli (-20% avg.)
optional decompression yes yes
compression levels yes yes (1-11)
different build tags for each file yes no
exclude / ignore files glob glob
unexported vars/funcs optional optional
virtual memory file system yes yes
http file system yes yes
replace text in files yes no
glob support yes yes
regex support no no
config file yes no
update files remotely yes no
.gitignore support no yes

How does it compare to others?

Broccoli seems to outperform the existing solutions.

We did benchmarks, please feel free to review them and correct us whenever our methodology could be flawed.

Usage

$ broccoli
Usage: broccoli [options]

Broccoli uses brotli compression to embed a virtual file system in Go executables.

Options:
	-src folder[,file,file2]
		The input files and directories, "public" by default.
	-o
		Name of the generated file, follows input by default.
	-var=br
		Name of the exposed variable, "br" by default.
	-include *.html,*.css
		Wildcard for the files to include, no default.
	-exclude *.wasm
		Wildcard for the files to exclude, no default.
	-opt
		Optional decompression: if enabled, files will only be decompressed
		on the first time they are read.
	-gitignore
		Enables .gitignore rules parsing in each directory, disabled by default.
	-quality [level]
		Brotli compression level (1-11), the highest by default.

Generate a broccoli.gen.go file with the variable broccoli:
	//go:generate broccoli -src assets -o broccoli -var broccoli

Generate a regular public.gen.go file, but include all *.wasm files:
	//go:generate broccoli -src public -include="*.wasm"

How broccoli is used in the user code:

//go:generate broccoli -src=public,others -o assets

func init() {
    br.Walk("public", func(path string, info os.FileInfo, err error) error {
        // walk...
        return nil
    })
}

func main() {
    http.ListenAndServe(":8080", br.Serve("public"))
}

Credits

License: MIT

We would like to thank brotli development team from Google and Andy Balholm, for his c2go pure-Go port of the library. Broccoli itself is an effort of a mentoring experiment, lead by @tucnak on the foundation of Aletheia.

Issues
  • Inflexibility WRT Go source and VFS data locations

    Inflexibility WRT Go source and VFS data locations

    Currently, broccoli must be called from a Go source directory (I call it manually via go run aletheia.icu/broccoli and don't use Go's generation feature). If this is not the case, it errors out with broccoli: cannot parse package: no buildable Go source files in ..

    Now if the data I want to have broccoli slurp resides somewhere completely else (in my case ../../../../frontend/dist; making this absolute wouldn't help either), the path ends up in the VFS, which will rarely be desirable. Because of the requirement to run broccoli from a source directory, I cannot opt to, say, run it from the dist directory in the example and have it write a Go file to a completely different directory.

    There should be an option to map the given source directories to a (common) prefix that replaces any path implied by the physical location. For example, I'd like to make the contents of the distant directory referenced above available at the root of the VFS. If not that, an option to have it strip n levels of directories would help in my case as well.

    opened by curio77 11
  • File size seems bigger than packr

    File size seems bigger than packr

    Hello,

    We chatted a little on Reddit. I wanted to use what you had so far and see how much smaller is the output.

    I have a project called Dozzle. I compared the output with what I currently have with packr and the binary is actually bigger.

    See below:

    Screen Shot 2020-04-16 at 10 07 48 AM

    I have pushed my branch to broccoli. Not sure what I am doing wrong. You can run yarn build to get a static folder that is the source.

    opened by amir20 5
  • Go mod dependency issue

    Go mod dependency issue

    In the course of trying this (with Go v1.14.2), I encountered the following error running go mod tidy after adding broccoli as a dependency (via a side-effect-only tool import, import _ "aletheia.icu/broccoli"):

    go: finding module for package aletheia.icu/broccoli
    go: downloading aletheia.icu/broccoli v1.0.0-beta
    go: found aletheia.icu/broccoli in aletheia.icu/broccoli v1.0.0-beta
    go: aletheia.icu/[email protected] requires
            aletheia.icu/broccoli/[email protected]: invalid version: unknown revision 000000000000
    
    opened by curio77 4
  • 404 on downloads

    404 on downloads

    Seems when trying to run a go generate utilising this library that the following error is returned:

    go: aletheia.icu/broccoli/[email protected]: unrecognized import path "aletheia.icu/broccoli/fs": reading https://aletheia.icu/broccoli/fs?go-get=1: 404 Not Found

    I can see that https://aletheia.icu is still functional but seems the path for the library isn't working as intended.

    opened by nightah 1
  • Issues importing the package

    Issues importing the package

    Something happened to your hosting that the go package can't be retrieved any more.

    $ go get aletheia.io/[email protected]
    go get aletheia.io/[email protected]: unrecognized import path "aletheia.io/broccoli": https fetch: Get "https://aletheia.io/broccoli?go-get=1": x509: certificate is valid for blog.inkanex.com, not aletheia.io
    
    $ go get aletheia.io/[email protected]: unrecognized import path "aletheia.io/broccoli": reading https://aletheia.io/broccoli?go-get=1: 401 Unauthorized```
    opened by mariusor 1
  • Concerns about dependency size

    Concerns about dependency size

    This package uses https://github.com/andybalholm/brotli to handle the actual compression. This package has 260 thousand lines of code:

    ❯ gocloc brotli
    -------------------------------------------------------------------------------
    Language                     files          blank        comment           code
    -------------------------------------------------------------------------------
    Go                              61           2451           1941         252967
    Plain Text                       1            815              0           8471
    Markdown                         1              1              0              4
    -------------------------------------------------------------------------------
    TOTAL                           63           3267           1941         261442
    -------------------------------------------------------------------------------
    

    Almost none of that are tests, so all of that code is getting compiled in. In my testing adding this package increases my binary size by about 5.5mb. In comparison, github.com/klauspost/compress/zstd is 10 thousand lines of code with tests, and only adds about 700kb to the binary.

    I'm afraid that the increase in binary size due to https://github.com/andybalholm/brotli offsets the gains made by better compression.

    opened by bojanz 0
  • Allow direct `go generate` usage (no CLI)

    Allow direct `go generate` usage (no CLI)

    Right now I need to modify my Makefile to install broccoli in $PATH to be able to use it from go generate. It would be simpler if I could instead invoke my own gen.go which just calls the broccoli API, like vfsgen allows. From what I can see, main.go is already a very thin wrapper, all that's needed is to export Generator options and parsePackage() / generate() functions. Is that something you'd be open to? If yes, I could send in a PR.

    opened by bojanz 0
  • Consider moving generated file comment to satisfy golint

    Consider moving generated file comment to satisfy golint

    Because the current comment is put at the top it causes golint to complain:

    domains.gen.go:1:1: package comment should be of the form "Package swot ..."
    
    // Code generated by broccoli at 2020-04-20T23:26:30+07:00.
    package swot
    
    import "aletheia.icu/broccoli/fs"
    
    var br = fs.New("...")
    
    Screen Shot 2020-04-20 at 22 57 35

    I suggest moving it to above the variable:

    package swot
    
    import "aletheia.icu/broccoli/fs"
    
    // Code generated by broccoli at 2020-04-20T23:26:30+07:00.
    var br = fs.New("...")
    
    opened by montanaflynn 3
  • Output flag not working as expected

    Output flag not working as expected

    The help shows -o flag allows you to name the generated file.

    	-o
    		Name of the generated file, follows input by default.
    

    If I use broccoli -src domains -o domains I get a file called:

    domains.gen.go

    But when I use broccoli -src domains -o domains.go I still get a file called:

    domains.gen.go

    opened by montanaflynn 0
Owner
Aletheia
Aletheia
a better customizable tool to embed files in go; also update embedded files remotely without restarting the server

fileb0x What is fileb0x? A better customizable tool to embed files in go. It is an alternative to go-bindata that have better features and organized c

null 613 Jul 21, 2022
Generates go code to embed resource files into your library or executable

Deprecating Notice go is now going to officially support embedding files. The go command will support //go:embed tags. Go Embed Generates go code to e

Peter 6.3k Jun 2, 2021
Embed files into a Go executable

statik statik allows you to embed a directory of static files into your Go binary to be later served from an http.FileSystem. Is this a crazy idea? No

Jaana Dogan 3.5k Aug 7, 2022
A tool to be used with 'go generate' to embed external template files into Go code.

templify A tool to be used with 'go generate' to embed external template files into Go code. Scenario An often used scenario in developing go applicat

Michael Wolber 27 Jan 24, 2022
:file_folder: Embeds static resources into go files for single binary compilation + works with http.FileSystem + symlinks

Package statics Package statics embeds static files into your go applications. It provides helper methods and objects to retrieve embeded files and se

Go Playgound 64 Mar 3, 2022
Get an embed.FS from inside an embed.FS

embed.FS wrapper providing additional functionality Features Get an embed.FS from an embedded subdirectory Handy Copy(sourcePath, targetPath) method t

Lea Anthony 20 Aug 6, 2022
The simple and easy way to embed static files into Go binaries.

NOTICE: Please consider migrating your projects to github.com/markbates/pkger. It has an idiomatic API, minimal dependencies, a stronger test suite (t

Buffalo - The Go Web Eco-System 3.4k Aug 4, 2022
Embed static files in Go binaries (replacement for gobuffalo/packr)

Pkger github.com/markbates/pkger is a tool for embedding static files into Go binaries. It will, hopefully, be a replacement for github.com/gobuffalo/

Mark Bates 1.2k Jul 27, 2022
Pure Go Brotli encoder and decoder

This package is a brotli compressor and decompressor implemented in Go. It was translated from the reference implementation (https://github.com/google

Andy Balholm 415 Aug 4, 2022
Zibr - Repack ZIPs into Brotli

zibr zipr is a command-line utility that repacks a compressed ZIP or PNG file in

Dmitry Chestnykh 5 Jul 22, 2022
a better customizable tool to embed files in go; also update embedded files remotely without restarting the server

fileb0x What is fileb0x? A better customizable tool to embed files in go. It is an alternative to go-bindata that have better features and organized c

null 613 Jul 21, 2022
Generates go code to embed resource files into your library or executable

Deprecating Notice go is now going to officially support embedding files. The go command will support //go:embed tags. Go Embed Generates go code to e

Peter 6.3k Jun 2, 2021
Embed files into a Go executable

statik statik allows you to embed a directory of static files into your Go binary to be later served from an http.FileSystem. Is this a crazy idea? No

Jaana Dogan 3.5k Aug 7, 2022
A tool to be used with 'go generate' to embed external template files into Go code.

templify A tool to be used with 'go generate' to embed external template files into Go code. Scenario An often used scenario in developing go applicat

Michael Wolber 27 Jan 24, 2022
Embed files into a Go executable

statik statik allows you to embed a directory of static files into your Go binary to be later served from an http.FileSystem. Is this a crazy idea? No

Jaana Dogan 3.5k Aug 9, 2022
Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

Patrick D'appollonio 177 Aug 10, 2022
Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

kubectl-slice: split Kubernetes YAMLs into files kubectl-slice is a neat tool that allows you to split a single multi-YAML Kubernetes manifest into mu

Patrick D'appollonio 175 Aug 5, 2022
Simple image compression using SVD

SVD image compression An implementation image compression using SVD decomposition on Go Built With Go 1.17 Gonum Compression examples Header Image Ori

null 4 Mar 30, 2022
Go package to embed the Mozilla Included CA Certificate List

rootcerts Package rootcerts provides an embedded copy of the Mozilla Included CA Certificate List, more specifically the PEM of Root Certificates in M

Lucas Bremgartner 84 Jul 24, 2022
Recreate embedded filesystems from embed.FS type in current working directory.

rebed Recreate embedded filesystems from embed.FS type in current working directory. Expose the files you've embedded in your binary so users can see

Patricio Whittingslow 22 Feb 28, 2022
go:embed and the golang-standards project layout

An example of using the golang-standards project layout and the go:embed directive.

Leon Rowland 6 May 17, 2022
Embed arbitrary resources into a go executable at runtime, after the executable has been built.

ember Ember is a lightweight library and tool for embedding arbitrary resources into a go executable at runtime. The resources don't need to exist at

null 55 Aug 4, 2022
This command line converts .webarchive file to resources embed .html file

webarchive-to-singlefile This command line converts Safari's .webarchive file to complete .html. Notice Only tested on MacOS. Google Chrome required.

会有猫的 3 Aug 10, 2022
Ghostinthepdf - This is a small tool that helps to embed a PostScript file into a PDF

This is a small tool that helps to embed a PostScript file into a PDF in a way that GhostScript will run the PostScript code during the

Emil Lerner 123 Jul 5, 2022
Little golang app that allows you to download a youtube video as mp3, and optionally embed ID3 tags -Cover Art, Artist ...-

yt2mp3 Little golang app that allows you to download a youtube video as mp3, and optionally embed ID3 tags -Cover Art, Artist ...- Instructions At the

null 0 Dec 25, 2021
Use this program to embed sh scripts in binaries

sh2bin Use this program to embed sh scripts in binaries. Feel free to fork this

Carlos Polop 17 Jul 21, 2022
PoC de embed de arquivos arbitrários em imagens e vídeos

imgcode PoC de embed de arquivos arbitrários em imagens e vídeos Esse projeto de Abril de 2020 eu achei perdido no meu HD externo hoje (Janeiro de 202

Lucas Eduardo 0 Jan 8, 2022
This command line converts .html file into .html with images embed.

embed-html This command line converts .html file into .html with images embed. Install > go get github.com/gonejack/embed-html Usage > embed-html *.ht

会有猫的 0 Jan 13, 2022
Optimized compression packages

compress This package provides various compression algorithms. zstandard compression and decompression in pure Go. S2 is a high performance replacemen

Klaus Post 3.1k Aug 11, 2022