Include files in your binary the easy way

Overview

New Projects should use the official embed package instead, which was added in go 1.16.

binclude

PkgGoDev Test Go Report Card codecov

binclude is a tool for including static files into Go binaries.

  • focuses on ease of use
  • the bincluded files add no more than the filesize to the binary
  • uses go/ast for typesafe parsing
  • each package can have its own binclude.FileSystem
  • binclude.FileSystem implements the http.FileSystem interface
  • ioutil like functions FileSystem.ReadFile, FileSystem.ReadDir
  • include all files/ directories under a given path by calling binclude.Include("./path")
  • include files based on a glob pattern binclude.IncludeGlob("./path/*.txt")
  • add file paths from a textfile binclude.IncludeFromFile("includefile.txt")
  • high test coverage
  • supports execution of executables directly from a binclude.FileSystem via binexec (os/exec wrapper)
  • optional compression of files with gzip binclude -gzip
  • debug mode to read files from disk binclude.Debug = true

Install

GO111MODULE=on go get -u github.com/lu4p/binclude/cmd/binclude

Usage

package main

//go:generate binclude

import (
	"io/ioutil"
	"log"
	"path/filepath"

	"github.com/lu4p/binclude"
)

var assetPath = binclude.Include("./assets") // include ./assets with all files and subdirectories

func main() {
	binclude.Include("file.txt") // include file.txt

	// like os.Open
	f, err := BinFS.Open("file.txt")
	if err != nil {
		log.Fatalln(err)
	}

	out, err := ioutil.ReadAll(f)
	if err != nil {
		log.Fatalln(err)
	}

	log.Println(string(out))

	// like ioutil.Readfile
	content, err := BinFS.ReadFile(filepath.Join(assetPath, "asset1.txt"))
	if err != nil {
		log.Fatalln(err)
	}

	log.Println(string(content))

	// like ioutil.ReadDir
	infos, err := BinFS.ReadDir(assetPath)
	if err != nil {
		log.Fatalln(err)
	}

	for _, info := range infos {
		log.Println(info.Name())
	}
}

To build use:

go generate
go build

A more detailed example can be found here.

Binary size

The resulting binary, with the included files can get quite large.

You can reduce the final binary size by building without debug info (go build -ldflags "-s -w") and compressing the resulting binary with upx (upx binname).

Note: If you don't need to access the compressed form of the files I would advise to just use upx and don't add seperate compression to the files.

You can add compression to the included files with -gzip

Note: decompression is optional to allow for the scenario where you want to serve compressed files for a webapp directly.

OS / Arch Specific Includes

binclude supports including files/binaries only on specific architectures and operating systems. binclude follows the same pattern as Go's implicit Build Constraints. It will generate files for the specific platforms like binclude_windows.go which contains all windows specific files.

If a file's name, matches any of the following patterns:

*_GOOS.go
*_GOARCH.go
*_GOOS_GOARCH.go

binclude will consider all files included by binclude.Include in this file as files which should only be included on a specific GOOS and/ or GOARCH.

For example, if you want to include a binary only on Windows you could have a file static_windows.go and reference the static file:

package main

import "github.com/lu4p/binclude"

func bar() {
  binclude.Include("./windows-file.dll")
}

OS / Arch Specific Includes are used in the binexec example.

Note: explicit build tags like // +build debug are not supported for including files conditionally.

Advanced Usage

The generator can also be included into your package to allow for the code generator to run after all module dependencies are installed. Without installing the binclude generator to the PATH seperatly.

main.go:

// +build !gen

//go:generate go run -tags=gen .
package main

import (
	"github.com/lu4p/binclude"
)

func main() {
	binclude.Include("./assets")
}

main_gen.go:

// +build gen

package main

import (
	"github.com/lu4p/binclude"
	"github.com/lu4p/binclude/bincludegen"
)

func main() {
	bincludegen.Generate(binclude.None, ".")
	// binclude.None == no compression 
	// binclude.Gzip == gzip compression
}

To build use:

go generate
go build

Contributing

If you find a feature missing, which you consider a useful addition please open an issue.

If you plan on implementing a new feature please open an issue so we can discuss it first (I don't want you to waste your time).

Pull requests are welcome, and will be timely reviewed and merged, please provide tests for your code, if you don't know how to test your code open a pull request and I will suggest the right testing method.

Comments
  • os/exec wrapper caching

    os/exec wrapper caching

    The os/exec wrapper is a really useful feature. As I understand the feature it copies the embedded byte slice to a temporary file in the os.CacheUserDir(), executes it, and then removes it. This is a sane default, but if the embedded binary is large (in my case, 80mb), this will take a long time to copy and use each time. I wonder if there should be an option to disable the removal, and a potential warning logged when calling os/exec on a large binary, or if this is a feature that people should just implement on their own (extracting the embedded file to wherever and calling os/exec on it). I can really see this -not- being a feature that is added, but if it is not, I wonder if there should be some documentation or warning that this could be expensive for large binaries because it copies them each time.

    opened by audiolion 10
  • Does this works for binary files

    Does this works for binary files

    I am trying for binary files, but I am always getting error

    
    binclude.Include("./out/MyExe")
    
    err := example.BinFS.CopyFile("MyExe", filepath.Join(destination, "MyExe"))
    		if err != nil {
    			fmt.Printf("1 :: Failed to copy file to destination, error = %v\n", err)
    			//os.Exit(1)
    		}
    		err = example.BinFS.CopyFile("out/MyExe, filepath.Join(destination, "MyExe"))
    		if err != nil {
    			fmt.Printf("2 :: Failed to copy file to destination, error = %v\n", err)
    			//os.Exit(1)
    		}
    		err = example.BinFS.CopyFile("./out/MyExe", filepath.Join(destination, "MyExe"))
    		if err != nil {
    			fmt.Printf("3 ::Failed to copy file to destination, error = %v\n", err)
    			//os.Exit(1)
    		}
    
    opened by davinash 9
  • Evaluate binclude.Include based on build tags

    Evaluate binclude.Include based on build tags

    Is it possible to use binclude to selectively include files based on build tags? For instance, if I am building for windows amd64, include X file?

    //+build windows
    
    import "github.com/lu4p/binclude"
    
    binclude.Include("/windows-thing.dll")
    
    opened by audiolion 6
  • Remove support for Brotli

    Remove support for Brotli

    This is something I've never done before, an anti-feature request :)

    You recently added optional GZip and Brotli support. They cover the same use case, with Brotli offering slightly better filesizes. Since Brotli isn't supported by Go by default, you had to pull in the github.com/andybalholm/brotli package. This package is anything but small:

    ❯ gocloc brotli
    -------------------------------------------------------------------------------
    Language                     files          blank        comment           code
    -------------------------------------------------------------------------------
    Go                              61           2451           1941         252967
    Plain Text                       1            815              0           8471
    Markdown                         1              1              0              4
    -------------------------------------------------------------------------------
    TOTAL                           63           3267           1941         261442
    -------------------------------------------------------------------------------
    

    Is an optional feature worth adding 260k lines of code? I'd argue no. Furthermore, aletheia-icu/broccoli already covers the "embed files with brotli compression" use case, for those interested in it (and not afraid of the large dependency).

    opened by bojanz 5
  • Binclude generated code doesn't always match go fmt

    Binclude generated code doesn't always match go fmt

    When I run binclude, some of the time the generated code does not match go fmt so I have to run go fmt against the file after generation. Specifically there is extra space between the variable declarations and the = sign that needs to be removed:

    Current binclude generated output

    image

    After I run go fmt

    image

    opened by audiolion 4
  • Allow direct `go generate` usage (no CLI)

    Allow direct `go generate` usage (no CLI)

    Here's an official feature request, repeating what I said on Reddit:

     I would appreciate being able to run the generator from my own _gen.go file, instead of having to rely on the binclude binary (which requires ensuring I have installed it up front, and I guess maintaining a tools.go file for that purpose).
    

    This would require moving the relevant code from cmd/bininclude to the parent package, so that it can be called both by random callers and the CLI.

    Right now only vfsgen in the entire ecosystem supports this, so it would be a big plus for this package to cover it.

    opened by bojanz 3
  • Include expects a constant string literal

    Include expects a constant string literal

    I may be using the API incorrectly. binclude.Include always expects string literal. binclude.Include(fmt.Sprintf("%s", "./out/myexe")) Does not work and give error

    [binclude] 2020/06/26 06:09:47 argument is not string literal

    Is there a way to avoid this, I am looking for reading the file name from say input json

    opened by davinash 3
  • Update module rogpeppe/go-internal to v1.7.0

    Update module rogpeppe/go-internal to v1.7.0

    WhiteSource Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | github.com/rogpeppe/go-internal | require | minor | v1.6.2 -> v1.7.0 |


    Release Notes

    rogpeppe/go-internal

    v1.7.0

    Compare Source


    Renovate configuration

    :date: Schedule: At any time (no schedule defined).

    :vertical_traffic_light: Automerge: Enabled.

    :recycle: Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    :no_bell: Ignore: Close this PR and you won't be reminded about this update again.


    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by WhiteSource Renovate. View repository job log here.

    opened by renovate[bot] 1
  • Update module rogpeppe/go-internal to v1.6.1

    Update module rogpeppe/go-internal to v1.6.1

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | github.com/rogpeppe/go-internal | require | patch | v1.6.0 -> v1.6.1 |


    Release Notes

    rogpeppe/go-internal

    v1.6.1

    Compare Source

    • testscript: relax constraints around actual source in cmp with update (#​107)

    Renovate configuration

    :date: Schedule: At any time (no schedule defined).

    :vertical_traffic_light: Automerge: Enabled.

    :recycle: Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    :no_bell: Ignore: Close this PR and you won't be reminded about this update again.


    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by WhiteSource Renovate. View repository job log here.

    opened by renovate[bot] 1
  • Cmd.Run() in binexec.go doesn't remove executable as documented

    Cmd.Run() in binexec.go doesn't remove executable as documented

    // Run is similar to (*Cmd).Run() in the os/exec package,
    // but deletes the executable at Cmd.Path
    func (c *Cmd) Run() error {
    	return c.OsCmd.Run()
    }
    
    opened by the-goodies 1
  • Generates \ slashes instead of /

    Generates \ slashes instead of /

    I'm using windows 10 and the generated BinFS map has \ in it's keys as path seperator. Generated error with value at that point \r: .\binclude.go:54:54: unknown escape sequence.

    opened by the-goodies 1
  • Action Required: Fix Renovate Configuration

    Action Required: Fix Renovate Configuration

    There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

    Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

    opened by renovate[bot] 0
Owner
null
A quick and easy way to setup a RESTful JSON API

Go-Json-Rest A quick and easy way to setup a RESTful JSON API Go-Json-Rest is a thin layer on top of net/http that helps building RESTful JSON APIs ea

Antoine Imbert 3.5k Jan 3, 2023
A simple blog framework built with GO. Uses HTML files and a JSON dict to give you more control over your content.

Go-Blog A simple template based blog framework. Instructions Built for GO version: 1 See the Documentation or Getting Started pages in the wiki. Notes

Matt West 45 Sep 10, 2022
This package provides a generic way of deep copying Go objects

Package for copying Go values This package provides a generic way of deep copying Go objects. It is designed with performance in mind and is suitable

Google 42 Nov 9, 2022
An efficient, extensible and easy-to-use RPC framework.

eRPC eRPC is an efficient, extensible and easy-to-use RPC framework. Suitable for RPC, Microservice, Peer-to-Peer, IM, Game and other fields. 简体中文 Ins

henrylee2cn 2.4k Dec 29, 2022
Simple and easy go web micro framework

DotWeb Simple and easy go web micro framework Important: Now need go1.9+ version support, and support go mod. Document: https://www.kancloud.cn/devfee

devfeel 1.4k Dec 22, 2022
lazy-go is an easy-to-use WEB framework.

lazy-go lazy-go is an easy-to-use WEB framework. Installation Run the following command under your project: go get -u github.com/NICEXAI/lazy-go Quick

Afeyer 3 Mar 25, 2022
Easy-to-use web fuzzer, written in Go.

Brutal A lightweight, simple to use web fuzzer. Usage Brutal is pretty easy to use. Command Description --debug print more details about the runtime -

Andrew 4 Jul 8, 2022
Impersonate Your GitHub App In A GitHub Action

Impersonate Your GitHub App In A GitHub Action This action helps you retrieve an authenticated app token with a GitHub app id and a app private key. Y

null 1 Nov 29, 2021
Start and finish your Go apps gracefully, even in the case of panics

Relax Relax - verb antonym for panic. In the context of Go programs, relax means to make critical failures less severe so that graceful shutdown is ne

Serge Radinovich 7 May 7, 2023
A quick and easy password protected web server for your files. httpfolder makes downloading/uploading files from your current working directory easy, even for fairly large files.

httpfolder A quick and easy password protected web server for your files. httpfolder makes downloading/uploading files from your current working direc

null 14 Sep 12, 2022
Cfginterpolator is an interpolate library in golang allowing to include data from external sources in your configuration

cfginterpolator cfginterpolator is an interpolate library in golang allowing to include data from external sources in your configuration cfginterpolat

Benoit Bayszczak 0 Dec 14, 2021
Write your SQL queries in raw files with all benefits of modern IDEs, use them in an easy way inside your application with all the profit of compile time constants

About qry is a general purpose library for storing your raw database queries in .sql files with all benefits of modern IDEs, instead of strings and co

Sergey Treinis 26 Dec 25, 2022
upload to gdrive include shared drive

gdrive_uploader upload to gdrive include shared drive How to get go get github.com/heat1024/gdrive_uploader About credential You must set your client

ryu 3 Oct 6, 2022
Tool to modify Snapmaker laser GCode for a rotary module to include z movements

lasery2z Tool to modify Snapmaker laser GCode for a rotary module to include z movements. Background Snapmaker's Luban tool to generate 4D laser proje

null 3 Sep 13, 2021
Gitlab include draftsman in golang

Gitlab include draftsman WIP Утилита, которая рисует graph include'ов из gitlab-ci Как работает пока: Считывает из app.env файла(который должен быть в

Roman Mironov 3 Oct 31, 2021
Simple tool that updates Visual Studio Code workspace(s) to include Go modules in gopath/src, then launches VSCode if only one modified.

Simple tool that updates Visual Studio Code workspace(s) to include Go modules in gopath/src, then launches VSCode if only one modified.

Valérie 0 Jan 27, 2022
Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

Patrick D'appollonio 204 Dec 29, 2022
Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

kubectl-slice: split Kubernetes YAMLs into files kubectl-slice is a neat tool that allows you to split a single multi-YAML Kubernetes manifest into mu

Patrick D'appollonio 205 Jan 3, 2023
searchHIBP is a golang tool that implements binary search over a hash ordered binary file.

searchHIBP is a golang tool that implements binary search over a hash ordered binary file.

fblz 0 Nov 9, 2021