Takes an input http.FileSystem (likely at go generate time) and generates Go code that statically implements it.

Overview

vfsgen

Build Status GoDoc

Package vfsgen takes an http.FileSystem (likely at go generate time) and generates Go code that statically implements the provided http.FileSystem.

Features:

  • Efficient generated code without unneccessary overhead.

  • Uses gzip compression internally (selectively, only for files that compress well).

  • Enables direct access to internal gzip compressed bytes via an optional interface.

  • Outputs gofmted Go code.

Installation

go get -u github.com/shurcooL/vfsgen

Usage

Package vfsgen is a Go code generator library. It has a Generate function that takes an input filesystem (as a http.FileSystem type), and generates a Go code file that statically implements the contents of the input filesystem.

For example, we can use http.Dir as a http.FileSystem implementation that uses the contents of the /path/to/assets directory:

var fs http.FileSystem = http.Dir("/path/to/assets")

Now, when you execute the following code:

err := vfsgen.Generate(fs, vfsgen.Options{})
if err != nil {
	log.Fatalln(err)
}

An assets_vfsdata.go file will be generated in the current directory:

// Code generated by vfsgen; DO NOT EDIT.

package main

import ...

// assets statically implements the virtual filesystem provided to vfsgen.Generate.
var assets http.FileSystem = ...

Then, in your program, you can use assets as any other http.FileSystem, for example:

file, err := assets.Open("/some/file.txt")
if err != nil {
	return err
}
defer file.Close()
http.Handle("/assets/", http.FileServer(assets))

vfsgen can be more useful when combined with build tags and go generate directives. This is described below.

go generate Usage

vfsgen is great to use with go generate directives. The code invoking vfsgen.Generate can go in an assets_generate.go file, which can then be invoked via "//go:generate go run assets_generate.go". The input virtual filesystem can read directly from disk, or it can be more involved.

By using build tags, you can create a development mode where assets are loaded directly from disk via http.Dir, but then statically implemented for final releases.

For example, suppose your source filesystem is defined in a package with import path "example.com/project/data" as:

// +build dev

package data

import "net/http"

// Assets contains project assets.
var Assets http.FileSystem = http.Dir("assets")

When built with the "dev" build tag, accessing data.Assets will read from disk directly via http.Dir.

A generate helper file assets_generate.go can be invoked via "//go:generate go run -tags=dev assets_generate.go" directive:

// +build ignore

package main

import (
	"log"

	"example.com/project/data"
	"github.com/shurcooL/vfsgen"
)

func main() {
	err := vfsgen.Generate(data.Assets, vfsgen.Options{
		PackageName:  "data",
		BuildTags:    "!dev",
		VariableName: "Assets",
	})
	if err != nil {
		log.Fatalln(err)
	}
}

Note that "dev" build tag is used to access the source filesystem, and the output file will contain "!dev" build tag. That way, the statically implemented version will be used during normal builds and go get, when custom builds tags are not specified.

vfsgendev Usage

vfsgendev is a binary that can be used to replace the need for the assets_generate.go file.

Make sure it's installed and available in your PATH.

go get -u github.com/shurcooL/vfsgen/cmd/vfsgendev

Then the "//go:generate go run -tags=dev assets_generate.go" directive can be replaced with:

//go:generate vfsgendev -source="example.com/project/data".Assets

vfsgendev accesses the source variable using "dev" build tag, and generates an output file with "!dev" build tag.

Additional Embedded Information

All compressed files implement httpgzip.GzipByter interface for efficient direct access to the internal compressed bytes:

// GzipByter is implemented by compressed files for
// efficient direct access to the internal compressed bytes.
type GzipByter interface {
	// GzipBytes returns gzip compressed contents of the file.
	GzipBytes() []byte
}

Files that have been determined to not be worth gzip compressing (their compressed size is larger than original) implement httpgzip.NotWorthGzipCompressing interface:

// NotWorthGzipCompressing is implemented by files that were determined
// not to be worth gzip compressing (the file size did not decrease as a result).
type NotWorthGzipCompressing interface {
	// NotWorthGzipCompressing is a noop. It's implemented in order to indicate
	// the file is not worth gzip compressing.
	NotWorthGzipCompressing()
}

Comparison

vfsgen aims to be conceptually simple to use. The http.FileSystem abstraction is central to vfsgen. It's used as both input for code generation, and as output in the generated code.

That enables great flexibility through orthogonality, since helpers and wrappers can operate on http.FileSystem without knowing about vfsgen. If you want, you can perform pre-processing, minifying assets, merging folders, filtering out files and otherwise modifying input via generic http.FileSystem middleware.

It avoids unneccessary overhead by merging what was previously done with two distinct packages into a single package.

It strives to be the best in its class in terms of code quality and efficiency of generated code. However, if your use goals are different, there are other similar packages that may fit your needs better.

Alternatives

  • go-bindata - Reads from disk, generates Go code that provides access to data via a custom API.
  • go-bindata-assetfs - Takes output of go-bindata and provides a wrapper that implements http.FileSystem interface (the same as what vfsgen outputs directly).
  • becky - Embeds assets as string literals in Go source.
  • statik - Embeds a directory of static files to be accessed via http.FileSystem interface (sounds very similar to vfsgen); implementation sourced from camlistore.
  • go.rice - Makes working with resources such as HTML, JS, CSS, images and templates very easy.
  • esc - Embeds files into Go programs and provides http.FileSystem interfaces to them.
  • staticfiles - Allows you to embed a directory of files into your Go binary.
  • togo - Generates a Go source file with a []byte var containing the given file's contents.
  • fileb0x - Simple customizable tool to embed files in Go.
  • embedfiles - Simple tool for embedding files in Go code as a map.
  • packr - Simple solution for bundling static assets inside of Go binaries.
  • rsrc - Tool for embedding .ico & manifest resources in Go programs for Windows.

Attribution

This package was originally based on the excellent work by @jteeuwen on go-bindata and @elazarl on go-bindata-assetfs.

License

Issues
  • Filter on which files to save as assets

    Filter on which files to save as assets

    Is there a way to filter which files within a directory will get save as assets?

    question documentation 
    opened by ghostsquad 12
  • Add a command-line version of vfsgen

    Add a command-line version of vfsgen

    • Inclusive of the change at https://github.com/shurcooL/vfsgen/pull/13 (should merge #13 before this one).
    • Also depends on the change at https://github.com/shurcooL/httpfs/pull/1
    • Fixes #2.

    See docs in main.go for usage examples / general idea of how it works.

    opened by slimsag 9
  • Add ForceAllTypes to Options

    Add ForceAllTypes to Options

    Adds a ForceAllTypes member to Options, forcing the generation of both the vfsgen۰CompressedFileInfo and vfsgen۰FileInfo types. Useful for other using packages that need to ensure that both types are present.

    thinking 
    opened by kenshaw 8
  • README: Add Alternatives and Comparison sections.

    README: Add Alternatives and Comparison sections.

    I've gotten this idea from someone on Twitter (I couldn't find the exact tweet, sadly). It went something like this:

    From now on, all my Go library READMEs will include an Alternatives section!

    I think it's a great idea for the following reasons:

    • This is good for users because they can either be more confident that vfsgen is the best choice for them, or if their needs are different, find a more well-suited package.
    • It's good for vfsgen because it communicates to users more clearly that it is indeed similar to some other existing packages.
    • It's good for alternative packages listed because they get free advertisement and exposure.
    • It's good for everyone because it can help reduce duplication of effort if something can be consolidated, and improved code quality and functionality due to competition. :)

    A great example of 2 packages that already do this is:

    • https://github.com/dchest/htmlmin#alternatives - A simple HTML minifier.
    • https://github.com/tdewolff/minify#alternatives - A more complete, heavyweight solution for minifying HTML, CSS, JS, JSON, SVG, XML.

    I am including some alternative/similar Go libraries that I am aware of. Most of them differ in scope or preferred API for input/output to various degrees.

    /cc @jteeuwen @elazarl @tv42 @rakyll @GeertJohan I mentioned your packages in the alternatives section, please let me know if you have a problem with that and would want me to edit it or remove your package, I will gladly do that.

    Motivation to create vfsgen

    I originally made vfsgen after using go-bindata+go-bindata-assetfs for a while but they suffered from https://github.com/jteeuwen/go-bindata/issues/22, https://github.com/elazarl/go-bindata-assetfs/issues/24 and I saw no way to resolve those issues easily without changing the interface significantly (see my proposal for fork here).

    I also wanted to have the freedom to experiment without being tied to a stable API and see what the best solution in this space is. And I wanted to simplify the code a lot, because I saw a lot of opportunities for that in the original codebases.

    I'm very happy with where vfsgen ended up and it enabled me to do things I couldn't otherwise (e.g., see https://github.com/shurcooL/Go-Package-Store/pull/18#issuecomment-120696770). It's useful to me, and I hope it'll be useful to more people who have similar needs. Together with the alternative packages, this problem space in Go has great coverage now.

    opened by dmitshur 6
  • vfsgendev: look up source in the current module

    vfsgendev: look up source in the current module

    Currently vfsgendev only considers GOROOT and GOPATH when looking up the source to use. This does not work as expected when running vfsgendev within a go module, as introduced in Go 1.11.

    Specifying the current directory (.) as the source directory for build.Import(), allows the module logic to work when running inside a go module.

    opened by antong 6
  • Add ./cmd/vfsgendev binary to address common use case.

    Add ./cmd/vfsgendev binary to address common use case.

    Add an opinionated binary for specialized use. It uses the convention of using "dev" build tag for development mode, and assumes that the source VFS is defined in a "dev" only environment; it uses "!dev" build tag in output _vfsdata.go file.

    Usage is simple:

    $ vfsgendev -source="import/path".VariableName
    

    Or in a go:generate directive:

    //go:generate vfsgendev -source="import/path".VariableName
    

    See sourcegraph/appdash#91 for additional background, motivation, and discussion.

    Resolves #2.

    Closes #14.

    /cc @slimsag

    opened by dmitshur 6
  • Make the VariableName option default to being exported.

    Make the VariableName option default to being exported.

    This change makes the VariableName default to the exported Assets instead of the previously unexported assets. The rational for this is that people creating Go packages will not have to specify a VariableName parameter at all because it will default to being exported. For creators of main packages, they cannot be imported anyway so keeping the field private by default is less valuable.

    opened by slimsag 6
  • Use more readable internal naming scheme.

    Use more readable internal naming scheme.

    The original motivation for using the ugly and non-idiomatic _vfsgen_ prefix for all generated internal types was to ensure no possibility of having a name collision with the user's own code. For example, if user had a type FileInfo, and vfsgen tried to generate FileInfo as well, that'd be a compile time error.

    By starting all identifiers with an underscore – something no manually written idiomatic Go code would do – we ensured no collisions.

    The idea here is to use a more readable alternative way of achieving the same goal. Since we're using a hard-to-type unicode character, it's unlikely people would write that by hand, so name collisions are still unlikely.

    That little dot ۰ is an Arabic zero numeral (U+06F0), categories [Nd].

    This idea is inspired by internal code of golang.org/x/tools/go/ssa/interp package. See https://github.com/golang/tools/blob/86ad1193da6aba71355e71191641b9cfb26daa67/go/ssa/interp/external.go#L36.

    opened by dmitshur 5
  • os.SEEK_{SET,CUR,END} constants are deprecated.

    os.SEEK_{SET,CUR,END} constants are deprecated.

    See https://godoc.org/os#SEEK_SET:

    Deprecated: Use io.SeekStart, io.SeekCurrent, and io.SeekEnd.

    Thanks @dominikh for spotting and reporting.

    opened by dmitshur 5
  • Adding gitignore to vfsgen

    Adding gitignore to vfsgen

    opened by tariq1890 4
  • Add UseGlobalModTime option to make application reproducible

    Add UseGlobalModTime option to make application reproducible

    This PR adds a new option named UseGlobalModTime to not store file or dir time into the generated files so that application builds could be reproducible.

    The default value of UseGlobalModTime is false. Then anything is not changed. When set UseGlobalModTime as true, a function named GlobalModTime(filename string) time.Time will be called from the generated files. So you could define that function in the same package of the generated files.

    opened by lunny 0
  • evaluate future direction after Go 1.16 changes

    evaluate future direction after Go 1.16 changes

    vfsgen was created to fill in a gap in the ability to embed files in Go programs, and uses net/http.FileSystem as the central filesystem interface.

    Go 1.16 will add //go:embed and io/fs.FS.

    This issue is to determine whether a vfsgen v2 should be made that makes use of the new Go 1.16 features, or if there isn't a need for vfsgen v2 because the Go 1.16 features on their own are sufficient for use cases that vfsgen is used for.

    What's clear is that vfsgen v0 (the current version) should be v1 and its API should not change.

    thinking 
    opened by dmitshur 0
  • Feature Request: path to generate file

    Feature Request: path to generate file

    I am writing a library that has a subpackage, which needs to have assets generated. The folder structure looks like this:

    mypackage/
      .air.toml
      cmd/
        example/
          main.go
          resources/
            generate.go
            asset.go
            templates/
              index.go.html
              login.go.html
    

    Here are my constrains:

    1. The asset.go file contains my development http.FileSystem declaration. The generate.go contains the development only main package that generates the code.
    2. I want to generate asset_vfsdata.go in the same folder as asset.go.
    3. I want to use air to live reload asset_vfsdata.go. So ideally I want to run the generate.go command at the root folder.

    My problem is this: if I run the generate.go at the root folder (mypackage), the generated go file will be placed directly at root, which fails to achieve (2). I tried adding path to vfsgen.Options.Filename, but it seems to be ignored by vfsgen.

    I think the clean to do it is to have a vfsgen.Options that specifies the relative output folder somehow.

    opened by yookoala 1
  • Symlink & File Permission Support?

    Symlink & File Permission Support?

    It appears that the default behavior is to strip the Mode and file permission bits and to read symlink targets rather than to save the path as the contents.

    I suppose this could be circumvented by first running tar and then running vfsgen, or running some other process to save the metadata and zipping it together later.

    I couldn't find any issues or docs about either symlinks or file permissions.

    Is this explicitly not a goal of the project? Or just an oversight?

    Why?

    I'm working on a project that has an init subcommand which generates project files, including a .env.

    # initializes .env and the ./scripts/ directory
    gitdeploy init
    
    # run with the default configuration
    gitdeploy run
    
    enhancement 
    opened by coolaj86 1
  • Add option to generate a function rather than global variable

    Add option to generate a function rather than global variable

    It would be great to have another field in Options struct called FunctionName, which would generate a function returning http.FileSystem, rather than a function assigned to global variable, as globals are magic and should not be used whenever possible (though a function in global is not that bad :smile:).

    opened by invidian 4
  • Generation of assets fails with GOOS=freebsd

    Generation of assets fails with GOOS=freebsd

    When generating contents prior to building for freebsd, generation fails with a segfault:

    I'm executing the generate step as a dependency of the build step, therefore the env variables are also set for the generate step. The behaviour is the same when I run GOOS=freebsd GOARCH=amd64 go generate code.vikunja.io/api/pkg/static directly.

    $ GOOS=freebsd GOARCH=amd64 make build
    go generate code.vikunja.io/api/pkg/static
    signal: segmentation fault (core dumped)
    pkg/static/static.go:17: running "go": exit status 1
    make: *** [Makefile:105: generate] Error 1
    

    The code is here.

    If I generate the assets for my own platform (linux) and run the build with all env set for freebsd, it seems to work, but only if I don't run the generate step.

    I'm not sure if this is an issue with vfsgen or some other part of the go toolchain. It is imho not critical since the workaround with running the generate step first and only then setting the env variables for freebsd when compiling the binary is an okay-ish workaround. I only happend to stuble upon this because of the way my makefile is set up.

    opened by kolaente 2
  • importPathToDir used by repos using this doesn't work in module mode

    importPathToDir used by repos using this doesn't work in module mode

    Trying the following example project that uses Go modules vfsgen-test2.zip, I get the following error:

    2020/07/23 00:40:54 cannot find module providing package github.com/segevfiner/vfsgen-test2/assets/assets: module github.com/segevfiner/vfsgen-test2/assets/assets: git ls-remote -q origin in /Users/segev/go/pkg/mod/cache/vcs/b1dce0b5aa30a69878ddaa81a87781bd321f95d17cec9ee5bf25190b026b94ea: exit status 128:
            remote: Repository not found.
            fatal: repository 'https://github.com/segevfiner/vfsgen-test2/' not found
    exit status 1
    

    Whether I try go run or go generate, caused by the importPathToDir trick that tries to avoid working directory problems. I resorted to just placing the go:generate comment at the main package and using Filename to get vfsgen to write its output under the assets package instead of at the main package.

    From https://github.com/shurcooL/vfsgen/issues/83

    opened by segevfiner 2
  • "ignore" build tag is special in module mode

    See https://github.com/golang/go/issues/29598

    The README shows using an "assets_generate.go" with the "ignore" build tag, but that is ignored for the purposes of modules which will make it not pin the version of vfsgen if there isn't another file importing it.

    Unrelated to this I had trouble getting this to work with modules, the importPathToDir trick doesn't seem to work in modules mode. So I had to move the go:generate comment to the main module and use the Filename options to vfsgen to get it to write the file to the right place. This package could really use better/updated docs.

    opened by segevfiner 1
  • Change name of output file `assets_vfsdata.go`?

    Change name of output file `assets_vfsdata.go`?

    Update: Solution

    Change the name of the http.FileSystem-typed variable and the name of the output file will change accordingly:

    From:

    // Creates assets_vfsdata.go
    var Assets http.FileSystem = http.Dir("assets")
    

    To:

    // Creates adminfs_vfsdata.go
    var AdminFS http.FileSystem = http.Dir("assets")
    

    The go generate comment must also be updated accordingly:

    //go:generate go run -mod vendor github.com/shurcooL/vfsgen/cmd/vfsgendev -source="github.com/example/my-project/my-module/admin".AdminFS
    

    Can the output file name be changed?

    When I run go generate -mod=vendor ./... I get a list like this:

    writing assets_vfsdata.go
    writing assets_vfsdata.go
    

    I'd much prefer to be able to change the name of assets_vfsdata.go to be more relevant, something like:

    • admin_vfsdata.go
    • user_vfsdata.go

    Being able to specify the filename would make it easier to visually verify that I did run the command that I though I ran.

    For example, sometimes I want to build with the admin UI baked-in, but the dev version of the user UI, etc, so it would just be nice to have the feedback match what's actually happening.

    Looking at the help, it does not appear to have an option to specify the output.

    go run -mod vendor github.com/shurcooL/vfsgen/cmd/vfsgendev --help
    

    Changing the folder name from assets to admin also does not cause any change.

    For reference

    Here's an example of the structure of my project:

    admin/admin.go:

    // +build !dev
    //go:generate go run -mod vendor github.com/shurcooL/vfsgen/cmd/vfsgendev -source="github.com/example/my-project/my-module/admin".Assets
    
    package admin
    

    admin/admin_dev.go:

    // +build dev
    
    package admin
    
    import "net/http"
    
    var Assets http.FileSystem = http.Dir("admin")
    

    admin/admin/index.html:

    <h1>Hello, Admin!</h1>
    

    tools/tools.go:

    // +build tools
    
    // tools is a faux package for tracking tooling-related dependencies
    package tools
    
    import (
    	_ "git.rootprojects.org/root/go-gitver"
    	_ "github.com/shurcooL/vfsgen"
    	_ "github.com/shurcooL/vfsgen/cmd/vfsgendev"
    )
    
    opened by coolaj86 7
  • generate bytes

    generate bytes

    public method to return output bytes instead of writing directly to file, can be useful to use the output in various ways during generation before writing to file.

    same signature as Generate(), for example:

    err := vfsgen.GenerateBytes(http.Dir("assets"), vfsgen.Options{})
    
    opened by speier 2
A set of io/fs filesystem abstractions and utilities for Go

A set of io/fs filesystem abstractions and utilities for Go Please ⭐ this project Overview This package provides io/fs interfaces for: Cloud providers

null 7 Oct 7, 2021
s3fs provides a S3 implementation for Go1.16 filesystem interface.

S3 FileSystem (fs.FS) implementation.Since S3 is a flat structure, s3fs simulates directories by using prefixes and "/" delim. ModTime on directories is always zero value.

Jacek Szwec 122 Dec 27, 2021
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Steve Francia 4.1k Jan 6, 2022
A package to allow one to concurrently go through a filesystem with ease

skywalker Skywalker is a package to allow one to concurrently go through a filesystem with ease. Features Concurrency BlackList filtering WhiteList fi

Will Dixon 67 Jan 3, 2022
An implementation of the FileSystem interface for tar files.

TarFS A wrapper around tar.Reader. Implements the FileSystem interface for tar files. Adds an Open method, that enables reading of file according to i

Eyal Posener 48 Nov 18, 2021
memfs: A simple in-memory io/fs.FS filesystem

memfs: A simple in-memory io/fs.FS filesystem memfs is an in-memory implementation of Go's io/fs.FS interface. The goal is to make it easy and quick t

Peter Sanford 50 Dec 19, 2021
A Go io/fs filesystem implementation for reading files in a Github gists.

GistFS GistFS is an io/fs implementation that enables to read files stored in a given Gist. Requirements This module depends on io/fs which is only av

Jean Hadrien Chabran 122 Dec 6, 2021
A Small Virtual Filesystem in Go

This is a virtual filesystem I'm coding to teach myself Go in a fun way. I'm documenting it with a collection of Medium posts that you can find here.

Alyson 28 Jan 12, 2022
CRFS: Container Registry Filesystem

CRFS: Container Registry Filesystem Discussion: https://github.com/golang/go/issues/30829 Overview CRFS is a read-only FUSE filesystem that lets you m

Google 1.2k Jan 4, 2022
Encrypted overlay filesystem written in Go

An encrypted overlay filesystem written in Go. Official website: https://nuetzlich.net/gocryptfs (markdown source). gocryptfs is built on top the exce

null 2.1k Jan 5, 2022
Go filesystem implementations for various URL schemes

hairyhenderson/go-fsimpl This module contains a collection of Go filesystem implementations that can discovered dynamically by URL scheme. All filesys

Dave Henderson 217 Dec 22, 2021
filesystem for golang

filesystem filesystem for golang installation go get github.com/go-component/filesystem import import "github.com/go-component/filesystem" Usage sup

null 4 Jul 9, 2021
Tarserv serves streaming tar files from filesystem snapshots.

tarserv A collection of tools that allow serving large datasets from local filesystem snapshots. It is meant for serving big amounts of data to shell

Aurora 1 Dec 1, 2021
A basic file server automatically generates self certificates and serves the given folder.

A basic file server automatically generates self certificates and serves the given folder.

Ahmet ÖZER 3 Oct 30, 2021
Package cae implements PHP-like Compression and Archive Extensions.

Compression and Archive Extensions 中文文档 Package cae implements PHP-like Compression and Archive Extensions. But this package has some modifications de

ᴜɴᴋɴᴡᴏɴ 35 Sep 18, 2021
Allows parsing CSV files into custom structs and implements required fields that can't be empty

Welcome to Go Custom CSV Parser ?? Allows parsing CSV files into custom structs and implements required fields that can't be empty ?? Homepage Install

Elmedin Turkeš 2 Nov 9, 2021
searchHIBP is a golang tool that implements binary search over a hash ordered binary file.

searchHIBP is a golang tool that implements binary search over a hash ordered binary file.

fblz 0 Nov 9, 2021
Generate your cv from a yaml configuration file

Requirements wkhtmltopdf git * Flags -i string path to the webpage (index.html) -o string pdf output path -y string conf

alex 2 Nov 27, 2021
Bigfile -- a file transfer system that supports http, rpc and ftp protocol https://bigfile.site

Bigfile ———— a file transfer system that supports http, rpc and ftp protocol 简体中文 ∙ English Bigfile is a file transfer system, supports http, ftp and

null 213 Dec 31, 2021