A simple file embedder for Go

Overview

esc

GoDoc

esc embeds files into go programs and provides http.FileSystem interfaces to them.

It adds all named files or files recursively under named directories at the path specified. The output file provides an http.FileSystem interface with zero dependencies on packages outside the standard library.

Installation

go get -u github.com/mjibson/esc

Usage

esc [flag] [name ...]

The flags are:

-o=""
	output filename, defaults to stdout
-pkg="main"
	package name of output file, defaults to main
-prefix=""
	strip given prefix from filenames
-ignore=""
	regular expression for files to ignore
-include=""
	regular expression for files to include
-modtime=""
	Unix timestamp to override as modification time for all files
-private
	unexport functions by prefixing them with esc, e.g. FS -> escFS
-no-compress
	do not compress files

Accessing Embedded Files

After producing an output file, the assets may be accessed with the FS() function, which takes a flag to use local assets instead (for local development).

  • (_esc)?FS(Must)?(Byte|String) returns an asset as a (byte slice|string).
  • (_esc)?FSMust(Byte|String) panics if the asset is not found.

Go Generate

esc can be invoked by go generate:

//go:generate esc -o static.go -pkg server static

Example

Embedded assets can be served with HTTP using the http.FileServer. Assuming you have a directory structure similar to the following:

.
├── main.go
└── static
    ├── css
    │   └── style.css
    └── index.html

Where main.go contains:

package main

import (
	"log"
	"net/http"
)

func main() {
	// FS() is created by esc and returns a http.Filesystem.
	http.Handle("/static/", http.FileServer(FS(false)))
	log.Fatal(http.ListenAndServe(":8080", nil))
}

  1. Generate the embedded data: esc -o static.go static
  2. Start the server: go run main.go static.go
  3. Access http://localhost:8080/static/index.html to view the files.

You can see worked example in example dir just run it as go run example/main.go example/static.go

Issues
  • Add support for private esc function names

    Add support for private esc function names

    Add a -private command option to generate package private function names. Standard esc exported functions will be prefixed by esc.

    Default behavior remains package exported.

    opened by mweagle 6
  • Readdir Implementation - and covering with tests

    Readdir Implementation - and covering with tests

    • [X] Added Readdir
    • [X] Moved all string manipulations to html/template
    • [X] Cover embed.go with tests (about 80%)
    • [X] Cover generated static.go with test (about 80%)
      1. htttest for checking FileServer
      2. checking both mode - useLocal=false and useLocal=true for the same testcases
      3. added benchmark
    • [X] Added testdata dir with some static files (some html5 template)
    • [X] Added example folder with http.FileServer and generated from testdata
    • [x] Added golang.org/x/tools/imports for generated file (like goftm for generated file)
    • [X] Added to travis go generate and go test after it, as smoke tests.
    opened by mekegi 5
  • Feature idea: Directory contents

    Feature idea: Directory contents

    Let me preface my feature request by first stating that I really have enjoyed using esc. It is simple, uses the Go toolchain (generate), and makes me happy.

    I have run across a use case where a possible feature enhancement could make this tool a little easier to use for specific use cases. Below is a sample of some code that demonstrates a scenario where I have implemented a renderer for use in the Echo framework. Simply put it provides a Render method that tells the framework how to render requests.

    In the LoadTemplates method is where the imaginary code comes in. In here I imagine the ability to do something similar to ioutil.ReadDir() and get a slice of FileInfo structs. This way I can then iterate over the files and compile them as templates.

    package ui
    
    import (
    	"fmt"
    	"html/template"
    	"io"
    	"path/filepath"
    	"strings"
    
    	"myproject/app/compiledassets"
    	"github.com/labstack/echo"
    )
    
    var templates map[string]*template.Template
    
    /*
    TemplateRenderer describes a handlers for rendering layouts/pages
    */
    type TemplateRenderer struct {
    	templates *template.Template
    }
    
    /*
    NewTemplateRenderer creates a new struct
    */
    func NewTemplateRenderer(debugMode bool) *TemplateRenderer {
    	result := &TemplateRenderer{}
    	result.LoadTemplates(debugMode)
    
    	return result
    }
    
    /*
    Implements render interface expected by Echo 
    */
    func (t *TemplateRenderer) Render(w io.Writer, name string, data interface{}, ctx echo.Context) error {
    	var tmpl *template.Template
    	var ok bool
    
    	if tmpl, ok = templates[name]; !ok {
    		return fmt.Errorf("Cannot find template %s", name)
    	}
    
    	return tmpl.ExecuteTemplate(w, "layout", data)
    }
    
    /*
    Traverses the asset pages directory and compiles templates into a package level variable
    */
    func (t *TemplateRenderer) LoadTemplates(debugMode bool) {
    	templates = make(map[string]*template.Template)
    
    	/*
    	 * This code isn't real. It is what I imagine esc could look like
    	 * to get a directory of assets. Perhaps returns a slice of 
    	 * os.FileInfo structs
    	 */
    	files := compiledassets.DirContents("/www")
    
    	for _, file := range files {
    		if !file.IsDir() {
    			basename := file.Name()
    			trimmedName := strings.TrimSuffix(basename, filepath.Ext(basename))
    
    			templates["mainLayout:"+trimmedName], _ = template.Must(
    				template.New("layout").Parse(www.FSMustString(debugMode, "/www/myproject/layouts/mainLayout.gohtml")),
    			).Parse(www.FSMustString(debugMode, "/www/myproject/pages/"+basename))
    		}
    	}
    }
    

    What are your thoughts on this? Thanks for reading!

    opened by adampresley 4
  • Strip leading / from local path of directories

    Strip leading / from local path of directories

    Was reading system's filesystem root for directories instead of go proc's cwd. Bug was evidenced by "get /" not being translated get /index.html and also If you look at the generated static.go (or whatever) file, all directories have 'local' starting with /

    Thanks for Esc!

    opened by msample 4
  • can't serve index.html

    can't serve index.html

    $ esc -o static.go index.html style.css
    
    http.Handle("/", http.FileServer(FS(Debug)))
    

    I can access /style.css but not /index.html.

    [[email protected] martd (master ✗)]$ curl -I localhost:54321/index.html
    HTTP/1.1 301 Moved Permanently
    Location: ./
    Date: Mon, 24 Nov 2014 08:29:07 GMT
    Content-Type: text/plain; charset=utf-8
    
    [[email protected] martd (master ✗)]$ curl -I localhost:54321/
    HTTP/1.1 404 Not Found
    Content-Type: text/plain; charset=utf-8
    Date: Mon, 24 Nov 2014 08:29:15 GMT
    Content-Length: 19
    
    [[email protected] martd (master ✗)]$ curl -I localhost:54321/style.css
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Content-Length: 1378
    Content-Type: text/css; charset=utf-8
    Date: Mon, 24 Nov 2014 08:31:26 GMT
    
    [[email protected] martd (master ✗)]$
    
    opened by amitu 4
  • esc package imports embed locally

    esc package imports embed locally

    I see the reasoning behind this commit https://github.com/mjibson/esc/commit/ce63115657a87bf5a798fb557e2db659742e509a#diff-7ddfb3e035b42cd70649cc33393fe32cR6

    and all the ways it's used so far https://github.com/search?q=%22github.com%2Fmjibson%2Fesc%2Fembed%22&type=Code&utf8=%E2%9C%93

    But I figured for the base esc package, it made sense to import locally rather than with the github URL. This allows me to go run the esc/main.go package in some //go:generate commands where I've vendored esc ... Otherwise it complains about not finding esc in any of my path/bin ... which kinda breaks the spirit of vendoring in my use case.

    Didn't seem like this change would preclude others from using ...esc/embed as a library on its own, but I could be wrong.

    opened by hitjim 3
  • How does this differ from other similar projects?

    How does this differ from other similar projects?

    Hi there,

    I just discovered this project. I have a similar one called vfsgen, that also generates a .go file that implements http.FileSystem and only uses standard library.

    In its README, I have a list of alternatives because I think it's a good idea, and I'd like to add your project there. Can you help me understand in what ways this project is unique, what does it focus on/optimize for, and what would be a good description?

    Thanks!

    opened by dmitshur 3
  • Add FS(Must)?(Byte|String) helper functions.

    Add FS(Must)?(Byte|String) helper functions.

    Hi Matt,

    I like esc better than all the alternatives, and here are some helper functions to get assets directly.

    Use case: storing more complex templates in external files that are going to be embedded, e.g.

    pageTemplateStr := FSMustString(false, "/page.html")
    pageTemplate = template.Must(template.New("page").Parse(pageTemplateStr))
    

    If you don't like the names, feel free to change to whatever you think is best.

    opened by vkryukov 3
  • local file not being served

    local file not being served

    $ esc -o static.go static/
    

    Generates:

    var data = map[string]*file{
        "/static/mart.js": {compressed: "....", size: 519, local: "static/static/mart.js"},
    }
    

    Which is wrong as I do not have static/static/mart.js, and hence it fails.

    opened by amitu 3
  • [question] will this repository suddenly disappear?

    [question] will this repository suddenly disappear?

    Hi,

    Thank you so much for this project.

    Many of us felt much pain when go-bindata unexpectedly went down and hundreds of projects stopped building. This is to verify that there are no plans for esc to repeat the same trick.

    opened by vmarkovtsev 2
  • Making core logic availible as library

    Making core logic availible as library

    This may help solve a few different pain points with distributing and using esc. Currently we are invoking esc with //go:generate esc -modtime 0 -o static.go ...

    This works, but depends on esc being present on the user's system, and being the same version. For the ci server we have to fetch it each time the build runs (in travis, for example).

    This pr should not change the behavior of the app at all. What it does it pull all relevant logic into a separate package that can be imported by other code, and vendored appropriately. The main package now only exists to handle flags and invoke the embed package.

    I can make a generate.go file that includes it and runs it, and do //go:generate go run build/generate.go instead of invoking esc directly. Now all users, and the ci server will use the vendored copy of esc instead of depending on whatever has been installed independently.

    Thoughts?

    opened by captncraig 2
  • security vulnerability in embedded example jquery.min.js

    security vulnerability in embedded example jquery.min.js

    Hey,

    This is the lowest of priorities I know... I have recently started using a library scanner that detects vulnerabilities in libraries that I use. It seems to have caught on to the usage of the embedded jquery in example/static.go

    Would it be possible to update the example to something else?

    opened by trdyer 0
  • Data race on _escFile.name

    Data race on _escFile.name

    I hit this data race in my CI:

    ==================
    WARNING: DATA RACE
    Read at 0x00c0000d2a68 by goroutine 7:
      github.com/hortbot/hortbot/internal/db/migrations/esc.(*_escFile).Name()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/esc/esc.go:130 +0x3e
      github.com/golang-migrate/migrate/v4/source/httpfs.(*PartialDriver).Init()
          /home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/source/httpfs/partial_driver.go:48 +0x34c
      github.com/golang-migrate/migrate/v4/source/httpfs.New()
          /home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/source/httpfs/driver.go:21 +0xaf
      github.com/hortbot/hortbot/internal/db/migrations.newMigrate()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/migrations.go:51 +0x77
      github.com/hortbot/hortbot/internal/db/migrations.Up()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/migrations.go:15 +0x8d
      github.com/hortbot/hortbot/internal/db/migrations_test.withDatabase()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/migrations_test.go:67 +0x11e
      github.com/hortbot/hortbot/internal/db/migrations_test.TestUp()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/migrations_test.go:33 +0x52
      testing.tRunner()
          /home/travis/.gimme/versions/go/src/testing/testing.go:993 +0x1eb
    
    Previous write at 0x00c0000d2a68 by goroutine 77:
      github.com/hortbot/hortbot/internal/db/migrations/esc._escStaticFS.prepare.func1()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/esc/esc.go:59 +0xad
      sync.(*Once).doSlow()
          /home/travis/.gimme/versions/go/src/sync/once.go:66 +0x103
      sync.(*Once).Do()
          /home/travis/.gimme/versions/go/src/sync/once.go:57 +0x68
      github.com/hortbot/hortbot/internal/db/migrations/esc._escStaticFS.prepare()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/esc/esc.go:58 +0x160
      github.com/hortbot/hortbot/internal/db/migrations/esc._escStaticFS.Open()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/esc/esc.go:78 +0x46
      github.com/hortbot/hortbot/internal/db/migrations/esc.(*_escStaticFS).Open()
          <autogenerated>:1 +0x5f
      github.com/golang-migrate/migrate/v4/source/httpfs.(*PartialDriver).ReadUp()
          /home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/source/httpfs/partial_driver.go:111 +0x1b9
      github.com/golang-migrate/migrate/v4/source/httpfs.(*driver).ReadUp()
          <autogenerated>:1 +0x5b
      github.com/golang-migrate/migrate/v4.(*Migrate).newMigration()
          /home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/migrate.go:837 +0x96
      github.com/golang-migrate/migrate/v4.(*Migrate).readUp()
          /home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/migrate.go:612 +0x22b
    

    It appears as though there's a path where the static file is being returned before it has been prepared, but I can't see how at first glance.

    opened by zikaeroh 0
  • Precompute files MD5 / SHA1 / ...

    Precompute files MD5 / SHA1 / ...

    It would be useful to have the hashes of each files pre-computed and accessible with according methods so that we can send strong HTTP Etag headers for caching.

    opened by sylr 0
  • Doc Suggestion: Add example of Handlers for serving individual files

    Doc Suggestion: Add example of Handlers for serving individual files

    First of all, thanks for creating this! It works well and the API is clean and simple.

    I used it to bundle wasm and js into an example app so that a server and all content is bundled into the app. Some of my content is generated dynamically, so delegating everything to

    http.FileServer(FS(false))
    

    didn't seem workable. I ended up writing handler functions like the following to allow setting proper content headers. Seems to work very well, though I don't know if that's the cleanest possible way to code it.

    // fsSendStatic serves the requested file from the ESC FS.
    func fsSendStatic(w http.ResponseWriter, r *http.Request) {
    	_, err := w.Write(FSMustByte(false, r.URL.Path))
    	if err != nil {
    		log.Fatalf("Writing file %s returned error: %v", r.URL.Path, err)
    	}
    }
    
    // fsSendStaticJS serves the requested javascript file from the ESC FS.
    func fsSendStaticJS(w http.ResponseWriter, r *http.Request) {
    	w.Header().Set("Content-Type", "application/javascript")
    	fsSendStatic(w, r)
    }
    
    // fsSendStaticWasm serves the requested WebAssembly file  from the ESC FS.
    func fsSendStaticWasm(w http.ResponseWriter, r *http.Request) {
    	w.Header().Set("Content-Type", "application/wasm")
    	fsSendStatic(w, r)
    }
    

    It took a bit of trial and error to get it working. Just wondering if it might save others some time if the README included an example showing how to serve a mixture of static and dynamic content.

    opened by Michael-F-Ellis 0
  • new esc.Run signature requires more template

    new esc.Run signature requires more template

    I recently upgraded to the newest version of this module and I noticed that esc.Run() now requires an io.Writer. That's awesome, however old code requires a lot of template (see https://github.com/StackExchange/dnscontrol/pull/515 for an example).

    Would it make sense to have (for example) a nil writer result in the file being created automatically? This would help with migrating old code, and also would reduce the amount of template code needed.

    Just a though!

    opened by TomOnTime 1
Releases(v0.2.0)
  • v0.2.0(Apr 1, 2019)

  • v0.1.0(Sep 7, 2017)

    Initial tagged release. New recent features:

    • standard Go generated file syntax
    • -no-compress flag to disable compression
    • the default compression mode is gzip.BestCompression
    • -prefix now detects when it would have generated duplicate file names
    • safer file handling during output writing
    • embed package now returns errors instead of running log.Fatal
    Source code(tar.gz)
    Source code(zip)
A Go (golang) Custom Flutter Engine Embedder for desktop

Go Flutter desktop embedder ⚠️ Warning: this project has been moved to its own organization. Please take a look at its new location: github.com/go-flu

Pierre Champion | Drakirus 96 Jul 23, 2022
A zero dependency asset embedder for Go

Mewn A zero dependency asset embedder for Go. About Mewn is perhaps the easiest way to embed assets in a Go program. Here is an example: package main

Lea Anthony 86 Mar 5, 2022
This is a simple file storage server. User can upload file, delete file and list file on the server.

Simple File Storage Server This is a simple file storage server. User can upload file, delete file and list file on the server. If you want to build a

BH_Lin 0 Jan 19, 2022
A feature flag solution, with only a YAML file in the backend (S3, GitHub, HTTP, local file ...), no server to install, just add a file in a central system and refer to it. 🎛️

??️ go-feature-flag A feature flag solution, with YAML file in the backend (S3, GitHub, HTTP, local file ...). No server to install, just add a file i

Thomas Poignant 480 Aug 2, 2022
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).

中文 English 愿景:为用户提供最简单、可靠、高效的分布式文件系统。 go-fastdfs是一个基于http协议的分布式文件系统,它基于大道至简的设计理念,一切从简设计,使得它的运维及扩展变得更加简单,它具有高性能、高可靠、无中心、免维护等优点。 大家担心的是这么简单的文件系统,靠不靠谱,可不

小张 3.2k Aug 1, 2022
Simple Golang API to demonstrate file upload to fireabase storage and retrieving url of uploaded file.

go-firebase-storage -Work in progress ??️ Simple Golang API that uses Firebase as its backend to demonstrate various firebase services using Go such a

Victor Kabata 4 Oct 4, 2021
Simple-read-file - Example of how to read file in Go

simple-read-file This repository contains a simple example of how to read file i

Yessy Purnama 0 Jan 11, 2022
Sqlyog-password-decoder - Simple decode passwords from .sycs file (SQLyog export connections file)

Decode password: ./sqlyog-password-decoder -str password -action decode Encode p

Maxim Yanin 0 Nov 21, 2021
Convert scanned image PDF file to text annotated PDF file

Jisui (自炊) This tool is PoC (Proof of Concept). Jisui is a helper tool to create e-book. Ordinary the scanned book have not text information, so you c

Takumasa Sakao 27 Apr 7, 2022
Parse awesome-go README file and generate a new README file with repo info.

Awesome Go Extra All data are from awesome-go and GitHub API. Audio and Music Libraries for manipulating audio. Name Description Star Open Issues Crea

Wendell Sun 20 Jul 22, 2022
This is a Go Cli app that receives an string path to a log file, and based on it generates and prints in console an encoded polyline with the locations found in the log file.

GEOENCODE GO CLI APP DESCRIPTION This is a Go Cli app that receives an string path to a log file, and based on it generates and prints in console an e

Jose Luis Ojeda 1 Oct 1, 2021
This command line converts .webarchive file to resources embed .html file

webarchive-to-singlefile This command line converts Safari's .webarchive file to complete .html. Notice Only tested on MacOS. Google Chrome required.

会有猫的 2 May 12, 2022
Distributed File Store Application Consist of API Server to handle file operations and command line tool to do operations

Filestore Distributed File Store Application Consist of API Server to handle file operations and command line tool to do operations (store named binar

null 0 Nov 7, 2021
Sort the emails contained in a .csv file into a text file

Go convert csv to txt This snippet of code allows you to sort the emails contained in a .csv file into a text file.

Maxence Brochier 1 Nov 23, 2021
CLI filters the contents of the csv file according to the filters from the another file.

filtercsv CLI filters the contents of the csv file according to the filters from the another file. Made to process big files by a lots of filters. By

null 0 Dec 2, 2021
This command line converts thuderbird's exported RSS .eml file to .html file

thunderbird-rss-html This command line tool converts .html to .epub with images fetching. Install > go get github.com/gonejack/thunderbird-rss-html Us

会有猫的 0 Dec 15, 2021
DeepCopy a portable app that allows you to copy all forms of specified file types from your entire file system of the computer

DeepCopy a portable app that allows you to copy all forms of specified file types from your entire file system of the computer

subrahmanya  s hegade 1 Dec 20, 2021
Go-file-downloader-ftctl - A file downloader cli built using golang. Makes use of cobra for building the cli and go concurrent feature to download files.

ftctl This is a file downloader cli written in Golang which uses the concurrent feature of go to download files. The cli is built using cobra. How to

Dipto Chakrabarty 2 Jan 2, 2022
File-generator - Creates a file filled with bytes

creates a file filled with bytes ./filegen from to from : index of first piece t

Asaad Belarbi 0 Feb 14, 2022
Data visualization with chart, Create CSV file, Read Write CSV file

Data visualization with chart, Create CSV file, Read Write CSV file, Read from json file and many more in single project ......

MD FAISAL PORAG 1 Jan 13, 2022
Generates file.key file for IPFS Private Network.

ipfs-keygen Generates file.key file for IPFS Private Network. Installation go get -u github.com/reixmor/ipfs-keygen/ipfs-keygen Usage ipfs-keygen > ~/

Camilo Abel Monreal Aguero 0 Jan 18, 2022
Works with HashiCorp HCL. Allows to append the input file with blocks and attributes from the template file

About hclmergetool Works with HashiCorp HCL. Allows to append the input file with blocks and attributes from the template file Installation Binary Rel

Max Fedorov 0 Feb 6, 2022
Kitten is a distributed file system optimized for small file storage, inspired by Facebook's Haystack.

Kitten is a distributed file system optimized for small file storage, inspired by Facebook's Haystack.

Jack Lee 11 Jul 11, 2022
a tool for handling file uploads simple

baraka a tool for handling file uploads for http servers makes it easier to make operations with files from the http request. Contents Install Simple

Enes Furkan Olcay 44 Jul 14, 2022
Simple hosts file management in Golang (deprecated).

Goodhosts (deprecated) This library is now deprecated. See the goodhosts organisation for the current maintained version. Simple hosts file (/etc/host

Lex T 69 Mar 17, 2022
A simple logging module for go, with a rotating file feature and console logging.

A simple logging module for go, with a rotating file feature and console logging. Installation go get github.com/jbrodriguez/mlog Usage Sample usage W

Juan B. Rodriguez 25 Jun 11, 2022
A simple and light excel file reader to read a standard excel as a table faster | 一个轻量级的Excel数据读取库,用一种更`关系数据库`的方式解析Excel。

Intro | 简介 Expect to create a reader library to read relate-db-like excel easily. Just like read a config. This library can read all xlsx file correct

Back Yu 151 Jul 10, 2022
Simple, lightweight and faster response (JSON, JSONP, XML, YAML, HTML, File) rendering package for Go

Package renderer Simple, lightweight and faster response (JSON, JSONP, XML, YAML, HTML, File) rendering package for Go Installation Install the packag

Saddam H 241 Jul 3, 2022