A set of io/fs filesystem abstractions and utilities for Go

Related tags

File Handling fs
Overview

gopherfs logo-sm

A set of io/fs filesystem abstractions and utilities for Go

GoDoc

Please this project

Overview

This package provides io/fs interfaces for:

  • Cloud providers
  • Memory storage
  • Wrappers for the "os" package
  • Utilities for merging io.FS packages
  • A caching system with support for:
    • Redis
    • GroupCache
    • Disk cache

If you are looking to use a single group of interfaces to access any type of filesystem, look no further. This package brings the power of Go 1.16's io/fs package with new interfaces to allow for writable filesystems.

With these standard sets of interfaces we have expanded the reach of the standard library to cover several common sets of filesystems. In addition we provide a caching system allowing a cascade of cache fills to handle your file caching needs.

Below we will break down the packages and you can locate documentation within the GoDoc or the README in various packages.

Packages breakdown

└── fs
    ├── io
	│   ├── cache
	│   │   ├── disk
	│   │   ├── groupcache
	│   │   │   └── peerpicker
	│   │   └── redis
	│   ├── cloud
	│   │   └── azure
	│   │       └── blob
	│   │           ├── auth
	│   │           └── blob.go
	│   ├── mem
	│   │   └── simple
	│   └── os
  • fs: Additional interfaces to allow writeable filesystems and filesystem utility functions
  • fs/io/cache: Additional interfaces and helpers for our cache system
    • disk: A disk based cache filesystem
    • groupcache: A groupcache based filesystem
      • peerpicker: A multicast based peerpicker for groupcache (does not work in the cloud)
    • redis: A Redis based filesystem
  • fs/io/cloud: A collection of cloud provider filesystems
    • azure: A collection of Microsoft Azure filesystems
      • blob: A filesystem implementation based on Azure's Blob storage
  • fs/io/mem: A collection of local memory based filesystems
    • simple: A memory filesystem that requires ASCII based file paths, supports RO Pearson hasing
  • fs/io/os: A filesystem wrapper based around the "os" package

Examples

The most complete examples will be in the GoDoc for individual packages. But here are some excerpts for a few use cases.

Optimize embed.FS when not in debug mode

Choices

embed.FS is great. But what if you want to have readable JS for debug and compact code when in production? What if you'd also like to take several embed.FS and merge into a single tree?

Merge() and our simple memory storage to the rescue:

optimized := simple.New(simple.WithPearson())

err := Merge(
	optimized, 
	somePkg.Embeded, 
	"/js/", // Puts the content of the embed fs into a sub-directory
	WithTransform(
		func(name string, content []byte) ([]byte, error){
			// If we are in debug mode, we want unoptimized Javascript
			if debug {
				return content, nil
			}
			switch path.Ext(name){
			case "js":
				return optimizeJS(content)
			case "go":
				return nil, nil
			}
			return content, nil
		},
	),
)
if err != nil {
	// Do something
}
optimized.RO() // Locks this filesystem for readonly

Access Redis as a filesystem

Just Cause

One of the more popular caching systems around is Redis. Redis of course has a lot of options around it, but most use cases are simply as a filesystem. If this is your use, you can gain access to Redis using our fs/io/cache/redis implementation.

Here we simply create a connection to our local Redis cache, set a 5 minute expiration time on all files and then write a file.

redisFS, err := redis.New(
	redis.Args{Addr: "127.0.0.1:6379"},
	// This causes all files to exire in 5 minutes.
	// You can write a complex ruleset to handle different files at
	// different rates.
	redis.WithWriteFileOFOptions(
		regexp.MustCompile(`.*`),
		redis.ExpireFiles(5 * time.Minute),
	),
)
if err != nil {
	// Do something
}

if err := redisFS.WriteFile("gopher.jpg", gopherBytes, 0644); err != nil {
	// Do something
}

Build a Cascading Cache

Need for speed

Here we are going to build a cascading cache system. The goal is to have multiple layers of cache to look at before finally going to the source. This code will:

  • Pull from a groupcache first
  • Try a disk cache second
  • Pull from Azure's Blob Storage as the final resort

Note: This example uses a peerpicker for groupcache that will not work on major cloud providers, as they block broadcast and local multicast packets. You would need your own peerpicker to work for your cloud vendor.

// This sets up a filesystem accessing Azure's Blob storage. This is where
// our permanent location for files will be located.
blobStore, err := blob.NewFS("account", "container", *cred)
if err != nil {
	// Do something
}

// A new peerpicker that broadcasts on port 7586 to find new peers.
picker, err := peerpicker.New(7586)
if err != nil {
	// Do something
}

// A groupcache that our app uses to find cached entries.
gc, err := groupcache.New(picker)
if err != nil {
	// Do something
}

// A disk cache for when the groupcache doesn't have the data.
diskFS, err := disk.New(
	"", 
	disk.WithExpireCheck(1 * time.Minute), 
	disk.WithExpireFiles(30 * time.Minute),
)
if err != nil {
	// Do something
}

// Creates our diskCache that looks at our disk for a file and if it
// cannot find it, pulls from blob storage.
diskCache, err := cache.New(diskFS, blobStore)
if err != nil {
	// Do something
}

// Creates our cascader that will search the groupcache first, then
// search the disk cache and finally will pull from Azure Blob storage.
cascacder, err := cache.New(gc, diskCache)
if err != nil {
	// Do something
}

// This reads a file. Since this is our first read of this file, it will
// come from Azure Blob storage and back fill our caches.
b, err := cascacder.ReadFile("/path/to/file")
if err != nil {
	// Do something
}

Contributions

This project is open to contributions. The best way to contribute:

  1. Open a feature/bug request for the feature
  2. After a brief discusssion, fork the repo
  3. Commit your changes
  4. Create a Pull Request

#1 and #2 simple prevents any time wasted by for things that might not be within the scope for this project or to make sure a bug solution is the right solution.

We are looking for contributors to:

  • Support Azure Append and Page Blobs (we already support block blobs)
  • Support GCP Blob storage
  • Support GCP Filestore
  • Support AWS S3
  • Support AWS Elastic storage
  • Support SFTP

Alternatives

I should point out that there is already a great package for filesystem abstractions Afero. While I've never used it, spf13 is the author of several great packages and it looks like it has great support for several different filesystem types.

So why gopherfs? When I started writing this I was simply interested in trying to take advantage of io/fs. I saw Afero after I had written a couple of filesystems and it did not have io/fs support.

Afero was also geared towards its own method of abstraction that was built long before io/fs was a twinkle in the Go authors' eyes.

Most of my services don't need complicated file permissions that afero provides. For my use cases, the service is access control and has full rights to the file system.

I find Afero more complicated to use for my use cases and it doesn't have support for cloud provider filesystems (though you could write one).

If you need to support more complicated setups, I would use Aferno. I expect I might add wrappers around some of its filesytems at some point in the future.

A Go filesystem package for working with files and directories

Stowage A Go filesystem package for working with files and directories, it features a simple API with support for the common files and directories ope

null 19 May 28, 2021
s3fs provides a S3 implementation for Go1.16 filesystem interface.

S3 FileSystem (fs.FS) implementation.Since S3 is a flat structure, s3fs simulates directories by using prefixes and "/" delim. ModTime on directories is always zero value.

Jacek Szwec 122 Dec 27, 2021
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Steve Francia 4.1k Jan 6, 2022
A package to allow one to concurrently go through a filesystem with ease

skywalker Skywalker is a package to allow one to concurrently go through a filesystem with ease. Features Concurrency BlackList filtering WhiteList fi

Will Dixon 67 Jan 3, 2022
An implementation of the FileSystem interface for tar files.

TarFS A wrapper around tar.Reader. Implements the FileSystem interface for tar files. Adds an Open method, that enables reading of file according to i

Eyal Posener 48 Nov 18, 2021
memfs: A simple in-memory io/fs.FS filesystem

memfs: A simple in-memory io/fs.FS filesystem memfs is an in-memory implementation of Go's io/fs.FS interface. The goal is to make it easy and quick t

Peter Sanford 50 Dec 19, 2021
A Go io/fs filesystem implementation for reading files in a Github gists.

GistFS GistFS is an io/fs implementation that enables to read files stored in a given Gist. Requirements This module depends on io/fs which is only av

Jean Hadrien Chabran 122 Dec 6, 2021
A Small Virtual Filesystem in Go

This is a virtual filesystem I'm coding to teach myself Go in a fun way. I'm documenting it with a collection of Medium posts that you can find here.

Alyson 28 Jan 12, 2022
CRFS: Container Registry Filesystem

CRFS: Container Registry Filesystem Discussion: https://github.com/golang/go/issues/30829 Overview CRFS is a read-only FUSE filesystem that lets you m

Google 1.2k Jan 4, 2022
Encrypted overlay filesystem written in Go

An encrypted overlay filesystem written in Go. Official website: https://nuetzlich.net/gocryptfs (markdown source). gocryptfs is built on top the exce

null 2.1k Jan 5, 2022
Go filesystem implementations for various URL schemes

hairyhenderson/go-fsimpl This module contains a collection of Go filesystem implementations that can discovered dynamically by URL scheme. All filesys

Dave Henderson 217 Dec 22, 2021
filesystem for golang

filesystem filesystem for golang installation go get github.com/go-component/filesystem import import "github.com/go-component/filesystem" Usage sup

null 4 Jul 9, 2021
Tarserv serves streaming tar files from filesystem snapshots.

tarserv A collection of tools that allow serving large datasets from local filesystem snapshots. It is meant for serving big amounts of data to shell

Aurora 1 Dec 1, 2021
🍱 yet another collection of go utilities & tools

gut ?? Yet another collection of Go utilities & tools. A simple one. Just go with your gut feeling. Shortcuts Symbol ?? 弁当 Document License Build Stat

A Set of Useful Libraries and Utilities 21 Jan 3, 2022
gsheet is a CLI tool (and Golang package) for piping csv data to and from Google Sheets

gsheet Table of Contents Introduction Why? Installation Authentication and Authorization What about OAuth authentication? CLI Usage Sheet commands Dri

chris 13 Aug 5, 2021
Bigfile -- a file transfer system that supports http, rpc and ftp protocol https://bigfile.site

Bigfile ———— a file transfer system that supports http, rpc and ftp protocol 简体中文 ∙ English Bigfile is a file transfer system, supports http, ftp and

null 213 Dec 31, 2021
QueryCSV enables you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to a CSV file

QueryCSV enable you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to CSV file

Mohamed Shapan 100 Dec 22, 2021
🏵 Gee is tool of stdin to each files and stdout

Gee is tool of stdin to each files and stdout. It is similar to the tee command, but there are more functions for convenience. In addition, it was written as go. which provides output to stdout and files.

HAHWUL 46 Dec 25, 2021