A set of io/fs filesystem abstractions and utilities for Go

Related tags

File Handling fs

gopherfs logo-sm

A set of io/fs filesystem abstractions and utilities for Go


Please this project


This package provides io/fs interfaces for:

  • Cloud providers
  • Memory storage
  • Wrappers for the "os" package
  • Utilities for merging io.FS packages
  • A caching system with support for:
    • Redis
    • GroupCache
    • Disk cache

If you are looking to use a single group of interfaces to access any type of filesystem, look no further. This package brings the power of Go 1.16's io/fs package with new interfaces to allow for writable filesystems.

With these standard sets of interfaces we have expanded the reach of the standard library to cover several common sets of filesystems. In addition we provide a caching system allowing a cascade of cache fills to handle your file caching needs.

Below we will break down the packages and you can locate documentation within the GoDoc or the README in various packages.

Packages breakdown

└── fs
    ├── io
	│   ├── cache
	│   │   ├── disk
	│   │   ├── groupcache
	│   │   │   └── peerpicker
	│   │   └── redis
	│   ├── cloud
	│   │   └── azure
	│   │       └── blob
	│   │           ├── auth
	│   │           └── blob.go
	│   ├── mem
	│   │   └── simple
	│   └── os
  • fs: Additional interfaces to allow writeable filesystems and filesystem utility functions
  • fs/io/cache: Additional interfaces and helpers for our cache system
    • disk: A disk based cache filesystem
    • groupcache: A groupcache based filesystem
      • peerpicker: A multicast based peerpicker for groupcache (does not work in the cloud)
    • redis: A Redis based filesystem
  • fs/io/cloud: A collection of cloud provider filesystems
    • azure: A collection of Microsoft Azure filesystems
      • blob: A filesystem implementation based on Azure's Blob storage
  • fs/io/mem: A collection of local memory based filesystems
    • simple: A memory filesystem that requires ASCII based file paths, supports RO Pearson hasing
  • fs/io/os: A filesystem wrapper based around the "os" package


The most complete examples will be in the GoDoc for individual packages. But here are some excerpts for a few use cases.

Optimize embed.FS when not in debug mode


embed.FS is great. But what if you want to have readable JS for debug and compact code when in production? What if you'd also like to take several embed.FS and merge into a single tree?

Merge() and our simple memory storage to the rescue:

optimized := simple.New(simple.WithPearson())

err := Merge(
	"/js/", // Puts the content of the embed fs into a sub-directory
		func(name string, content []byte) ([]byte, error){
			// If we are in debug mode, we want unoptimized Javascript
			if debug {
				return content, nil
			switch path.Ext(name){
			case "js":
				return optimizeJS(content)
			case "go":
				return nil, nil
			return content, nil
if err != nil {
	// Do something
optimized.RO() // Locks this filesystem for readonly

Access Redis as a filesystem

Just Cause

One of the more popular caching systems around is Redis. Redis of course has a lot of options around it, but most use cases are simply as a filesystem. If this is your use, you can gain access to Redis using our fs/io/cache/redis implementation.

Here we simply create a connection to our local Redis cache, set a 5 minute expiration time on all files and then write a file.

redisFS, err := redis.New(
	redis.Args{Addr: ""},
	// This causes all files to exire in 5 minutes.
	// You can write a complex ruleset to handle different files at
	// different rates.
		redis.ExpireFiles(5 * time.Minute),
if err != nil {
	// Do something

if err := redisFS.WriteFile("gopher.jpg", gopherBytes, 0644); err != nil {
	// Do something

Build a Cascading Cache

Need for speed

Here we are going to build a cascading cache system. The goal is to have multiple layers of cache to look at before finally going to the source. This code will:

  • Pull from a groupcache first
  • Try a disk cache second
  • Pull from Azure's Blob Storage as the final resort

Note: This example uses a peerpicker for groupcache that will not work on major cloud providers, as they block broadcast and local multicast packets. You would need your own peerpicker to work for your cloud vendor.

// This sets up a filesystem accessing Azure's Blob storage. This is where
// our permanent location for files will be located.
blobStore, err := blob.NewFS("account", "container", *cred)
if err != nil {
	// Do something

// A new peerpicker that broadcasts on port 7586 to find new peers.
picker, err := peerpicker.New(7586)
if err != nil {
	// Do something

// A groupcache that our app uses to find cached entries.
gc, err := groupcache.New(picker)
if err != nil {
	// Do something

// A disk cache for when the groupcache doesn't have the data.
diskFS, err := disk.New(
	disk.WithExpireCheck(1 * time.Minute), 
	disk.WithExpireFiles(30 * time.Minute),
if err != nil {
	// Do something

// Creates our diskCache that looks at our disk for a file and if it
// cannot find it, pulls from blob storage.
diskCache, err := cache.New(diskFS, blobStore)
if err != nil {
	// Do something

// Creates our cascader that will search the groupcache first, then
// search the disk cache and finally will pull from Azure Blob storage.
cascacder, err := cache.New(gc, diskCache)
if err != nil {
	// Do something

// This reads a file. Since this is our first read of this file, it will
// come from Azure Blob storage and back fill our caches.
b, err := cascacder.ReadFile("/path/to/file")
if err != nil {
	// Do something


This project is open to contributions. The best way to contribute:

  1. Open a feature/bug request for the feature
  2. After a brief discusssion, fork the repo
  3. Commit your changes
  4. Create a Pull Request

#1 and #2 simple prevents any time wasted by for things that might not be within the scope for this project or to make sure a bug solution is the right solution.

We are looking for contributors to:

  • Support Azure Append and Page Blobs (we already support block blobs)
  • Support GCP Blob storage
  • Support GCP Filestore
  • Support AWS S3
  • Support AWS Elastic storage
  • Support SFTP


I should point out that there is already a great package for filesystem abstractions Afero. While I've never used it, spf13 is the author of several great packages and it looks like it has great support for several different filesystem types.

So why gopherfs? When I started writing this I was simply interested in trying to take advantage of io/fs. I saw Afero after I had written a couple of filesystems and it did not have io/fs support.

Afero was also geared towards its own method of abstraction that was built long before io/fs was a twinkle in the Go authors' eyes.

Most of my services don't need complicated file permissions that afero provides. For my use cases, the service is access control and has full rights to the file system.

I find Afero more complicated to use for my use cases and it doesn't have support for cloud provider filesystems (though you could write one).

If you need to support more complicated setups, I would use Aferno. I expect I might add wrappers around some of its filesytems at some point in the future.

  • Change *redis.FS.client type from *goredis.Client to goredis.Cmdable

    Change *redis.FS.client type from *goredis.Client to goredis.Cmdable

    Hey there,

    Thanks for the awesome library.

    I made a change to the *redis.FS struct to allow for a goredis.Cmdable interface instead of the default *goredis.Client so that *goredis.ClusterClients can also be used with the library.

    Additionally, the current default ofOptions sets the TTL field to goredis.KeepTTL, which requires redis server version >= 6.0, else it throws a syntax error. I added a comment in the code, though perhaps it should also be reflected in the docs as well.


    opened by dhh93 0
  • Changing *FS.client type definition to goredis.Cmdable interface

    Changing *FS.client type definition to goredis.Cmdable interface

    Addresses issue #2 with the following changes:

    • Changes the New constructor to accept redis.Cmdable interface instead of redis.Client
    • Adds redis.Cmdable implementation checks on various clients: i. redis.Client ii. redis.ClusterClient iii. redis.Pipeline
    • Removed Args type
    • Added note regarding redis server version >= 6.0 dependency based on KeepTTL arg. See:


    opened by dhh93 0
  • Changing *FS.client type definition to goredis.Cmdable interface

    Changing *FS.client type definition to goredis.Cmdable interface

    Hey there,

    Apologies for the earlier PR.

    What are your thoughts on the following changes:

    • Change *FS.client type definition from *goredis.Client to goredis.Cmdable interface to support *goredis.ClusterClient https://github.com/gopherfs/fs/blob/4538e04c7abb1262082ece90f190d2efcd7fccc9/io/cache/redis/redis.go#L49

    • Change New function to only accept Option parameters and add three functional options -- WithRedisClient, WithRedisCluster, and WithExistingClient.


    Example of the functional option:

    // WithRedisClient uses a single Redis client as the backend.
    func WithRedisClient(args Args) Option {
    	return func(f *FS) error {
    		c := redis.NewClient(&args)
    		f.client = c
    		return nil


    opened by dhh93 5
A Go filesystem package for working with files and directories

Stowage A Go filesystem package for working with files and directories, it features a simple API with support for the common files and directories ope

null 19 May 28, 2021
Grep archive search in any files on the filesystem, in archive and even inner archive.

grep-archive Grep archive search for string in any files on the filesystem, in archive and even inner archive. Supported archive format are : Tar Form

Michel Prunet 0 Jan 26, 2022
s3fs provides a S3 implementation for Go1.16 filesystem interface.

S3 FileSystem (fs.FS) implementation.Since S3 is a flat structure, s3fs simulates directories by using prefixes and "/" delim. ModTime on directories is always zero value.

Jacek Szwec 132 Jun 29, 2022
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Steve Francia 4.5k Jun 27, 2022
A package to allow one to concurrently go through a filesystem with ease

skywalker Skywalker is a package to allow one to concurrently go through a filesystem with ease. Features Concurrency BlackList filtering WhiteList fi

Will Dixon 80 Jun 18, 2022
An implementation of the FileSystem interface for tar files.

TarFS A wrapper around tar.Reader. Implements the FileSystem interface for tar files. Adds an Open method, that enables reading of file according to i

Eyal Posener 50 May 17, 2022
memfs: A simple in-memory io/fs.FS filesystem

memfs: A simple in-memory io/fs.FS filesystem memfs is an in-memory implementation of Go's io/fs.FS interface. The goal is to make it easy and quick t

Peter Sanford 64 Jun 21, 2022
A Go io/fs filesystem implementation for reading files in a Github gists.

GistFS GistFS is an io/fs implementation that enables to read files stored in a given Gist. Requirements This module depends on io/fs which is only av

Jean Hadrien Chabran 124 May 1, 2022
A Small Virtual Filesystem in Go

This is a virtual filesystem I'm coding to teach myself Go in a fun way. I'm documenting it with a collection of Medium posts that you can find here.

Alyson 31 Apr 18, 2022
CRFS: Container Registry Filesystem

CRFS: Container Registry Filesystem Discussion: https://github.com/golang/go/issues/30829 Overview CRFS is a read-only FUSE filesystem that lets you m

Google 1.2k Jul 1, 2022
Encrypted overlay filesystem written in Go

An encrypted overlay filesystem written in Go. Official website: https://nuetzlich.net/gocryptfs (markdown source). gocryptfs is built on top the exce

null 2.3k Jun 30, 2022
Go filesystem implementations for various URL schemes

hairyhenderson/go-fsimpl This module contains a collection of Go filesystem implementations that can discovered dynamically by URL scheme. All filesys

Dave Henderson 225 Jun 27, 2022
filesystem for golang

filesystem filesystem for golang installation go get github.com/go-component/filesystem import import "github.com/go-component/filesystem" Usage sup

null 4 Jul 9, 2021
Tarserv serves streaming tar files from filesystem snapshots.

tarserv A collection of tools that allow serving large datasets from local filesystem snapshots. It is meant for serving big amounts of data to shell

Aurora 1 Jan 11, 2022
Warp across your filesystem in ~5 ms

WarpDrive: the Go version. What does this do? Instead of having a huge cd routine to get where you want, with WarpDrive you use short keywords to warp

Ishan Goel 17 Jun 13, 2022
🍱 yet another collection of go utilities & tools

gut ?? Yet another collection of Go utilities & tools. A simple one. Just go with your gut feeling. Shortcuts Symbol ?? 弁当 Document License Build Stat

A Set of Useful Libraries and Utilities 24 Mar 5, 2022
gsheet is a CLI tool (and Golang package) for piping csv data to and from Google Sheets

gsheet Table of Contents Introduction Why? Installation Authentication and Authorization What about OAuth authentication? CLI Usage Sheet commands Dri

chris 17 Jun 21, 2022
Bigfile -- a file transfer system that supports http, rpc and ftp protocol https://bigfile.site

Bigfile ———— a file transfer system that supports http, rpc and ftp protocol 简体中文 ∙ English Bigfile is a file transfer system, supports http, ftp and

null 227 Jun 30, 2022