A FileSystem Abstraction System for Go

Related tags

File System afero
Overview

afero logo-sm

A FileSystem Abstraction System for Go

Build Status Build status GoDoc Join the chat at https://gitter.im/spf13/afero

Overview

Afero is a filesystem framework providing a simple, uniform and universal API interacting with any filesystem, as an abstraction layer providing interfaces, types and methods. Afero has an exceptionally clean interface and simple design without needless constructors or initialization methods.

Afero is also a library providing a base set of interoperable backend filesystems that make it easy to work with afero while retaining all the power and benefit of the os and ioutil packages.

Afero provides significant improvements over using the os package alone, most notably the ability to create mock and testing filesystems without relying on the disk.

It is suitable for use in any situation where you would consider using the OS package as it provides an additional abstraction that makes it easy to use a memory backed file system during testing. It also adds support for the http filesystem for full interoperability.

Afero Features

  • A single consistent API for accessing a variety of filesystems
  • Interoperation between a variety of file system types
  • A set of interfaces to encourage and enforce interoperability between backends
  • An atomic cross platform memory backed file system
  • Support for compositional (union) file systems by combining multiple file systems acting as one
  • Specialized backends which modify existing filesystems (Read Only, Regexp filtered)
  • A set of utility functions ported from io, ioutil & hugo to be afero aware
  • Wrapper for go 1.16 filesystem abstraction io/fs.FS

Using Afero

Afero is easy to use and easier to adopt.

A few different ways you could use Afero:

  • Use the interfaces alone to define your own file system.
  • Wrapper for the OS packages.
  • Define different filesystems for different parts of your application.
  • Use Afero for mock filesystems while testing

Step 1: Install Afero

First use go get to install the latest version of the library.

$ go get github.com/spf13/afero

Next include Afero in your application.

import "github.com/spf13/afero"

Step 2: Declare a backend

First define a package variable and set it to a pointer to a filesystem.

var AppFs = afero.NewMemMapFs()

or

var AppFs = afero.NewOsFs()

It is important to note that if you repeat the composite literal you will be using a completely new and isolated filesystem. In the case of OsFs it will still use the same underlying filesystem but will reduce the ability to drop in other filesystems as desired.

Step 3: Use it like you would the OS package

Throughout your application use any function and method like you normally would.

So if my application before had:

os.Open("/tmp/foo")

We would replace it with:

AppFs.Open("/tmp/foo")

AppFs being the variable we defined above.

List of all available functions

File System Methods Available:

Chmod(name string, mode os.FileMode) : error
Chown(name string, uid, gid int) : error
Chtimes(name string, atime time.Time, mtime time.Time) : error
Create(name string) : File, error
Mkdir(name string, perm os.FileMode) : error
MkdirAll(path string, perm os.FileMode) : error
Name() : string
Open(name string) : File, error
OpenFile(name string, flag int, perm os.FileMode) : File, error
Remove(name string) : error
RemoveAll(path string) : error
Rename(oldname, newname string) : error
Stat(name string) : os.FileInfo, error

File Interfaces and Methods Available:

io.Closer
io.Reader
io.ReaderAt
io.Seeker
io.Writer
io.WriterAt

Name() : string
Readdir(count int) : []os.FileInfo, error
Readdirnames(n int) : []string, error
Stat() : os.FileInfo, error
Sync() : error
Truncate(size int64) : error
WriteString(s string) : ret int, err error

In some applications it may make sense to define a new package that simply exports the file system variable for easy access from anywhere.

Using Afero's utility functions

Afero provides a set of functions to make it easier to use the underlying file systems. These functions have been primarily ported from io & ioutil with some developed for Hugo.

The afero utilities support all afero compatible backends.

The list of utilities includes:

DirExists(path string) (bool, error)
Exists(path string) (bool, error)
FileContainsBytes(filename string, subslice []byte) (bool, error)
GetTempDir(subPath string) string
IsDir(path string) (bool, error)
IsEmpty(path string) (bool, error)
ReadDir(dirname string) ([]os.FileInfo, error)
ReadFile(filename string) ([]byte, error)
SafeWriteReader(path string, r io.Reader) (err error)
TempDir(dir, prefix string) (name string, err error)
TempFile(dir, prefix string) (f File, err error)
Walk(root string, walkFn filepath.WalkFunc) error
WriteFile(filename string, data []byte, perm os.FileMode) error
WriteReader(path string, r io.Reader) (err error)

For a complete list see Afero's GoDoc

They are available under two different approaches to use. You can either call them directly where the first parameter of each function will be the file system, or you can declare a new Afero, a custom type used to bind these functions as methods to a given filesystem.

Calling utilities directly

fs := new(afero.MemMapFs)
f, err := afero.TempFile(fs,"", "ioutil-test")

Calling via Afero

fs := afero.NewMemMapFs()
afs := &afero.Afero{Fs: fs}
f, err := afs.TempFile("", "ioutil-test")

Using Afero for Testing

There is a large benefit to using a mock filesystem for testing. It has a completely blank state every time it is initialized and can be easily reproducible regardless of OS. You could create files to your heart’s content and the file access would be fast while also saving you from all the annoying issues with deleting temporary files, Windows file locking, etc. The MemMapFs backend is perfect for testing.

  • Much faster than performing I/O operations on disk
  • Avoid security issues and permissions
  • Far more control. 'rm -rf /' with confidence
  • Test setup is far more easier to do
  • No test cleanup needed

One way to accomplish this is to define a variable as mentioned above. In your application this will be set to afero.NewOsFs() during testing you can set it to afero.NewMemMapFs().

It wouldn't be uncommon to have each test initialize a blank slate memory backend. To do this I would define my appFS = afero.NewOsFs() somewhere appropriate in my application code. This approach ensures that Tests are order independent, with no test relying on the state left by an earlier test.

Then in my tests I would initialize a new MemMapFs for each test:

func TestExist(t *testing.T) {
	appFS := afero.NewMemMapFs()
	// create test files and directories
	appFS.MkdirAll("src/a", 0755)
	afero.WriteFile(appFS, "src/a/b", []byte("file b"), 0644)
	afero.WriteFile(appFS, "src/c", []byte("file c"), 0644)
	name := "src/c"
	_, err := appFS.Stat(name)
	if os.IsNotExist(err) {
		t.Errorf("file \"%s\" does not exist.\n", name)
	}
}

Available Backends

Operating System Native

OsFs

The first is simply a wrapper around the native OS calls. This makes it very easy to use as all of the calls are the same as the existing OS calls. It also makes it trivial to have your code use the OS during operation and a mock filesystem during testing or as needed.

appfs := afero.NewOsFs()
appfs.MkdirAll("src/a", 0755)

Memory Backed Storage

MemMapFs

Afero also provides a fully atomic memory backed filesystem perfect for use in mocking and to speed up unnecessary disk io when persistence isn’t necessary. It is fully concurrent and will work within go routines safely.

mm := afero.NewMemMapFs()
mm.MkdirAll("src/a", 0755)

InMemoryFile

As part of MemMapFs, Afero also provides an atomic, fully concurrent memory backed file implementation. This can be used in other memory backed file systems with ease. Plans are to add a radix tree memory stored file system using InMemoryFile.

Network Interfaces

SftpFs

Afero has experimental support for secure file transfer protocol (sftp). Which can be used to perform file operations over a encrypted channel.

Filtering Backends

BasePathFs

The BasePathFs restricts all operations to a given path within an Fs. The given file name to the operations on this Fs will be prepended with the base path before calling the source Fs.

bp := afero.NewBasePathFs(afero.NewOsFs(), "/base/path")

ReadOnlyFs

A thin wrapper around the source Fs providing a read only view.

fs := afero.NewReadOnlyFs(afero.NewOsFs())
_, err := fs.Create("/file.txt")
// err = syscall.EPERM

RegexpFs

A filtered view on file names, any file NOT matching the passed regexp will be treated as non-existing. Files not matching the regexp provided will not be created. Directories are not filtered.

fs := afero.NewRegexpFs(afero.NewMemMapFs(), regexp.MustCompile(`\.txt$`))
_, err := fs.Create("/file.html")
// err = syscall.ENOENT

HttpFs

Afero provides an http compatible backend which can wrap any of the existing backends.

The Http package requires a slightly specific version of Open which returns an http.File type.

Afero provides an httpFs file system which satisfies this requirement. Any Afero FileSystem can be used as an httpFs.

httpFs := afero.NewHttpFs(<ExistingFS>)
fileserver := http.FileServer(httpFs.Dir(<PATH>))
http.Handle("/", fileserver)

Composite Backends

Afero provides the ability have two filesystems (or more) act as a single file system.

CacheOnReadFs

The CacheOnReadFs will lazily make copies of any accessed files from the base layer into the overlay. Subsequent reads will be pulled from the overlay directly permitting the request is within the cache duration of when it was created in the overlay.

If the base filesystem is writeable, any changes to files will be done first to the base, then to the overlay layer. Write calls to open file handles like Write() or Truncate() to the overlay first.

To writing files to the overlay only, you can use the overlay Fs directly (not via the union Fs).

Cache files in the layer for the given time.Duration, a cache duration of 0 means "forever" meaning the file will not be re-requested from the base ever.

A read-only base will make the overlay also read-only but still copy files from the base to the overlay when they're not present (or outdated) in the caching layer.

base := afero.NewOsFs()
layer := afero.NewMemMapFs()
ufs := afero.NewCacheOnReadFs(base, layer, 100 * time.Second)

CopyOnWriteFs()

The CopyOnWriteFs is a read only base file system with a potentially writeable layer on top.

Read operations will first look in the overlay and if not found there, will serve the file from the base.

Changes to the file system will only be made in the overlay.

Any attempt to modify a file found only in the base will copy the file to the overlay layer before modification (including opening a file with a writable handle).

Removing and Renaming files present only in the base layer is not currently permitted. If a file is present in the base layer and the overlay, only the overlay will be removed/renamed.

	base := afero.NewOsFs()
	roBase := afero.NewReadOnlyFs(base)
	ufs := afero.NewCopyOnWriteFs(roBase, afero.NewMemMapFs())

	fh, _ = ufs.Create("/home/test/file2.txt")
	fh.WriteString("This is a test")
	fh.Close()

In this example all write operations will only occur in memory (MemMapFs) leaving the base filesystem (OsFs) untouched.

Desired/possible backends

The following is a short list of possible backends we hope someone will implement:

  • SSH
  • S3

About the project

What's in the name

Afero comes from the latin roots Ad-Facere.

"Ad" is a prefix meaning "to".

"Facere" is a form of the root "faciō" making "make or do".

The literal meaning of afero is "to make" or "to do" which seems very fitting for a library that allows one to make files and directories and do things with them.

The English word that shares the same roots as Afero is "affair". Affair shares the same concept but as a noun it means "something that is made or done" or "an object of a particular type".

It's also nice that unlike some of my other libraries (hugo, cobra, viper) it Googles very well.

Release Notes

See the Releases Page.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Contributors

Names in no particular order:

License

Afero is released under the Apache 2.0 license. See LICENSE.txt

Comments
  • Add GCS Fs implementation

    Add GCS Fs implementation

    An implementation of GCS Fs, based on https://github.com/spf13/afero/pull/161

    Thanks @Zatte for the inital implementation!

    Changes vs the original PR:

    • Based on the latest master
    • Uses client as its base abstraction, not bucket => bucket is thus a part of path, the first component in it (!)
    • Has kinks and quirks of the GCS API "ironed out"
    • Has its own test suite now, based off the TarFS one with extra parts to test the writing/folders making scenarios
    • The suite allows for testing both with mocks and with the real bucket (see commented out block in the tests setup)
    • File modes are exactly those that are provided during files/directories opening. If not provided, defaults to 0755

    Bonus:

    • Supports gs://<bucket>/<path> URLs too, even though it's explicitly excluded from tests, since afero in general does not work with URLs.

    The original PR notes, which still apply to this version of the implementation + extra for this version:

    Limitations:

    • No Chmod support - The GCS ACL could probably be mapped to *nix style permissions but that would add another level of complexity and is ignored in this version.
    • No Chtimes support - Could be simulated with attributes (gcs a/m-times are set implicitly) but that's is left for another version.
    • NOTE: Not thread safe - Also assumes all file operations are done through the same instance of the GcsFs. File operations between different GcsFs instances are not guaranteed to be consistent.

    Performance implications

    • Sequential reads are performant
    • Sequential writes are performant.
    • Seek + Read or ReadAt is performant after the initial seek. (produces a warning)
    • Alternating reads/writes to the same file handler are highly inefficient. To get consistent FS behavior using an API that separates readers and writers we close any open readers before an write as well close open writers before a read (ensure the data is committed).
    • Seek + Write such as WriteAt, Truncate, Seek+Write will work as expected but with significant overhead. Doing a seek + write will in effect download the old file/object, overlay it with the new writes and save it back. This is done in a streaming fashion so large files will not clog the memory but will trigger a full download and upload of the file/object.
    opened by VOvchinnikov 16
  • S3 FS implementation

    S3 FS implementation

    This is a mostly-working/mostly-tested implementation of S3 as an FS.

    There are some gaps in the implementation, but it is otherwise pretty functional:

    • It doesn't support non-offset == 0 reads or writes
    • It doesn't support Truncate
    • It doesn't support Seek with whence set to 2
    • It doesn't support opening a file in append mode

    All of these things increase the complexity of the implementation significantly (some are questionably well-defined, e.g. append in an eventually-consistent FS like S3), so for now I've left them out.

    In order to reduce clutter in the root package github.com/spf13/afero, I put the whole implementation + tests in a sub package, github.com/spf13/afero/s3. In order to leverage the generic Fs tests in afero_test.go, I copy-pasted them into a sub package github.com/spf13/afero/tests. I also added a few tests to this file as I went along.

    In order to run the tests you must pass a bucket: go test -v github.com/spf13/afero/s3 -s3bucket <test bucket>. You also need to set AWS_REGION, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY.

    Let me know what you think 😄

    opened by rgarcia 12
  • Fix data race in InMemoryFile by locking Read and Write.

    Fix data race in InMemoryFile by locking Read and Write.

    Adds a sync.Mutex to the InMemoryFile struct to lock f.data field access in the Read and Write methods.

    Also locks access to f.closed in Open and Close methods.

    Fixes: #8.

    opened by jaqx0r 12
  • Add afero.Symlink

    Add afero.Symlink

    Hi, afero.Symlink() does not seem to be implemented. Are there any plans to add this method? See os.Symlink for reference. If you point me to the right direction, I could aso try to implement it myself.

    enhancement 
    opened by relnod 11
  • Explain the difference between filter and filesystem better

    Explain the difference between filter and filesystem better

    @vetinari The recent additions you've provided are great. Well written and thoughtfully done. In reading through the readme, it's not at all apparent what the difference is between a filter and a filesystem considering both adhere to the same interface.

    As I see it the main difference is that the filter is stackable at call time where any combination of filesystems as a filesystem needs to be predefined. It does beg the question though if everything should be a filter (or if we should incorporate that functionality into the FS since that's a compelling feature.

    If you could elaborate on the readme I'm sure many would benefit.

    opened by spf13 11
  • Add optional interfaces for Symlink and Readlink

    Add optional interfaces for Symlink and Readlink

    I created two interfaces to facilitate file systems optionally supporting the Symlink and Readlink operations from the os implementation.

    type Linker interface {
    	SymlinkIfPossible(oldname, newname string) error
    }
    type LinkReader interface {
    	ReadlinkIfPossible(name string) (string, error)
    }
    

    I opted to write them this way to preserve consistency with the Lstater interface created by @bep

    The actual Symlink and Readlink are only implemented in the OsFs, with proxy support in:

    • BasePathFs,
    • CopyOnWriteFs, and
    • ReadOnlyFs

    for consistency with the other two I implemented the Linker interface on ReadOnlyFs with it just returning a not supported error. I would appreciate feedback if this is the correct pattern, or if I should instead remove the method?

    The motivation for this change was to allow wrapping an existing use of afero in a project with the go-billy interface to allow a git integration to support a client request. The implementation is in the process of being tested in our logic, but I wanted to create the pull request and get feedback and make any necessary changes early.

    opened by Maldris 9
  • Google cloud storage v1

    Google cloud storage v1

    A patch to add google cloud storage support to Afero using GCS strong consistency guarantees to implement almost all afero intefaces.

    V1 scope limitations:

    • No Chmod support - The GCS ACL could probably be mapped to *nix style permissions but that would add another level of complexity and is ignored in this version.
    • No Chtimes support - Could be simulated with attributes (gcs a/m-times are set implicitly) but that's is left for another version.
    • NOTE: Not thread safe - Also assumes all file operations are done through the same instance of the GcsFs. File operations between different GcsFs instances are not guaranteed to be consistent.
    • File modes will always bee 0755/0664 for folders/files when read back

    Performance implications

    • Sequential reads are performant
    • Sequential writes are performant.
    • Seek + Read or ReadAt is performant after the initial seek.
    • Alternating reads/writes to the same file handler are highly inefficient. To get consistent FS behavior using an API that separates readers and writers we close any open readers before an write as well close open writers before a read (ensure the data is committed).
    • Seek + Write such as WriteAt, Truncate, Seek+Write will work as expected but with significant overhead. Doing a seek + write will in effect download the old file/object, overlay it with the new writes and save it back. This is done in a streaming fashion so large files will not clog the memory but will trigger a full download and upload of the file/object.

    How to test:

    • Ensure you have a service account setup and $GOOGLE_APPLICATION_CREDENTIALS pointing to that file. Other methods for creating a client are supported by manually creating the gcloud client and using NewGcsFs(...) instead of NewGcsFsFromDefaultCredentials(...) as below
    • Update afero_test.go
    var gcsFs = NewGcsFsFromDefaultCredentials(context.Background(), "[GCS_BUCKET_USED_FOR_TESTING]", string(filepath.Separator))
    var Fss = []Fs{&MemMapFs{}, &OsFs{}, gcsFs }
    
    opened by Zatte 8
  • sftp_test_go

    sftp_test_go

    I assume this is failing ? I am needing to connect to a sftp server, that does not use ssh key pair, and thought i woudl give this a try, but the fact that the test for SFTP is stubbed out makes me think its stubbed out because it is not ready.

    The test is trying to setup a local SFTP server. I am working against this SFTP server. http://docs.ipswitch.com/WS_FTP_Server71/ReleaseNotes/index.htm?k_id=ipswitch_com_ftp_documents_worldwide_ws_ftpserverv71releasenotes

    Can you let me know please.

    question 
    opened by joeblew99 8
  • Add Symlinker, Readlinker, EvalSymlinks.

    Add Symlinker, Readlinker, EvalSymlinks.

    Related: issue #174

    This PR improves symlink-related functionality in all afero fs types except the memory fs (still not sure what to do there).

    Note that I haven't tested anywhere but linux.

    opened by mpictor 7
  • fix Readdir() and Readdirnames()

    fix Readdir() and Readdirnames()

    fixes #7

    I first had to fix registerWithParent(), unRegisterWithParent() and findParent() to correctly use the MemDir of the parent directory. Then I added versions of Open() and Mkdir() that don't acquire a lock, that they can be used atomically during Create() and Mkdir(). Finally I changed the output of the InMemoryFileInfo.Name() and Readdirnames() functions to only return the last element of the path, as does the os package. I also added some tests.

    opened by mbertschler 7
  • Consider allowing other maintainers or moving to a github organization

    Consider allowing other maintainers or moving to a github organization

    Hello!

    Would you consider allowing other maintainers or moving to a github organization?

    There are a number of pull requests that date about two years and that have not gotten responses.

    If there was an afero organization on github, it could contain several repositories with file system implementation packages. You wouldn't have to merge them in the main repository and it would help the community organize around something.

    You could keep control of the main repository but give commit access to people in other repositories inside the github organization.

    opened by aviau 6
  • CVE-2022-32149: golang.org/x/text < 0.3.8

    CVE-2022-32149: golang.org/x/text < 0.3.8

    opened by MichaelMcAleer 0
  • MemMapFs MkdirAll error in parallel tests after go 1.18

    MemMapFs MkdirAll error in parallel tests after go 1.18

    Creating a MemMapFs in a table driven test running in parallel can cause an internal compiler error.

    Minimal reproduction test code:

    func TestMkdirAll_fails(t *testing.T) {
    	var tests = []struct {
    		fs afero.Fs
    	}{
    		{
    			fs: func() afero.Fs {
    				fs := afero.NewMemMapFs()
    				fs.MkdirAll("/home/alice", 0755)
    				return fs
    			}(),
    		},
    	}
    
    	for _, tc := range tests {
    		tc := tc // capture range variable
    		t.Run("sub_test", func(t *testing.T) {
    			_ = tc.fs
    		})
    	}
    }
    

    Produces the error:

    ./mkdirall_test.go:16:16: internal compiler error: order.stmt CALLMETH
    
    Please file a bug report including a short program that triggers the error.
    https://go.dev/issue/new
    

    Versions

    This works without error on golang 1.17.13. It fails for me on go 1.18.8 and go 1.19.3.

    I have tried on an older afero 1.2.2 as well as the current afero 1.9.3 with the same results.

    Workaround

    A workaround is to make sure the fs is created inside the test Run() rather than in the declaration of the test cases. Such as:

    func TestMkdirAll_ok(t *testing.T) {
    	var tests = []struct {
    		path     string
    		createFs func() afero.Fs
    	}{
    		{
    			createFs: func() afero.Fs {
    				fs := afero.NewMemMapFs()
    				fs.MkdirAll("/home/alice", 0755)
    				return fs
    			},
    		},
    	}
    
    	for _, tc := range tests {
    		tc := tc // capture range variable
    		t.Run("sub_test", func(t *testing.T) {
    			_ = tc.createFs()
    		})
    	}
    }
    

    To be honest, I'm not convinced my original code that was trying to share the fs between different goroutines was sensible. If this is not to be fixed, perhaps it would be worth mentioning in the readme that a MemMapFs cannot be passed between goroutines.

    opened by biinari 0
  • Add sftpfs files ReadAt method

    Add sftpfs files ReadAt method

    This method is still a TODO after 6 years despite being a simple delegate, i don't think i missed any real reason why it wasn't implemented but i could be wrong.

    opened by MRtecno98 0
  • fix(deps): golang.org/x/crypto, golang.org/x/text

    fix(deps): golang.org/x/crypto, golang.org/x/text

    • x/crypto - CVE-2020-9283, improper signature verification
    • x/crypto - CVE-2020-29652, nil pointer dereference
    • x/text - out of bounds read in <= v0.3.6
    opened by gnanderson 2
  • dependency vulnerability golang.org/x/crypto

    dependency vulnerability golang.org/x/crypto

    golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa has high severity vulnerability, please consider bumping at least to 0.0.0-20220314234659-1baeb1ce4c0b

    CVE info: https://avd.aquasec.com/nvd/2022/cve-2022-27191/

    opened by riuvshyn 0
Releases(v1.9.3)
  • v1.9.3(Nov 14, 2022)

    What's Changed

    • Fix concurrency issue in MemMapFs.Mkdir/MkdirAll by @bep in https://github.com/spf13/afero/pull/379

    Full Changelog: https://github.com/spf13/afero/compare/v1.9.2...v1.9.3

    Source code(tar.gz)
    Source code(zip)
  • v1.9.2(Jul 19, 2022)

    What's Changed

    • Make mem.File implement fs.ReadDirFile by @bep in https://github.com/spf13/afero/pull/371

    Full Changelog: https://github.com/spf13/afero/compare/v1.9.1...v1.9.2

    Source code(tar.gz)
    Source code(zip)
  • v1.9.1(Jul 19, 2022)

    What's Changed

    • Fix sorting in IOFS.ReadDir by @bep in https://github.com/spf13/afero/pull/372

    Full Changelog: https://github.com/spf13/afero/compare/v1.9.0...v1.9.1

    Source code(tar.gz)
    Source code(zip)
  • v1.9.0(Jul 14, 2022)

    What's Changed

    • Update go.mod to reflect that afero requires go >= 1.15.10 by @jeffwidman in https://github.com/spf13/afero/pull/346
    • Move CI tests to GitHub Action by @bep in https://github.com/spf13/afero/pull/367
    • Misc build/test fixes by @bep in https://github.com/spf13/afero/pull/368
    • Fix staticcheck lint errors by @bep in https://github.com/spf13/afero/pull/369
    • Make IOFS.ReadDir check for fs.ReadDirFile by @bep in https://github.com/spf13/afero/pull/366

    New Contributors

    • @jeffwidman made their first contribution in https://github.com/spf13/afero/pull/346

    Full Changelog: https://github.com/spf13/afero/compare/v1.8.2...v1.9.0

    Source code(tar.gz)
    Source code(zip)
  • v1.8.2(Mar 11, 2022)

    • Fix gcsfs RemoveAll by @typerat in https://github.com/spf13/afero/pull/340
    • bugfix: RemoveAll by @yearnfar in https://github.com/spf13/afero/pull/329
    Source code(tar.gz)
    Source code(zip)
  • v1.8.0(Jan 5, 2022)

  • v1.7.0(Dec 22, 2021)

    Add experimental GCS support in Afero. Experimental because the CI infra of afero does not test with real GCS buckets.

    Limitations:

    • No Chmod support - The GCS ACL could probably be mapped to *nix style permissions but that would add another level of complexity and is ignored in this version.
    • No Chtimes support - Could be simulated with attributes (gcs a/m-times are set implicitly) but that's is left for another version.
    • NOTE: Not thread safe - Also assumes all file operations are done through the same instance of the GcsFs. File operations between different GcsFs instances are not guaranteed to be consistent.

    Performance implications

    • Sequential reads are performant
    • Sequential writes are performant.
    • Seek + Read or ReadAt is performant after the initial seek. (produces a warning)
    • Alternating reads/writes to the same file handler are highly inefficient. To get consistent FS behavior using an API that separates readers and writers we close any open readers before an write as well close open writers before a read (ensure the data is committed).
    • Seek + Write such as WriteAt, Truncate, Seek+Write will work as expected but with significant overhead. Doing a seek + write will in effect download the old file/object, overlay it with the new writes and save it back. This is done in a streaming fashion so large files will not clog the memory but will trigger a full download and upload of the file/object.
    Source code(tar.gz)
    Source code(zip)
  • v1.6.0(Mar 20, 2021)

  • v1.3.5(Sep 2, 2020)

  • v1.3.4(Aug 9, 2020)

  • v1.3.0(Jun 17, 2020)

    Notable changes:

    • Add AIX support
    • Add zipfs, an archive/zip-based read-only filesystem
    • Add an optional interface for Symlink and Readlink

    Other enhancements:

    • TempFile support for wildcards instead of only prefixes
    • Implement sftpfs.OpenFile method
    • Add support for os.O_EXCL in MemMapFs

    Bugfixes:

    • Fixed the mem.File implementation to respect the io.ReaderAt contract by storing the original offset before the Read and restoring the offset after the Read.
    • Return closed error in mem.File.Write() after mem.File.Close() has been called
    • Add missing error check in RegexpFs.Open()
    Source code(tar.gz)
    Source code(zip)
Owner
Steve Francia
13x engineer • @golang Product Lead @Google • Prior EVP @Docker, @MongoDB, @Drupal board • @gohugoio, Cobra & spf13-vim creator
Steve Francia
Sqlite as a filesystem for golang apps

sqlitefs sqlitefs - пакет для подключения sqlite3 таблицы в качестве файловой си

null 8 Aug 7, 2022
Cross-platform file system notifications for Go.

File system notifications for Go fsnotify utilizes golang.org/x/sys rather than syscall from the standard library. Ensure you have the latest version

fsnotify 7.6k Nov 28, 2022
JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native

Juicedata, Inc 7.1k Nov 30, 2022
A user-space file system for interacting with Google Cloud Storage

gcsfuse is a user-space file system for interacting with Google Cloud Storage. Current status Please treat gcsfuse as beta-quality software. Use it fo

Google Cloud Platform 1.6k Dec 1, 2022
The Swift Virtual File System

*** This project is not maintained anymore *** The Swift Virtual File System SVFS is a Virtual File System over Openstack Swift built upon fuse. It is

OVHcloud 375 Oct 5, 2022
SeaweedFS a fast distributed storage system for blobs, objects, files, and data lake, for billions of files

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

Chris Lu 15.9k Nov 27, 2022
Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go

Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go Overview Goofys allows you to mount an S3 bucket as a filey system. It's a

Ka-Hing Cheung 4.4k Nov 27, 2022
Cross-platform file system notifications for Go.

File system notifications for Go fsnotify utilizes golang.org/x/sys rather than syscall from the standard library. Ensure you have the latest version

alan liu 0 Aug 7, 2017
Kitten is a distributed file system optimized for small file storage, inspired by Facebook's Haystack.

Kitten is a distributed file system optimized for small file storage, inspired by Facebook's Haystack.

Jack Lee 61 Aug 18, 2022
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Steve Francia 4.9k Nov 22, 2022
FSS3 is an S3 filesystem abstraction layer for Golang

FSS3 is an S3 filesystem abstraction layer for Golang

Ayman Bagabas 54 Aug 8, 2022
Database Abstraction Layer (dbal) for Go. Support SQL builder and get result easily (now only support mysql)

godbal Database Abstraction Layer (dbal) for go (now only support mysql) Motivation I wanted a DBAL that No ORM、No Reflect、Concurrency Save, support S

徐佳军 56 Nov 17, 2022
igor is an abstraction layer for PostgreSQL with a gorm like syntax.

igor igor is an abstraction layer for PostgreSQL, written in Go. Igor syntax is (almost) compatible with GORM. When to use igor You should use igor wh

Paolo Galeone 88 Oct 25, 2022
Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more)

gokv Simple key-value store abstraction and implementations for Go Contents Features Simple interface Implementations Value types Marshal formats Road

Philipp Gillé 490 Nov 21, 2022
Payment abstraction library - one interface for multiple payment processors ( inspired by Ruby's ActiveMerchant )

Sleet Payment abstraction library - interact with different Payment Service Providers (PsP) with one unified interface. Installation go get github.com

Bolt 101 Nov 24, 2022
Go (golang) wrapper for GDAL, the Geospatial Data Abstraction Library

------------- About ------------- The gdal.go package provides a go wrapper for GDAL, the Geospatial Data Abstraction Library. More information about

null 232 Nov 12, 2022
go routine control, abstraction of the Main and some useful Executors.如果你不会管理Goroutine的话,用它

routine Routine Architecture Quick Start package main import ( "log" "context" "github.com/x-mod/routine" ) func main(){ if err := routine.Main

null 54 Oct 5, 2022
Abstraction layer for simple rabbitMQ connection, messaging and administration

Jazz Abstraction layer for quick and simple rabbitMQ connection, messaging and administration. Inspired by Jazz Jackrabbit and his eternal hatred towa

SOCIFI Ltd. 17 Sep 27, 2022
Read a tar file contents using go1.16 io/fs abstraction

go-tarfs Read a tar file contents using go1.16 io/fs abstraction Usage ⚠️ go-tarfs needs go>=1.16 Install: go get github.com/nlepage/go-tarfs Use: pac

Nicolas Lepage 19 Nov 5, 2022
Encryption Abstraction Layer and Utilities for ratnet

What is Bencrypt? Bencrypt is an abstraction layer for cryptosystems in Go, that lets applications use hybrid cryptosystems without being coupled to t

null 16 Nov 9, 2022