Pluggable, extensible virtual file system for Go

Related tags

File Handling vfs
Overview

vfs


GitHub tag Build Status GoDoc codecov License: MIT Go Report Card Mentioned in Awesome Go

Package vfs provides a pluggable, extensible, and opinionated set of file system functionality for Go across a number of file system types such as os, S3, and GCS.

Philosophy

When building our platform, initially we wrote a library that was something to the effect of

      if config.DISK == "S3" {
    	  // do some s3 file system operation
      } else if config.DISK == "mock" {
          // fake something
      } else {
          // do some native os.xxx operation
      }

Not only was ugly but because the behaviors of each "file system" were different and we had to constantly alter the file locations and pass a bucket string (even if the file system didn't know what a bucket was).

We found a handful of third-party libraries that were interesting but none of them had everything we needed/wanted. Of particular inspiration was https://github.com/spf13/afero in its composition of the super-powerful stdlib io.* interfaces. Unfortunately, it didn't support Google Cloud Storage and there was still a lot of passing around of strings and structs. Few, if any, of the vfs-like libraries provided interfaces to easily and confidently create new file system backends.

What we needed/wanted was the following(and more):
  • self-contained set of structs that could be passed around like a file/dir handle
  • the struct would represent an existing or nonexistent file/dir
  • provide common (and only common) functionality across all file system so that after initialization, we don't care what the underlying file system is and can therefore write our code agnostically/portably
  • use io.* interfaces such as io.Reader and io.Writer without needing to call a separate function
  • extensibility to easily add other needed file systems like Microsoft Azure Cloud File Storage
  • prefer native atomic functions when possible (ie S3 to S3 moving would use the native move api call rather than copy-delete)
  • a uniform way of addressing files regardless of file system. This is why we use complete URI's in vfssimple
  • fmt.Stringer interface so that the file struct passed to a log message (or other Stringer use) would show the URI
  • mockable file system
  • pluggability so that third-party implementations of our interfaces could be used

Install

Go install:

go get -u github.com/c2fo/vfs/v5

Usage

We provide vfssimple as basic way of initializing file system backends (see each implementations's docs about authentication). vfssimple pulls in every c2fo/vfs backend. If you need to reduce the backend requirements (and app memory footprint) or add a third party backend, you'll need to implement your own "factory". See backend doc for more info.

You can then use those file systems to initialize locations which you'll be referencing frequently, or initialize files directly

    osFile, err := vfssimple.NewFile("file:///path/to/file.txt")
    s3File, err := vfssimple.NewFile("s3://bucket/prefix/file.txt")

    osLocation, err := vfssimple.NewLocation("file:///tmp/")
    s3Location, err := vfssimple.NewLocation("s3://bucket/")

    osTmpFile, err := osLocation.NewFile("anotherFile.txt") // file at /tmp/anotherFile.txt

You can perform a number of actions without any consideration for the system's api or implementation details.

    osFileExists, err := osFile.Exists() // true, nil
    s3FileExists, err := s3File.Exists() // false, nil
    err = osFile.CopyToFile(s3File) // nil
    s3FileExists, err = s3File.Exists() // true, nil

    movedOsFile, err := osFile.MoveToLocation(osLocation)
    osFileExists, err = osFile.Exists() // false, nil (move actions delete the original file)
    movedOsFileExists, err := movedOsFile.Exists() // true, nil

    s3FileUri := s3File.URI() // s3://bucket/prefix/file.txt
    s3FileName := s3File.Name() // file.txt
    s3FilePath := s3File.Path() // /prefix/file.txt

File's io.* interfaces may be used directly:

    reader := strings.NewReader("Clear is better than clever")
    gsFile, err := vfssimple.NewFile("gs://somebucket/path/to/file.txt")

    byteCount, err := io.Copy(gsFile, reader)
    err := gsFile.Close()

Note: io.Copy() doesn't strictly define what happens if a reader is empty. This is complicated because io.Copy will first delegate actual copying in the following:

  1. if the io.Reader also implements io.WriterTo, WriteTo() will do the copy
  2. if the io.Writer also implements io.ReaderFrom, ReadFrom() will do the copy
  3. finally, if neither 1 or 2, io.Copy will do it's own buffered copy

In case 3, and most implementations of cases 1 and 2, if reader is empty, Write() never gets called. What that means for vfs is there is no way for us to ensure that an empty file does or doesn't get written on an io.Copy(). For instance OS always creates a file, regardless of calling Write() whereas S3 must Write() and Close().

As such, vfs cannot guarantee copy behavior except in our own CopyToFile, MoveToFile, CopyToLocation, and MoveToLocation functions. If you need to ensure a file gets copied/moved with io.Copy(), you must do so yourself OR use vfs's utils.TouchCopy

Third-party Backends

  • none so far

Feel free to send a pull request if you want to add your backend to the list.

See also:

Ideas

Things to add:

  • Add Azure storage backend
  • Provide better List() functionality with more abstracted filtering and paging (iterator?) Return File structs vs URIs?
  • Add better/any context.Context() support

Contributors

Brought to you by the Enterprise Pipeline team at C2FO:

https://github.com/c2fo/

Contributing

1. Fork it (<https://github.com/c2fo/vfs/fork>)
2. Create your feature branch (`git checkout -b feature/fooBar`)
3. Commit your changes (`git commit -am 'Add some fooBar'`)
4. Push to the branch (`git push origin feature/fooBar`)
5. Create a new Pull Request

License

Distributed under the MIT license. See `http://github.com/c2fo/vfs/License.md for more information.

Definitions

absolute path
  • A path is said to be absolute if it provides the entire context need to find a file, including the file system root. An absolute path must begin with a slash and may include . and .. directories.
file path
  • A file path ends with a filename and therefore may not end with a slash. It may be relative or absolute.
location path
  • A location/directory path must end with a slash. It may be relative or absolute.
relative path
  • A relative path is a way to locate a directory or file relative to another directory. A relative path may not begin with a slash but may include . and .. directories.
URI
  • A Uniform Resource Identifier (URI) is a string of characters that unambiguously identifies a particular resource. To guarantee uniformity, all URIs follow a predefined set of syntax rules, but also maintain extensibility through a separately defined hierarchical naming scheme (e.g. http://).

Interfaces

type File

type File interface {
	io.Closer
	io.Reader
	io.Seeker
	io.Writer
	fmt.Stringer

	// Exists returns boolean if the file exists on the file system.  Returns an error, if any.
	Exists() (bool, error)

	// Location returns the vfs.Location for the File.
	Location() Location

	// CopyToLocation will copy the current file to the provided location.
	//
	//   * Upon success, a vfs.File, representing the file at the new location, will be returned.
	//   * In the case of an error, nil is returned for the file.
	//   * CopyToLocation should use native functions when possible within the same scheme.
	//   * If the file already exists at the location, the contents will be overwritten with the current file's contents.
	//   * CopyToLocation will Close both the source and target Files which therefore can't be appended to without first
	//     calling Seek() to move the cursor to the end of the file.
	CopyToLocation(location Location) (File, error)

	// CopyToFile will copy the current file to the provided file instance.
	//
	//   * In the case of an error, nil is returned for the file.
	//   * CopyToLocation should use native functions when possible within the same scheme.
	//   * If the file already exists, the contents will be overwritten with the current file's contents.
	//   * CopyToFile will Close both the source and target Files which therefore can't be appended to without first
	//     calling Seek() to move the cursor to the end of the file.
	CopyToFile(file File) error

	// MoveToLocation will move the current file to the provided location.
	//
	//   * If the file already exists at the location, the contents will be overwritten with the current file's contents.
	//   * If the location does not exist, an attempt will be made to create it.
	//   * Upon success, a vfs.File, representing the file at the new location, will be returned.
	//   * In the case of an error, nil is returned for the file.
	//   * When moving within the same Scheme, native move/rename should be used where possible.
	//   * If the file already exists, the contents will be overwritten with the current file's contents.
	//   * MoveToLocation will Close both the source and target Files which therefore can't be appended to without first
	//     calling Seek() to move the cursor to the end of the file.
	MoveToLocation(location Location) (File, error)

	// MoveToFile will move the current file to the provided file instance.
	//
	//   * If the file already exists, the contents will be overwritten with the current file's contents.
	//   * The current instance of the file will be removed.
	//   * MoveToFile will Close both the source and target Files which therefore can't be appended to without first
	//     calling Seek() to move the cursor to the end of the file.
	MoveToFile(file File) error

	// Delete unlinks the File on the file system.
	Delete() error

	// LastModified returns the timestamp the file was last modified (as *time.Time).
	LastModified() (*time.Time, error)

	// Size returns the size of the file in bytes.
	Size() (uint64, error)

	// Path returns absolute path, including filename, ie /some/path/to/file.txt
	//
	// If the directory portion of a file is desired, call
	//   someFile.Location().Path()
	Path() string

	// Name returns the base name of the file path.
	//
	// For file:///some/path/to/file.txt, it would return file.txt
	Name() string
	
	// Touch creates a zero-length file on the vfs.File if no File exists.  Update File's last modified timestamp.
    	// Returns error if unable to touch File.
        Touch() error

	// URI returns the fully qualified absolute URI for the File.  IE, s3://bucket/some/path/to/file.txt
	URI() string
}

File represents a file on a file system. A File may or may not actually exist on the file system.

type FileSystem

type FileSystem interface {
	// NewFile initializes a File on the specified volume at path 'absFilePath'.
	//
	//   * Accepts volume and an absolute file path.
	//   * Upon success, a vfs.File, representing the file's new path (location path + file relative path), will be returned.
	//   * On error, nil is returned for the file.
	//   * Note that not all file systems will have a "volume" and will therefore be "":
	//       file:///path/to/file has a volume of "" and name /path/to/file
	//     whereas
	//       s3://mybucket/path/to/file has a volume of "mybucket and name /path/to/file
	//     results in /tmp/dir1/newerdir/file.txt for the final vfs.File path.
	//   * The file may or may not already exist.
	NewFile(volume string, absFilePath string) (File, error)

	// NewLocation initializes a Location on the specified volume with the given path.
	//
	//   * Accepts volume and an absolute location path.
	//   * The file may or may not already exist. Note that on key-store file systems like S3 or GCS, paths never truly exist.
	//   * On error, nil is returned for the location.
	//
	// See NewFile for note on volume.
	NewLocation(volume string, absLocPath string) (Location, error)

	// Name returns the name of the FileSystem ie: Amazon S3, os, Google Cloud Storage, etc.
	Name() string	

	// Scheme returns the uri scheme used by the FileSystem: s3, file, gs, etc.
	Scheme() string

	// Retry will return the retry function to be used by any file system.
	Retry() Retry
}

FileSystem represents a file system with any authentication accounted for.

type Location

type Location interface {
	// String returns the fully qualified absolute URI for the Location.  IE, file://bucket/some/path/
	fmt.Stringer

	// List returns a slice of strings representing the base names of the files found at the Location.
	//
	//   * All implementations are expected to return ([]string{}, nil) in the case of a non-existent directory/prefix/location.
	//   * If the user cares about the distinction between an empty location and a non-existent one, Location.Exists() should
	//     be checked first.
	List() ([]string, error)

	// ListByPrefix returns a slice of strings representing the base names of the files found in Location whose filenames
	// match the given prefix.
	//
	//   * All implementations are expected to return ([]string{}, nil) in the case of a non-existent directory/prefix/location.
	//   * "relative" prefixes are allowed, ie, listByPrefix from "/some/path/" with prefix "to/somepattern" is the same as
	//     location "/some/path/to/" with prefix of "somepattern"
	//   * If the user cares about the distinction between an empty location and a non-existent one, Location.Exists() should
	//     be checked first.
	ListByPrefix(prefix string) ([]string, error)

	// ListByRegex returns a slice of strings representing the base names of the files found in Location that matched the
	// given regular expression.
	//
	//   * All implementations are expected to return ([]string{}, nil) in the case of a non-existent directory/prefix/location.
	//   * If the user cares about the distinction between an empty location and a non-existent one, Location.Exists() should
	//     be checked first.
	ListByRegex(regex *regexp.Regexp) ([]string, error)

	// Volume returns the volume as string. In URI parlance, volume equates to authority.
	// For example s3://mybucket/path/to/file.txt, volume would return "mybucket".
	//
	// Note: Some file systems may not have a volume and will return "".
	Volume() string

	// Path returns absolute location path, ie /some/path/to/.  An absolute path must be resolved to it's shortest path:
	// see path.Clean
	Path() string

	// Exists returns boolean if the location exists on the file system. Returns an error if any.
	Exists() (bool, error)

	// NewLocation is an initializer for a new Location relative to the existing one.
	//
	// Given location:
	//     loc := fs.NewLocation(:s3://mybucket/some/path/to/")
	// calling:
	//     newLoc := loc.NewLocation("../../")
	// would return a new vfs.Location representing:
	//     s3://mybucket/some/
	//
	//   * Accepts a relative location path.
	NewLocation(relLocPath string) (Location, error)

	// Given location:
	// 	   loc := fs.NewLocation("file:///some/path/to/")
	// calling:
	//     loc.ChangeDir("../../")
	// would update the current location instance to
	// file:///some/.
	//
	//   * ChangeDir accepts a relative location path.
	ChangeDir(relLocPath string) error

	//FileSystem returns the underlying vfs.FileSystem struct for Location.
	FileSystem() FileSystem

	// NewFile will instantiate a vfs.File instance at or relative to the current location's path.
	//
	//   * Accepts a relative file path.
	//   * In the case of an error, nil is returned for the file.
	//   * Resultant File path will be the shortest path name equivalent of combining the Location path and relative path, if any.
	//       ie, /tmp/dir1/ as location and relFilePath "newdir/./../newerdir/file.txt"
	//       results in /tmp/dir1/newerdir/file.txt for the final vfs.File path.
	//   * Upon success, a vfs.File, representing the file's new path (location path + file relative path), will be returned.
	//   * The file may or may not already exist.
	NewFile(relFilePath string) (File, error)

	// DeleteFile deletes the file of the given name at the location.
	//
	// This is meant to be a short cut for instantiating a new file and calling delete on that, with all the necessary
	// error handling overhead.
	//
	// * Accepts relative file path.
	DeleteFile(relFilePath string) error

	// URI returns the fully qualified absolute URI for the Location.  IE, s3://bucket/some/path/
	//
	// URI's for locations must always end with a slash.
	URI() string
}

Location represents a file system path which serves as a start point for directory-like functionality. A location may or may not actually exist on the file system.

type Options

type Options interface{}

Options are structs that contain various options specific to the file system

type Retry

type Retry func(wrapped func() error) error

Retry is a function that can be used to wrap any operation into a definable retry operation. The wrapped argument is called by the underlying VFS implementation.

Ex:

    var retrier Retry = func(wrapped func() error) error {
      var ret error
      for i := 0; i < 5; i++ {
         if err := wrapped(); err != nil { ret = err; continue }
      }
      return ret
    }

func DefaultRetryer

func DefaultRetryer() Retry

DefaultRetryer returns a no-op retryer which simply calls the wrapped command without looping.

Issues
  • Using gs.File.CopyToFile after reading the file results in 0 byte file at the destination

    Using gs.File.CopyToFile after reading the file results in 0 byte file at the destination

    This is because the same reader is used for copying and for reading, so by the time you get to CopyToFile the reader is already at the end of the file.

    Probably the simplest solution to this fix will be to just Seek(0,0) before doing the copy.

    As a workaround the seek can be done in user code.

    Any inputs on possible resolutions is welcome, I think it would be good to just call Seek(0, 0) and I will prepare a patch for that.

    opened by aucampia 7
  • File.Write() creates empty files

    File.Write() creates empty files

    Code:

    var Location vfs.Location = initLocation()
    
    func initLocation() vfs.Location {
    
    	if location, err := vfssimple.NewLocation("file:///path/to/public"); err != nil {
    		log.Fatal(err.Error())
    	} else if ok, err := location.Exists(); err != nil {
    		log.Fatal(err.Error())
    	} else if !ok {
    		log.Fatal("Unexpected error")
    	} else {
    		return location
    	}
    
    	return nil
    }
    
    func MakeTestFile() {
    	f, err := Location.NewFile("test.txt")
    
    	if err != nil {
    		log.Fatal(err.Error())
    	}
    
    	if num, err := f.Write([]byte("This is a test text")); err != nil {
    		log.Fatal(err.Error())
    	} else {
    		log.Println("String was writed in file")
    	}
    
    	f.Close()
    }
    

    Problem:

    The function MakeTestFile creates a file without errors, but it is empty.

    System information:

    • Go version: 1.14.1
    • Os/Arch: linux/amd64
    • Package version: 1.6.2
    bug 
    opened by trixden 4
  • How to set S3ForcePathStyle for AWS Client?

    How to set S3ForcePathStyle for AWS Client?

    I'm use Minio as s3 storage. AWS Client can use it only with S3ForcePathStyle=True. s3.Options struct does not have this parameter. Each initialization I have to create a client manually. It is very uncomfortable. It may be possible to add this parameter to the s3.Options?

    enhancement 
    opened by yellow-sky 3
  • EP-5650 -> EP-5650 | Implement Azure Interfaces

    EP-5650 -> EP-5650 | Implement Azure Interfaces

    In order to add a new backend for Azure in VFS we need to implement the FileSystem, Location and File Interfaces in a new Azure package. For access to Azure, see linked SECOPS ticket.

    AC:

    • Docs are created for base implementation
    • Azure package is created
    • Base configs are added
    • FileSystem, Location and File Interfaces has been implemented for Azure
    • FIlesystem config is defined and added, environment config to be added later
    ep story 
    opened by dmcilvain44 2
  • bug fixes

    bug fixes

    [5.5.4] - 2020-07-06

    Fixed

    • fixes #21 - OS file.Location().ChangeDir(something/) should not affect location
    • fixes #44 - make S3 waitUntilFileExists Exists error more descriptive of root error
    • fixes #45 - fix ASCII diagram alignment for SFTP doc

    Added

    • Improved test coverage for os.FileSystem
    opened by funkyshu 2
  • Fix space issue

    Fix space issue

    Ensure that spaces (url encoded or not) in filenames and paths work as expected when copying (particularly for s3 to s3 native copyObject). Fixes #36

    opened by funkyshu 2
  • EP-2061 -> master | Prep vfs for opensource

    EP-2061 -> master | Prep vfs for opensource

    JIRA Link

    Description

    Our golang vfs library was built with the intention of releasing as opensource. We held off until we could make it prettier and more features but after more than a year in production and only minor changes, we still have yet to release it. Since then google has release a clould agnostic swiss army knife library call go-cloud that implements some vfs-type functionality but nowhere near as full-featured.

    We should prep VFS code for public release. Some steps include:

    • -Ensuring vfs works with the latest third party libs (google-sdk, aws-sdk, etc)- Unsafe to do this until we upgrade to 1.11
    • Ensure docs are up-to-date
    • Consider setting up go.mod (vgo) and/or dep (can these coexist with glide files?)
    • Clean repo? (there are one or two commit messages that contain Jira ticket info).
    • Get sign-off from SecOps and JC
    • Make repo public There may be other items i haven't considered.

    Test Steps

    Put your test steps here....

    task ep 
    opened by funkyshu 2
  • Go 1.18 upgrade

    Go 1.18 upgrade

    • Update go.mod min version to Go 1.17
    • Update actions/setup-go action to v3
    • Update actions/checkout action to v3
    • Update golangci/golangci-lint-action action to v3
    • Update actions to test only supported Go versions (1.17, 1.18)
    • Updated dependencies (ie, Google Cloud, Azure, AWS sdks)
    size/L 
    opened by funkyshu 1
  • Assume os backend by default in vfssimple

    Assume os backend by default in vfssimple

    Is your feature request related to a problem? Please describe. Currently you need to specify file paths with a schema. However, in many cases, existing code uses absolute/relative paths to read the local filesystem without specifying the file:// scheme. To make it easier to transition to vfssimple it would be great if those existing file paths continued to work transparently.

    Describe the solution you'd like

    vfssimple.NewFile("relative/file.txt") // Read using the os backend (currently results in ErrMissingScheme)
    vfssimple.newFile("/absolute/file.txt") // Read using the os backend (currently results in ErrMissingScheme)
    vfssimple.newFile("file:///absolute/file.txt") // Read using the os backend
    

    Describe alternatives you've considered n/a

    Additional context n/a

    opened by tomzx 1
  • Resolving registered FS

    Resolving registered FS

    I'm use vfssimple with regestered fs (s3) as shown in example: https://github.com/C2FO/vfs/blob/ab2673ff5f8388b94a8db22464de5c2f7e3e8105/docs/vfssimple.md#usage

    But this example don't work without small additional:

    backend.Unregister(s3.Scheme)

    Backend resolve fs by sheme\sheme+bucket\scheme+bucket+name and return first found - default registered fs.

    Maybe there is an option to disable automatic registration? If not, then it may be add to example?

    bug 
    opened by yellow-sky 1
  • Looking To Update Go Linter

    Looking To Update Go Linter

    golangci-lint run WARN [runner] The linter 'golint' is deprecated (since v1.41.0) due to: The repository of the linter has been archived by the owner. Replaced by revive.

    enhancement 
    opened by arnab-chaudhuri-c2fo 1
  • Add support for deleting file versions to s3 and gs

    Add support for deleting file versions to s3 and gs

    Changes Included:

    • file.delete() and location.deleteFile() now support optional argument to specify delete options
    • S3's implementation of file.delete() now removes all the versions of the file object if deleteOptions.DeleteAllVersions is passed in the arguments.
    • GS's implementation of file.delete() now removes all the generations of the file object if deleteOptions.DeleteAllVersions is passed in the arguments.
    • Other implementations of File interface, are noop for deleteOptions.DeleteAllVersions. (Azure implementation is pending)

    Design:

    • In order to keep the delete interface non-passive, decided to change it to a variadic function that takes an interface as argument.
    • If we wish to add additional options for delete, the new options will simply implement the options.DeleteOption interface and add any additional methods that are needed for that option.
    • This approach also allows us to add support for options for other methods as well. For example, if we want to pass file permissions while creating a file, we will just create an interface like FilePermissionsOption and create implementations according to the need.
    • Here is an example that shows the design https://github.com/C2FO/vfs/compare/judd-delete-options
    size/L 
    opened by dhondgepooja 2
  • mem.File doesn't store Write bytes in a way accessible to Reads

    mem.File doesn't store Write bytes in a way accessible to Reads

    See:

    package main
    
    import (
    	"fmt"
    
    	"github.com/c2fo/vfs/v5/backend/mem"
    )
    
    func main() {
    	m := mem.NewFileSystem()
    	f, err := m.NewFile("", "/path/to/file.txt")
    	if err != nil {
    		panic(err)
    	}
    
    	// write to file
    	b, err := f.Write([]byte("this is some sample text\n"))
    	if err != nil {
    		panic(err)
    	}
    
    	// check byte count
    	if b != 25 {
    		panic(fmt.Errorf("bytes written should be 25 but got: %d", b))
    	}
    
    	// close file (so we can read from it)
    	err = f.Close()
    	if err != nil {
    		panic(err)
    	}
    
    	checkBytes := make([]byte, 5)
    	_, err = f.Read(checkBytes)
    	if err != nil {
    		panic(err)
    	}
    
    	// should match, but doesn't
    	if string(checkBytes) != "bytes" {
    		panic(fmt.Errorf("read bytes doesn't match 'bytes', got: %s", string(checkBytes)))
    	}
    
    }
    

    I suspect there will be issues with Seek as well. Mem probably deserves a rewrite.

    bug 
    opened by funkyshu 0
  • mem backend fails backend integration test suite

    mem backend fails backend integration test suite

    running VFS_INTEGRATION_LOCATIONS="mem:///vfs_test/" go test -count=1 -tags=vfsintegration -v ./backend/testsuite/ results in:

    
    === RUN   TestVFS
    === RUN   TestVFS/TestScheme
    ************** TESTING scheme: mem **************
    ****** testing vfs.FileSystem ******
    ****** testing vfs.Location ******
    ****** testing vfs.File ******
    ** location mem:///vfs_test/dstLoc/ **
        suite.go:63: test panicked: runtime error: invalid memory address or nil pointer dereference
            goroutine 8 [running]:
            runtime/debug.Stack(0xc000189200, 0x1b89600, 0x24a04f0)
            	/Users/john.judd/.gvm/gos/go1.15.12/src/runtime/debug/stack.go:24 +0x9f
            github.com/stretchr/testify/suite.failOnPanic(0xc0003f4780)
            	/Users/john.judd/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:63 +0x57
            panic(0x1b89600, 0x24a04f0)
            	/Users/john.judd/.gvm/gos/go1.15.12/src/runtime/panic.go:969 +0x1b9
            github.com/c2fo/vfs/v5/backend/mem.(*File).MoveToLocation(0xc000440ac0, 0x1eac5c0, 0xc00079c8d0, 0x3, 0x3, 0x1, 0x1)
            	/Users/john.judd/go/src/github.com/c2fo/vfs/backend/mem/file.go:361 +0x622
            github.com/c2fo/vfs/v5/backend/testsuite.(*vfsTestSuite).File(0xc0003f1400, 0x1eac5c0, 0xc00046fc20)
            	/Users/john.judd/go/src/github.com/c2fo/vfs/backend/testsuite/backend_integration_test.go:796 +0x1aa8
            github.com/c2fo/vfs/v5/backend/testsuite.(*vfsTestSuite).TestScheme(0xc0003f1400)
            	/Users/john.judd/go/src/github.com/c2fo/vfs/backend/testsuite/backend_integration_test.go:119 +0x185
            reflect.Value.call(0xc0004772c0, 0xc000010f30, 0x13, 0x1cd89cf, 0x4, 0xc00008de30, 0x1, 0x1, 0xc00008dcf8, 0x100d88a, ...)
            	/Users/john.judd/.gvm/gos/go1.15.12/src/reflect/value.go:476 +0x8c7
            reflect.Value.Call(0xc0004772c0, 0xc000010f30, 0x13, 0xc00008de30, 0x1, 0x1, 0x3a, 0xc00007ce50, 0x1055e59)
            	/Users/john.judd/.gvm/gos/go1.15.12/src/reflect/value.go:337 +0xb9
            github.com/stretchr/testify/suite.Run.func1(0xc0003f4780)
            	/Users/john.judd/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:158 +0x379
            testing.tRunner(0xc0003f4780, 0xc000418cf0)
            	/Users/john.judd/.gvm/gos/go1.15.12/src/testing/testing.go:1123 +0xef
            created by testing.(*T).Run
            	/Users/john.judd/.gvm/gos/go1.15.12/src/testing/testing.go:1168 +0x2b3
    --- FAIL: TestVFS (0.01s)
        --- FAIL: TestVFS/TestScheme (0.00s)
    FAIL
    FAIL	github.com/c2fo/vfs/v5/backend/testsuite	0.251s
    FAIL
    
    bug 
    opened by funkyshu 0
  • Standardize options handling across providers (pointer to struct vs. pointer)

    Standardize options handling across providers (pointer to struct vs. pointer)

    Right now withOptions under file system takes an empty interface{} that can handle a pointer or a struct.

    There is no code in place to set the options from a pointer to options passed to an interface.

    Also, it seems Azure has a pointer to an Azure option item vs. using the standard interface from vfs. We should look at pros and cons here and standardize.

    These should be addressed in refactor work.

    enhancement v7 
    opened by arnab-chaudhuri-c2fo 0
  • Support for opening OS files that are not writable

    Support for opening OS files that are not writable

    Currently OS files are always opened in read/write mode. In cases where files are not writable to the process this causes problems. It would be good to have some way to accomodate read only OS files.

    enhancement 
    opened by aucampia 3
  • Add support for listing all files in a location, including files in

    Add support for listing all files in a location, including files in "subdirectories"

    Currently List() only returns files directly under a specific location. For example, with Google Cloud Storage (gs), if I have f0.txt, f1.txt, d0/f0.txt, d0/f1.txt in location loc, doing loc.List() only returns f0.txt, f1.txt, not files under d0/. It would be nice to have a way to find files with arbitrary depth.

    Possible names for the method could be Location.ListAll(), any other suggestions would be appreciated.

    enhancement v7 
    opened by aucampia 3
Releases(v6.3.0)
  • v6.3.0(May 16, 2022)

    Security

    • Update go.mod min version to Go 1.17
    • Update actions/setup-go action to v3
    • Update actions/checkout action to v3
    • Update golangci/golangci-lint-action action to v3
    • Update actions to test only supported Go versions (1.17, 1.18)
    • Updated dependencies (ie, Google Cloud, Azure, AWS sdks)
    Source code(tar.gz)
    Source code(zip)
  • v6.2.0(Apr 28, 2022)

  • v6.1.0(Feb 18, 2022)

  • v6.0.2(Dec 3, 2021)

    Fixed

    • fixed linting issues with missing godoc on exported functions and new build tag formatting.
    • fixed #92 (broken by #72) where calling ListByPrefix() was fail from non-root locations when calling file-level prefixes.
    • fixed azure helper func for vfssimple, ensuring it works on File URIs in addition to Location URIs
    • fixed #97 by updating vfssimple logic to ensure the most specific registered backend that matches a url is used, not just the first one it comes across. Updated vfssimple docs.
    • Added vfssimple tests. Zero to 100% coverage.
    • Fix codecov validation which got lost when converting to github actions. Removed .travis.yml.
    Source code(tar.gz)
    Source code(zip)
  • v6.0.1(Nov 7, 2021)

  • v6.0.0(Oct 6, 2021)

    [6.0.0] - 2021-10-06

    Changed

    • Modified sftp Key Exchange option to accept an array instead of a string, allowing multiple kex algorithms

    Upgrade steps

    With v6.0.0, sftp.Options struct changed to accept an array of Key Exchange algorithms rather than a string. To update, change the syntax of the auth commands.

    "keyExchanges":"diffie-hellman-group-a256"
    

    becomes

    "keyExchanges":["diffie-hellman-group-a256"]
    
    Source code(tar.gz)
    Source code(zip)
  • v5.10.0(Sep 16, 2021)

    Changed

    • Modified S3 file.go so that on the initial read when the remote file is downloaded, a temporary file is locally created using concurrent go routines to download parts of the file vs. a single request to download the whole object

    Fixed

    • Fixed #100 Rolled back Seek validation before Copy or Move in SFTP backend due to bug on some SFTP servers and how we cache open "file handles"
    Source code(tar.gz)
    Source code(zip)
  • v5.9.0(Sep 7, 2021)

    Added

    • Add support (and tests) for Go 1.17. Updated dependencies.
    • Updated golanci.yml config.

    Fixed

    • fixed broken backend integration test.
    Source code(tar.gz)
    Source code(zip)
  • v5.8.0(Aug 24, 2021)

    Fixed

    • fixed #82 Return error when CopyTo/MoveTo functions are called when Seek offset is not (0,0) for all backends, not just GCS.
    • fixed #84 where sftp connections were never disconnected in a long-lived app. Added Close to sftp backend Client interface. Close client automatically 10 seconds (configurable in Options) after connecting unless reset by calling some server request action.

    Deprecated

    • Deprecating utils.TouchCopy (#89).

    Added

    • Added utils.TouchCopyBuffered(#89). TouchCopyBuffered uses a default buffer size of 256KB and allows for a custom buffer size to be set with filesystem options. Improves large file transfer with negligible impact on smaller file transfers.
    Source code(tar.gz)
    Source code(zip)
  • v5.7.0(Jul 26, 2021)

  • v5.6.0(Jul 20, 2021)

    [5.6.0] - 2021-07-19

    Fixed

    • fixed #71 Listing a Google Cloud Storage bucket does not return things in the root of the bucket
    • fixed #78 Return error when CopyTo/MoveTo functions are called when Seek offset is not (0,0) for GS

    Performance

    • fixed #74 utils.TouchCopy() unnecessarily calls file.Size()
    Source code(tar.gz)
    Source code(zip)
  • v5.5.7(May 12, 2021)

    [5.5.7] - 2021-05-12

    Fixed

    • fixed: do not offset number of bytes read for memfs (#70) Jeffrey Larson [email protected]
    • fixed S3 VFS backend throws an error when you open then close a file. shivanigaurh-c2fo
    • fixed minor markdown doc formatting of type headers

    Added

    • Add Azure Blob Storage backend. Dusty McIlvain [email protected]
    • Add github actions for testing and linting (golangci-lint)
    Source code(tar.gz)
    Source code(zip)
  • 5.5.6(Feb 8, 2021)

    [5.5.6] - 2021-02-07

    Fixed

    • fixed failing test due to CI asking for mod tidy.
    • fixed ineffectual assignment failures keeping vfs from achieving 100% in goreportcard.
    • fixed #48 bug where 'invalid cross-device link' error occurs when renaming os files that are found on different devices/volumes including temp dirs.
    • fixed #53 bug where s3 file tests were failing due to a deferred removal of a temp file.

    Security

    • Upgraded dependencies. Required regen of S3 API mock due to a couple new methods. SSH returns a different error text since Go 1.14 (only affects 1 test).
    Source code(tar.gz)
    Source code(zip)
  • v5.5.5(Dec 11, 2020)

  • v5.5.4(Jul 9, 2020)

    [5.5.4] - 2020-07-06

    Fixed

    • fixes #21 - OS file.Location().ChangeDir(something/) should not affect location
    • fixes #44 - make S3 waitUntilFileExists Exists error more descriptive of root error
    • fixes #45 - fix ASCII diagram alignment for SFTP doc

    Added

    • Improved test coverage for os.FileSystem
    Source code(tar.gz)
    Source code(zip)
  • v5.5.3(May 12, 2020)

    [5.5.3] - 2020-05-11

    Fixed

    • Addresses an issue where writes/touch calls on the vfs.File backend did not work properly on AWS-hosted SFTP environments. (See https://github.com/pkg/sftp/pull/310/files)
    Source code(tar.gz)
    Source code(zip)
  • v5.5.2(Apr 29, 2020)

  • v5.5.1(Feb 10, 2020)

    [5.5.1] - 2020-02-20

    Fixed

    • Ensure that spaces (url encoded or not) in filenames and paths work as expected when copying (particularly for s3 to s3 native copyObject). Fixes #36.
    Source code(tar.gz)
    Source code(zip)
  • v5.5.0(Sep 16, 2019)

  • v5.4.0(Aug 27, 2019)

    • Added fallback to TouchCopy for S3/GCS implementations for CopyTo and MoveTo operations between files and locations which use different authentication and region configurations. When possible, the directoy copy/move operations will be used by the underlying clients.
    Source code(tar.gz)
    Source code(zip)
  • v5.3.0(Aug 25, 2019)

    Added

    • ACL can now be passed in as an s3.Option value. See https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl for values.
    Source code(tar.gz)
    Source code(zip)
  • v5.2.3(Aug 7, 2019)

    Fixed

    • The GS implementation of location.List() returned an empty string for files found inside a persistent "folder" object created in the UI or by createing a zero-length object ending with "/". These objects are now ignored.
    • Fixed a gs-to-gs copy bug that became visible with integration tests aded in v5.2.2.
    Source code(tar.gz)
    Source code(zip)
  • v5.2.2(Jul 29, 2019)

  • v5.2.1(Jul 17, 2019)

  • v5.2.0(Jul 17, 2019)

    [5.2.0] - 2019-07-16

    Added

    • In-Memory backend vfs implementation
    • Included the in-memory backend to the list of registered backends used in backend_integration_test
    • Checking for existence at the start of various File functions (size, seek, etc) and returning and error if the file does not exist on the OS backend
    • Tests in backend_integration_test that ensure that operations done on non-existent files throw an error
    • mem.md in vfs/docs, updated link to it in the README.md

    Fixed

    • Relative path validation in utils.go now disallows empty names

    Changed

    • utils_test.go now expects an empty string NOT to validate
    • updated README to include "Touch()" definition under the File interface rather than the Location interface
    • updated README to exclude "in-memory-backend" from the list of ideas
    Source code(tar.gz)
    Source code(zip)
  • v5.1.0(Jul 9, 2019)

    Added Touch() Refactored for consistency between backends Updated docs. Added more tests. Added integration test suite that tests every implementation against interface expectations and against each other.

    Source code(tar.gz)
    Source code(zip)
  • v4.0.0(May 23, 2019)

    Release Notes

    1. Added the Retry() method to the FileSystem interface, along with a no-op default retryer.
    2. Integrated the newly defined retryer interface into the GCS implementation of VFS and now allow the retry method to be injected as a FileSystem option.
    3. Integrated the request.Retryer from the s3 client as an injectable implementation for the S3 vfs.Option.

    Upgrade steps

    With v4.0.0, we introduced the ability to add a 'retry' option to each registered file system. This version bump will require an update to any existing FileSystem implementations. If a custom retryer is not required by the implemented file system, the default retryer can be returned.

    func (fs *FileSystem) Retry() vfs.Retry {
    	return vfs.DefaultRetryer()
    }
    
    Source code(tar.gz)
    Source code(zip)
  • v2.1.4(Apr 5, 2019)

  • v2.1.3(Mar 19, 2019)

    Fixed issue where there is an extraneous leading slash when calling location.Path() for GCS locations. Improved the discoverability of registered filesystems at the location-level.

    Source code(tar.gz)
    Source code(zip)
Owner
C2FO
C2FO
Fast extensible file name sanitizer that works in Windows/Linux

Sanity Sanity is a fast and easily extensible file name (and in fact any other string) sanitizer. Usage Built-in rule set Sanity provides a sensible d

null 2 Jun 8, 2022
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).

中文 English 愿景:为用户提供最简单、可靠、高效的分布式文件系统。 go-fastdfs是一个基于http协议的分布式文件系统,它基于大道至简的设计理念,一切从简设计,使得它的运维及扩展变得更加简单,它具有高性能、高可靠、无中心、免维护等优点。 大家担心的是这么简单的文件系统,靠不靠谱,可不

小张 3.2k Jun 30, 2022
A Small Virtual Filesystem in Go

This is a virtual filesystem I'm coding to teach myself Go in a fun way. I'm documenting it with a collection of Medium posts that you can find here.

Alyson 31 Apr 18, 2022
Bigfile -- a file transfer system that supports http, rpc and ftp protocol https://bigfile.site

Bigfile ———— a file transfer system that supports http, rpc and ftp protocol 简体中文 ∙ English Bigfile is a file transfer system, supports http, ftp and

null 227 Jun 30, 2022
File system event notification library on steroids.

notify Filesystem event notification library on steroids. (under active development) Documentation godoc.org/github.com/rjeczalik/notify Installation

Rafal Jeczalik 743 Jun 12, 2022
Cross-platform file system notifications for Go.

File system notifications for Go fsnotify utilizes golang.org/x/sys rather than syscall from the standard library. Ensure you have the latest version

fsnotify 7k Jun 22, 2022
Dragonfly is an intelligent P2P based image and file distribution system.

Dragonfly Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in o

dragonflyoss 5.8k Jun 30, 2022
Plik is a scalable & friendly temporary file upload system ( wetransfer like ) in golang.

Want to chat with us ? Telegram channel : https://t.me/plik_root_gg Plik Plik is a scalable & friendly temporary file upload system ( wetransfer like

root.gg 1k Jun 28, 2022
File system for GitHub

HUBFS · File System for GitHub HUBFS is a read-only file system for GitHub and Git. Git repositories and their contents are represented as regular dir

Bill Zissimopoulos 1.5k Jun 25, 2022
GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go

GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go Overview GeeseFS allows you to mount an S3 bucket as a file sys

Yandex.Cloud 258 Jun 21, 2022
Encrypted File System in Go

Getting Started: Setup the environment: Install GoLang: $ sudo apt update $ sudo apt upgrade $ sudo apt install libssl-dev gcc pkg-config $ sudo apt

Lucky Verma 0 Apr 30, 2022
A rudimentary go program that allows you to mount a mongo database as a FUSE file system

This is a rudimentary go program that allows you to mount a mongo database as a

Jay Goel 1 Dec 29, 2021
Gokrazy mkfs: a program to create an ext4 file system on the gokrazy perm partition

gokrazy mkfs This program is intended to be run on gokrazy only, where it will c

null 4 Jun 13, 2022
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Steve Francia 4.5k Jun 27, 2022
Experimental typesetting system

ETS Experimental typesetting system This software repository contains a Lua frontend for the typesetting library “Boxes and Glue” which is an algorith

speedata GmbH 4 Jan 30, 2022
Go-lang based sonos standup system

Overview This is an CLI tool that can handle timed standup playback on a sonos device. It allows you to add links to audio files that will be randomly

null 1 Nov 23, 2021
Abstract File Storage

afs - abstract file storage Please refer to CHANGELOG.md if you encounter breaking changes. Motivation Introduction Usage Matchers Content modifiers S

Viant, Inc 186 Jun 19, 2022
a tool for handling file uploads simple

baraka a tool for handling file uploads for http servers makes it easier to make operations with files from the http request. Contents Install Simple

Enes Furkan Olcay 43 Jun 4, 2022