s3fs provides a S3 implementation for Go1.16 filesystem interface.

Overview

s3fs Go Reference Go

Package s3fs provides a S3 implementation for Go1.16 filesystem interface.

Since S3 is a flat structure, s3fs simulates directories by using prefixes and "/" delim. ModTime on directories is always zero value.

The implementation wraps aws sdk go s3 client.

const bucket = "my-bucket"

s, err := session.NewSession()
if err != nil {
    log.Fatal(err)
}

s3fs := s3fs.New(s3.New(s), bucket)

// print out all files in s3 bucket.
_ = fs.WalkDir(s3fs, ".", func(path string, d fs.DirEntry, err error) error {
    if err != nil {
        return err
    }

    if d.IsDir() {
        fmt.Println("dir:", path)
        return nil
    }
    fmt.Println("file:", path)
    return nil
})

Installation

go get github.com/jszwec/s3fs

Requirements

  • Go1.16+
Issues
  • stat fails for existing directory

    stat fails for existing directory

    I believe I've identified a bug that causes Stat() to fail in certain circumstances. Given two objects, a/b.txt and a.txt, a call to fs.Stat(fsys, "a") will fail: open a: file does not exist. I believe this is because the MaxKeys budget (of 1) in the call to ListObjects is used-up by a.txt in the response Contents (link).

    I'll submit a failing test in a PR

    bug 
    opened by srerickson 2
  • Switch to v2 of the aws sdk

    Switch to v2 of the aws sdk

    Amazon has a version 2 of the api which is already the official predecessor of v1.

    The main advantages of the new API are performance improvements, easier error handling, easier credentials handling, ...

    https://aws.amazon.com/blogs/developer/client-updates-in-the-preview-version-of-the-aws-sdk-for-go-v2/

    opened by guenhter 15
Releases(release-1.45.1)
Owner
Jacek Szwec
Jacek Szwec
Read a tar file contents using go1.16 io/fs abstraction

go-tarfs Read a tar file contents using go1.16 io/fs abstraction Usage ⚠️ go-tarfs needs go>=1.16 Install: go get github.com/nlepage/go-tarfs Use: pac

Nicolas Lepage 17 Mar 26, 2022
A Go io/fs filesystem implementation for reading files in a Github gists.

GistFS GistFS is an io/fs implementation that enables to read files stored in a given Gist. Requirements This module depends on io/fs which is only av

Jean Hadrien Chabran 124 May 1, 2022
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Steve Francia 4.5k Jun 27, 2022
A package to allow one to concurrently go through a filesystem with ease

skywalker Skywalker is a package to allow one to concurrently go through a filesystem with ease. Features Concurrency BlackList filtering WhiteList fi

Will Dixon 80 Jun 18, 2022
Takes an input http.FileSystem (likely at go generate time) and generates Go code that statically implements it.

vfsgen Package vfsgen takes an http.FileSystem (likely at go generate time) and generates Go code that statically implements the provided http.FileSys

null 945 May 31, 2022
memfs: A simple in-memory io/fs.FS filesystem

memfs: A simple in-memory io/fs.FS filesystem memfs is an in-memory implementation of Go's io/fs.FS interface. The goal is to make it easy and quick t

Peter Sanford 64 Jun 21, 2022
A Small Virtual Filesystem in Go

This is a virtual filesystem I'm coding to teach myself Go in a fun way. I'm documenting it with a collection of Medium posts that you can find here.

Alyson 31 Apr 18, 2022
CRFS: Container Registry Filesystem

CRFS: Container Registry Filesystem Discussion: https://github.com/golang/go/issues/30829 Overview CRFS is a read-only FUSE filesystem that lets you m

Google 1.2k Jun 14, 2022
Encrypted overlay filesystem written in Go

An encrypted overlay filesystem written in Go. Official website: https://nuetzlich.net/gocryptfs (markdown source). gocryptfs is built on top the exce

null 2.3k Jun 22, 2022
Go filesystem implementations for various URL schemes

hairyhenderson/go-fsimpl This module contains a collection of Go filesystem implementations that can discovered dynamically by URL scheme. All filesys

Dave Henderson 225 Jun 27, 2022
A Go filesystem package for working with files and directories

Stowage A Go filesystem package for working with files and directories, it features a simple API with support for the common files and directories ope

null 19 May 28, 2021
filesystem for golang

filesystem filesystem for golang installation go get github.com/go-component/filesystem import import "github.com/go-component/filesystem" Usage sup

null 4 Jul 9, 2021
A set of io/fs filesystem abstractions and utilities for Go

A set of io/fs filesystem abstractions and utilities for Go Please ⭐ this project Overview This package provides io/fs interfaces for: Cloud providers

null 8 Mar 24, 2022
Tarserv serves streaming tar files from filesystem snapshots.

tarserv A collection of tools that allow serving large datasets from local filesystem snapshots. It is meant for serving big amounts of data to shell

Aurora 1 Jan 11, 2022
Grep archive search in any files on the filesystem, in archive and even inner archive.

grep-archive Grep archive search for string in any files on the filesystem, in archive and even inner archive. Supported archive format are : Tar Form

Michel Prunet 0 Jan 26, 2022
Warp across your filesystem in ~5 ms

WarpDrive: the Go version. What does this do? Instead of having a huge cd routine to get where you want, with WarpDrive you use short keywords to warp

Ishan Goel 17 Jun 13, 2022
A go compress library for fs.FS interface

compress: a go compress library for fs.FS interface Format Test Charset Decoder Encoder Password Info zip local true true true false used go std rar l

null 1 Apr 16, 2022
Go implementation of the Open Packaging Conventions (OPC)

opc Package opc implements the ISO/IEC 29500-2, also known as the Open Packaging Convention. The Open Packaging specification describes an abstract mo

Quim Muntal 72 May 8, 2022
A test repo to demonstrate the current (go1.17.2) issue when trying to use retractA test repo to demonstrate the current (go1.17.2) issue when trying to use retract

test-go-mod-retract This is a test repo to demonstrate the current (go1.17.2) issue when trying to use retract in go.mod to retract a version in a non

Yuxuan 'fishy' Wang 0 Oct 16, 2021
An implementation of the FileSystem interface for tar files.

TarFS A wrapper around tar.Reader. Implements the FileSystem interface for tar files. Adds an Open method, that enables reading of file according to i

Eyal Posener 50 May 17, 2022
Package gorilla/sessions provides cookie and filesystem sessions and infrastructure for custom session backends.

sessions gorilla/sessions provides cookie and filesystem sessions and infrastructure for custom session backends. The key features are: Simple API: us

Gorilla Web Toolkit 2.3k Jun 28, 2022
Some helper types for go1: priority queue, slice wrapper.

go-villa Package villa contains some helper types for Go: priority queue, slice wrapper, binary-search, merge-sort. GoDoc Link: http://godoc.org/githu

Yi Deng 53 Apr 24, 2021
Read a tar file contents using go1.16 io/fs abstraction

go-tarfs Read a tar file contents using go1.16 io/fs abstraction Usage ⚠️ go-tarfs needs go>=1.16 Install: go get github.com/nlepage/go-tarfs Use: pac

Nicolas Lepage 17 Mar 26, 2022
YSHOP-GO基于当前流行技术组合的前后端RBAC管理系统:Go1.15.x+Beego2.x+Jwt+Redis+Mysql8+Vue 的前后端分离系统,权限控制采用 RBAC,支持数据字典与数据权限管理,支持动态路由等

YSHOP-GO 后台管理系统 项目简介 YSHOP-GO基于当前流行技术组合的前后端RBAC管理系统:Go1.15.x+Beego2.x+Jwt+Redis+Mysql8+Vue 的前后端分离系统,权限控制采用 RBAC,支持数据字典与数据权限管理,支持动态路由等 体验地址: https://go

null 102 Jun 23, 2022
beego-vue-admin基于当前流行技术组合的前后端RBAC管理系统:Go1.15.x+Beego2.x+Jwt+Redis+Mysql8+Vue 的前后端分离系统,权限控制采用 RBAC,支持数据字典与数据权限管理,支持动态路由等

beego-vue-admin 后台管理系统 项目简介 beego-vue-admin基于当前流行技术组合的前后端RBAC管理系统:Go1.15.x+Beego2.x+Jwt+Redis+Mysql8+Vue 的前后端分离系统,权限控制采用 RBAC,支持数据字典与数据权限管理,支持动态路由等 体验

null 102 Jun 23, 2022
experimental promises in go1.18 with generics

async go a prototype of "promises" in go1.18. note: this is just an experiment used to test alternate patterns for dealing with asynchronous code in g

nick comer 55 May 26, 2022
A go1.18 wrapper to provide simple generics based API for defining command line flags.

gflag A go1.18 wrapper to provide simple generics based API for defining command line flags. Example package main import ( "flag" "fmt" "time" "

Clifton Kaznocha 0 Dec 20, 2021
A go1.18+ package to (maybe) simplify performing operations on slices in a fluent-like style.

sop ✨ W.I.P. ✨ sop (slices operation) is a go1.18+ package to (maybe) simplify performing operations on slices in a fluent-like style with common oper

Ringo Hoffmann 3 Mar 17, 2022
Go.work-workspace-example - Go1.18 workspace example

Go.work-workspace-example - Go1.18 workspace example

Andrey Burov 0 Jan 20, 2022