a high-performance, POSIX-ish Amazon S3 file system written in Go

Overview

Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go

Build Status Github All Releases Twitter Follow

Overview

Goofys allows you to mount an S3 bucket as a filey system.

It's a Filey System instead of a File System because goofys strives for performance first and POSIX second. Particularly things that are difficult to support on S3 or would translate into more than one round-trip would either fail (random writes) or faked (no per-file permission). Goofys does not have an on disk data cache (checkout catfs), and consistency model is close-to-open.

Installation

$ brew cask install osxfuse
$ brew install goofys
  • Or build from source with Go 1.10 or later:
$ export GOPATH=$HOME/work
$ go get github.com/kahing/goofys
$ go install github.com/kahing/goofys

Usage

$ cat ~/.aws/credentials
[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY
$ $GOPATH/bin/goofys <bucket> <mountpoint>
$ $GOPATH/bin/goofys <bucket:prefix> <mountpoint> # if you only want to mount objects under a prefix

Users can also configure credentials via the AWS CLI or the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

To mount an S3 bucket on startup, make sure the credential is configured for root, and can add this to /etc/fstab:

goofys#bucket   /mnt/mountpoint        fuse     _netdev,allow_other,--file-mode=0666,--dir-mode=0777    0       0

See also: Instruction for Azure Blob Storage, Azure Data Lake Gen1, and Azure Data Lake Gen2.

Got more questions? Check out questions other people asked

Benchmark

Using --stat-cache-ttl 1s --type-cache-ttl 1s for goofys -ostat_cache_expire=1 for s3fs to simulate cold runs. Detail for the benchmark can be found in bench.sh. Raw data is available as well. The test was run on an EC2 m5.4xlarge in us-west-2a connected to a bucket in us-west-2. Units are seconds.

Benchmark result

To run the benchmark, configure EC2's instance role to be able to write to $TESTBUCKET, and then do:

$ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v /tmp/cache:/tmp/cache kahing/goofys-bench
# result will be written to $TESTBUCKET

See also: cached benchmark result and result on Azure.

License

Copyright (C) 2015 - 2019 Ka-Hing Cheung

Licensed under the Apache License, Version 2.0

Current Status

goofys has been tested under Linux and macOS.

List of non-POSIX behaviors/limitations:

  • only sequential writes supported
  • does not store file mode/owner/group
    • use --(dir|file)-mode or --(uid|gid) options
  • does not support symlink or hardlink
  • ctime, atime is always the same as mtime
  • cannot rename directories with more than 1000 children
  • unlink returns success even if file is not present
  • fsync is ignored, files are only flushed on close

In addition to the items above, the following are supportable but not yet implemented:

  • creating files larger than 1TB

Compatibility with non-AWS S3

goofys has been tested with the following non-AWS S3 providers:

  • Amplidata / WD ActiveScale
  • Ceph (ex: Digital Ocean Spaces, DreamObjects, gridscale)
  • EdgeFS
  • EMC Atmos
  • Google Cloud Storage
  • Minio (limited)
  • OpenStack Swift
  • S3Proxy
  • Wasabi

Additionally, goofys also works with the following non-S3 object stores:

  • Azure Blob Storage
  • Azure Data Lake Gen1
  • Azure Data Lake Gen2

References

Comments
  • Goofys consumes all memory with extended concurrent transfers

    Goofys consumes all memory with extended concurrent transfers

    I'm using goofys to expose test data on s3 to an ec2 instance.

    Out load test script runs continuous jobs with the s3 transfer rate being the limiting factor. There are 5 concurrent transfers gong at a time. Data file size is often 100-400MB.

    One test run completed successfully, then a subsequent one failed as the s3 data transfers quit working.

    From syslog: (trimmed for previty)

    Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289878] slurmctld invoked oom-killer: gfp_mask=0x3000d0, order=1, oom_score_adj=0 Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290013] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290064] [11362] 0 11362 3653986 3494764 6877 0 0 goofys

    Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290106] Out of memory: Kill process 11362 (goofys) score 826 or sacrifice child Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.295024] Killed process 11362 (goofys) total-vm:14615944kB, anon-rss:13979056kB, file-rss:0kB

    Is there any log/debug flag I can set to gather data from goofys?

    Full log here: https://gist.github.com/6b9ee07c1b8c0b5ebeca

    The current status shows that it's still mounted, but it's not working:

    [email protected]:/var/log# ls /content ls: cannot access /content: Transport endpoint is not connected [email protected]:/var/log# mount | grep content s3fs on /content-s3fs type fuse.s3fs (rw,nosuid,nodev,allow_other) v4test.demo.nextissue.com on /content-goofys type fuse (rw,nosuid,nodev,default_permissions,allow_other)

    Remounting the bucket resolved the hangup issue. From logs after umount/mount:

    Mar 7 22:50:39 ip-10-33-48-7 /usr/local/bin/goofys[1467]: s3.INFO Switching from region 'us-west-2' to 'us-east-1' Mar 7 22:50:39 ip-10-33-48-7 /usr/local/bin/goofys[1467]: main.INFO File system has been successfully mounted.

    bug Need Info 
    opened by bedge 35
  • Mount from fstab runs in foreground

    Mount from fstab runs in foreground

    used goofys#bucketname /mnt/mountpoint fuse allow_other,--file-mode=0777,--dir-mode=0777,--uid="33,--gid="33",--storage-class="REDUCED_REDUNDANCY" in /etc/fstab.

    sudo mount /mnt/mountpoint mounted the volume with the following messages left open in foreground:

    2015/11/07 21:34:10.777609 s3.INFO Switching from region 'us-west-2' to 'us-east-1' 2015/11/07 21:34:10.816434 main.INFO File system has been successfully mounted.

    CTRL-C unmounted the volume.

    opened by greerso 29
  • disk cache

    disk cache

    You write that goofys does not have a disk cache. Is that because of a design decision or has it just not been implemented yet. Would you possibly accept a patch that implements it?

    I'm asking because we are currently using s3fs but we have run into stability problem and would perhaps use an alternative. Some kind of cache is an absolute requirement, though.

    Have you thought about goofys => caching http proxy => s3 yet as an alternative to a disk cache? Has this been tried by anybody, yet?

    opened by ghaering 24
  • About get object with Range

    About get object with Range

    Hello, We are using goofys with very big file : ~ 20 G We are using mmap() on this file and then we do a lot of memcpy of block < 128K. We are observing a lot of big range request on S3 bucket, ~ 5M compare to 128K maximum. Can you explain how get object with http range are done with goofys ?

    Regards, Nicolas Prochazka

    opened by limbo127 22
  • index out of range panic.

    index out of range panic.

    I have some long running processing that are (ab)using goofys with a lot of seeks followed by reads. I ran with both debug flags and at the end of megabytes of output is the traceback below. this is with v0.0.12. Let me know if you need the full output. I'm pulling from cloudwatch logs so it's easier to get the tail.

    2017/06/19 18:28:58.654265 fuse.DEBUG Op 0x00012145 connection.go:479] -> OK ()
    2017/06/19 18:28:58.654184 fuse.DEBUG < ReadFile 78 sarcoma/764-SS-767/764-SS-767.bam [0 <nil>]
    2017/06/19 18:28:58.654830 fuse.DEBUG ReadFile 78 sarcoma/764-SS-767/764-SS-767.bam [4690632704 65536]
    2017/06/19 18:28:58.654848 fuse.DEBUG out of order read 78 sarcoma/764-SS-767/764-SS-767.bam [4690632704 4690890752]
    panic: runtime error: index out of range
    goroutine 133887 [running]:
    github.com/kahing/goofys/internal.(*MBuf).WriteFrom(0xc43ec47000, 0x7f0c7baab8f8, 0xc43c2ca980, 0x7f0c7baab8f8, 0xc43c2ca980, 0x1)
    created by github.com/kahing/goofys/internal.Buffer.Init
    /home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:312 +0xff
    /home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:264 +0x1ce
    github.com/kahing/goofys/internal.Buffer.Init.func1(0xc43ec47040, 0xc48c2c0e40)
    /home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:311 +0x35
    /home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:334 +0x99
    github.com/kahing/goofys/internal.(*Buffer).readLoop(0xc43ec47040, 0xc48c2c0e40)
    github.com/kahing/goofys/internal.Buffer.Init.func1(0xc43ec47100, 0xc48c2c0f00)
    goroutine 133888 [running]:
    github.com/kahing/goofys/internal.(*MBuf).WriteFrom(0xc43ec470c0, 0x7f0c7baab8f8, 0xc42679fce0, 0x7f0c7baab8f8, 0xc42679fce0, 0x1)
    /home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:312 +0xff
    github.com/kahing/goofys/internal.(*Buffer).readLoop(0xc43ec47100, 0xc48c2c0f00)
    /home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:334 +0x99
    panic: runtime error: index out of range
    /home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:311 +0x35
    created by github.com/kahing/goofys/internal.Buffer.Init
    /home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:264 +0x1ce
    
    opened by brentp 21
  • Is this a bug in Goofys or FUSE?

    Is this a bug in Goofys or FUSE?

    I'm having a problem with some of my processes that are using S3 files on a goofys mount.

    I have also tried S3FS and experience the exact same problem. I'm not yet sure if this is a Goofys/S3FS problem or a FUSE problem. I'm hoping you can help.

    We have an S3 bucket with many large files (1000's of files. 200->800mb each).

    We have a multithreaded process that:

    • Untar one of the tgzs to /tmp on the local filesystem.
    • Parses the the text files inside.

    So our use case for goofys is pure read.

    Several times a day one of these processes will get stuck in the untar and hang forever.

    ps shows a process in state "D" [[email protected] ~]$ ps aux | grep xlog_create dashv 23394 0.0 0.0 12136 956 ? D 01:29 0:10 /opt/dashv/bin/processfile /mnt/s3/20170221-000203.tar.gz [[email protected] ~]$

    State D means that the process is uninterruptable. So "kill -9" will not work (in fact no kill command will work).

    I tried to cat the status:

    [[email protected] ~]$ cat /proc/23394/status Name: processfile State: D (disk sleep) Tgid: 23394 Ngid: 0 Pid: 23394 PPid: 23393 TracerPid: 15471 Uid: 2031 2031 2031 2031 Gid: 502 502 502 502 FDSize: 64 Groups: 502 NStgid: 23394 NSpid: 23394 NSpgid: 22855 NSsid: 22855 VmPeak: 12136 kB VmSize: 12136 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 956 kB VmRSS: 956 kB VmData: 224 kB VmStk: 136 kB VmExe: 12 kB VmLib: 3512 kB VmPTE: 48 kB VmPMD: 12 kB VmSwap: 0 kB HugetlbPages: 0 kB Threads: 1 SigQ: 1/64117 SigPnd: 0000000000040100 ShdPnd: 0000000000000100 SigBlk: 0000000000000000 SigIgn: 0000000001001000 SigCgt: 0000000180004007 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 0000003fffffffff CapAmb: 0000000000000000 Seccomp: 0 Cpus_allowed: 7fff Cpus_allowed_list: 0-14 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 3386 nonvoluntary_ctxt_switches: 86 [[email protected] ~]$

    The important part is at the very top and at the bottom: State D is "Disk Sleep". So I have confirmed it's IO (FUSE) voluntary_ctxt_switches and nonvoluntary_ctxt_switches crudely indicate how many times the process has been interrupted. I ran a couple of checks a few minutes apart and the number for each never changed. This verifies the process is stuck and not being interrupted. Usually uninterruptable code means it's a kernel system call. So I confirmed:

    [[email protected] ~]$ sudo cat /proc/23394/syscall 0 0x6 0x7fffd74cca30 0x4000 0x0 0x1000 0x7fffd74cca30 0x7fffd74bc968 0x7f71f4193bb0:

    The first number (0) is the number of the system call. Looking it up here: http://blog.rchapman.org/posts/Linux_System_Call_Table_for_x86_64/ I see that it's "sys_read".

    Not helpful by itself. I need to know what's calling it. So I took a look at the stack for the process:

    [[email protected] ~]$ sudo cat /proc/23394/stack [] request_wait_answer+0xf8/0x250 [fuse] [] __fuse_request_send+0x67/0x90 [fuse] [] fuse_request_send+0x27/0x30 [fuse] [] fuse_send_readpages.isra.30+0xd2/0x120 [fuse] [] fuse_readpages+0xdf/0x100 [fuse] [] __do_page_cache_readahead+0x174/0x200 [] ondemand_readahead+0x135/0x260 [] page_cache_async_readahead+0x6c/0x70 [] generic_file_read_iter+0x378/0x590 [] fuse_file_read_iter+0x4c/0x70 [fuse] [] __vfs_read+0xa7/0xd0 [] vfs_read+0x7f/0x130 [] SyS_read+0x46/0xa0 [] entry_SYSCALL_64_fastpath+0x12/0x71 [] 0xffffffffffffffff [[email protected] ~]$

    It looks like a fuse call to: request_wait_answer is never returning when trying to populate the read-ahead-cache. Googling it I see this is can happen when using a filesystem that relies on networking.

    Both Goofys and S3FS share this exact same problem so either they have the same bug (Shouldn't this scenario be handled as a page fault?) OR its a problem with FUSE.

    I'll openly admit I'm in a bit over my head here. Hoping someone with more knowledge of this can chime in and help lead me to a resolution.

    Otherwise I'm dead in the water.

    Need Info 
    opened by xavierpayne 20
  • Why is the speed so slow?

    Why is the speed so slow?

    I use goofys , by following command:

    1. download the executable file (goofys), then *chmod 775 goofys*

    2. cat ~/.aws/credentials [default] aws_access_key_id = *********** aws_secret_access_key = ********

    3. ./goofys --endpoint ******** --region *****

    mount an S3 bucket successful ,But upload speed is only 7MB/s 。 I do not kown this why 。 Is thers any thing package ,I need install ? image

    Last Time,I use goofys ,it's speed is 300MB/s ,which Cloud host‘s Memory is 32G 。 image

    opened by XiaoGe2030 18
  • bucket mount prefix doesn't work if prefixed with slash

    bucket mount prefix doesn't work if prefixed with slash

    I am using CMS to upload, copy and add files into the mounted directory, say /asset/files/. s3fs has no issue doing these jobs, but goofys can create files in S3 only when upload. When I try to copy a file inside /asset/files/, it creates nothing in the new destination. I used --debug_fuse to check the issue. I found that when I try to copy a file, no action of creating file is recorded in the debug log. What else I could do to tackle this? Thanks.

    opened by karltdev 18
  • Flush() doesn't block until its done, and other IO ops can execute in between

    Flush() doesn't block until its done, and other IO ops can execute in between

    Hi Team,

    looks like i have many errors and not able to figure out further.

    [[email protected] bin]# grep "no such file or directory" /var/log/messages | wc -l 12844

    Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df703 connection.go:476] -> Error: "invalid argument" Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df704 connection.go:395] <- GetInodeAttributes (inode 72157) Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df704 connection.go:474] -> OK Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df705 connection.go:395] <- OpenFile (inode 72157) Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df705 connection.go:474] -> OK Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df706 connection.go:395] <- ReadFile (inode 72157, handle 92893, offset 0, 4096 bytes) Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df706 connection.go:476] -> Error: "no such file or directory" Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df707 connection.go:395] <- FlushFile (inode 72157) Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df707 connection.go:474] -> OK Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df708 connection.go:395] <- ReadFile (inode 72157, handle 92893, offset 0, 4096 bytes) Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df708 connection.go:476] -> Error: "no such file or directory" Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df709 connection.go:395] <- ReleaseFileHandle Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df709 connection.go:474] -> OK Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df70a connection.go:395] <- LookUpInode (parent 936, name "6713_sku_1280x720Padded_9caf092d7a7dd2466a5c8c215d006561.jpg") Mar 15 10:34:34 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df70a connection.go:476] -> Error: "no such file or directory" Mar 15 10:34:34 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df70b connection.go:395] <- FlushFile (inode 72157) Mar 15 10:34:34 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df70b connection.go:476] -> Error: "invalid argument" Mar 15 10:34:34 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df70c connection.go:395] <- GetInodeAttributes (inode 72157) Mar 15 10:34:34 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df70c connection.go:474] -> OK Mar 15 10:34:34 img01 rsyslogd-2177: imuxsock begins to drop messages from pid 26093 due to rate-limiting Mar 15 10:34:40 img01 rsyslogd-2177: imuxsock lost 322 messages from pid 26093 due to rate-limiting Mar 15 10:34:40 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df750 connection.go:395] <- LookUpInode (parent 16, name "2") Mar 15 10:34:40 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df750 connection.go:474] -> OK Mar 15 10:34:40 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df751 connection.go:395] <- LookUpInode (parent 275, name "3") Mar 15 10:34:40 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df751 connection.go:474] -> OK Mar 15 10:34:40 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df752 connection.go:395] <- LookUpInode (parent 278, name "c") Mar 15 10:34:41 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df752 connection.go:474] -> OK Mar 15 10:34:41 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df753 connection.go:395] <- LookUpInode (parent 43060, name "6713_sku_ATOM_SIZE_MACRO_23c6d55b8d500f3bac05c71835198cea.jpg") Mar 15 10:34:41 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df753 connection.go:476] -> Error: "no such file or directory"

    [[email protected] bin]# ./goofys --version goofys version 0.0.5

    /etc/fstab: goofys#size-img01 /img01 fuse _netdev,allow_other,--region='ap-southeast-1',--file-mode=0666,--debug_fuse,--debug_s3,--storage-class="REDUCED_REDUNDANCY" 0 0

    Please help me to see if anything i need to make changes.

    Thanks, Vadiraj

    bug Need Info 
    opened by vadirajks 18
  • fstab ignores allow_other

    fstab ignores allow_other

    Here is my fstab entry:

    `goofys#cpirepsarchive   /mnt/cpirepsarchive.s3 fuse    allow_other,--file-mode=0777,--dir-mode=0777,--uid="33,--gid="33",--storage-class="REDUCED_REDUNDANCY"    0       0`
    

    Here is the strace:

    `[email protected]:~$ sudo strace -f mount /mnt/mountpoint >& xout
    ^C
    [email protected]:~$ grep exec xout
    execve("/bin/mount", ["mount", "/mnt/mountpoint"], [/* 15 vars */]) = 0
    [pid  6240] execve("/sbin/mount.fuse", ["/sbin/mount.fuse", "goofys#bucketname", "/mnt/mountpoint", "-o", "rw,allow_other,--file-mode=0777,"...], [/* 11 vars */]) = 0
    [pid  6240] execve("/bin/sh", ["/bin/sh", "-c", "'goofys' 'bucketname' '/mnt/"...], [/* 12 vars */]) = 0
    [pid  6241] execve("/usr/bin/goofys", ["goofys", "bucketname", "/mnt/mountpoint", "-o", "rw,allow_other,--file-mode=0777,"...], [/* 13 vars */]) = 0
    [pid  6244] execve("/home/ubuntu/.gvm/pkgsets/go1.5.1/global/bin/goofys", ["/home/ubuntu/.gvm/pkgsets/go1.5."..., "bucketname", "/mnt/mountpoint", "-o", "rw,allow_other,--file-mode=0777,"...], [/* 14 vars */]) = 0
    [pid  6250] execve("/bin/fusermount", ["fusermount", "-o", "default_permissions,fsname=cpire"..., "--", "/mnt/mountpoint"], [/* 16 vars */] <unfinished ...>
    [pid  6250] <... execve resumed> )      = 0
    [pid  6253] execve("/bin/mount", ["/bin/mount", "--no-canonicalize", "-i", "-f", "-t", "fuse", "-o", "rw,default_permissions,allow_oth"..., "bucketname", "/mnt/mountpoint"], [/* 0 vars */]) = 0`
    
    opened by greerso 17
  • Add support to Assume an IAM Role with an External ID

    Add support to Assume an IAM Role with an External ID

    Hi All,

    It would be great if goofys could assume a role to get access to a bucket owned by a third party: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html

    The Role ARN to be assumed and an External ID should be provided to the tool to mount the S3 bucket. To do this the STS API must be used to get the temporary security credentials. This is an example with Go:

    https://docs.aws.amazon.com/sdk-for-go/api/service/sts/#example_STS_AssumeRole_shared00

    help wanted Need Info 
    opened by aquexata 16
  • Syntax Error

    Syntax Error

    Hi, on a new Debian 11 virtual machine, I installed Goofys to connect to object storage in Contabo. I already connected this storage to another server, works perfectly. I have setup everything. Here is the error: /usr/local/bin/goofys: 3: Syntax error: ")" unexpected

    Any help ? Thanks

    opened by docles2000 0
  • Fix potential memory consumption and OOMs by fixing logic in `getCgroupAvailableMem`

    Fix potential memory consumption and OOMs by fixing logic in `getCgroupAvailableMem`

    os.Stat is never returning err which satisfies os.IsExist(err) == true the only way to make sure that file exists using os.Stat is to check err == nil

    if _, err := os.Stat("/file/that/exists"); os.IsExist(err) {
      // will never trigger!
    
      // why? because os.Stat runs normally if file exists.
      // it's expected behaviour for os.Stat, so it doesn't throw an error
    
      // os.IsExist() does not receive an error ( it's nil ), so it can't tell you
      // if the error message was "file not found"
    }
    

    from https://pkg.go.dev/os#File.Stat

    Stat returns the FileInfo structure describing file. If there is an error, it will be of type *PathError.
    

    also I installed go mods from scratch and it required additional method for FusePanicLogger to satisfy fuseutil.FileSystem I added the method

    opened by wheredyoufindthis 0
  • Stop file read after passing the Unencrypted length

    Stop file read after passing the Unencrypted length

    Is it possible to "stop" the file read after reading the: "Unencrypted length" of bytes, which will be usually less than the encrypted length as presented in the "ls" / stat.

    There are many client-side AES-256 implementations. The S3 stored binary is rounded up to the block size, whereas the real content will be usually shorter. The info about unencrypted length is often stored in the metadata in e.g. x-amz-unencrypted-content-length and is available after headObject or getObject. Unencrypted length isn't available via standard listing methods e.g. listObjectsV2.

    I am experimenting with Goofys and modified the: internal/backend_s3.go:GetBlob() so it can decrypt the content, however if I don't pad the last block (so it matches the "stat" size) the GetBlob seem to be called again (until it returns the amount of data to satisfy the size advertised in the ls)

    I don't know the Goofys/FUSE and file system internals and I am not sure who requests (during file read process) file contents up to the last byte. Would there be possibility to make Goofys/FUSE/filesystem aware during the execution (technically after first getObject response received with metadata) that file length is actually different then the one reported to ls, so we can return the file exactly as it is, instead of trying some padding tricks? Perhaps there is some EOF character I could return?

    Sorry for my ignorance, but file systems and Go are entirely new things to me from the dev perspective.

    opened by tomekit 0
  • Failed to compile from source neither on arm64 or x86

    Failed to compile from source neither on arm64 or x86

    Hi,

    I am trying to compile the goofys on Graviton2, as there are no arm64 package available. However when I follow the steps to build from source, following error messages shown up

    github.com/kahing/goofys/api

    api/api.go:124:56: cannot use common.FusePanicLogger{...} (type common.FusePanicLogger) as type fuseutil.FileSystem in argument to fuseutil.NewFileSystemServer: common.FusePanicLogger does not implement fuseutil.FileSystem (missing BatchForget method)

    Then I try to use x86 instance, but failed as well, with same error message

    I also have tried different go version including 1.17, 1.18, 1.19 but no lucky

    Please help to verify this

    Thanks, Vincent

    opened by ddynwzh1992 4
  • Cannot mount GCS bucket

    Cannot mount GCS bucket

    I have been using goofys with AWS S3 without issues, working great.

    When I try to run experiments on GCS, I am not able to mount the bucket.

    This is the commands I tried:

    [email protected]:$ goofys -o allow_other --uid=1001 --gid=1002 --dir-mode=0755 --file-mode=0644 --debug_fuse --profile=gcs -f --endpoint https://storage.googleapis.com  goofys-testing-us-west-2 aperturedb/db/images
    2022/11/04 21:38:46.597680 s3.ERROR code=NoCredentialProviders msg=no valid providers in chain. Deprecated.
    	For verbose messaging see aws.Config.CredentialsChainVerboseErrors, err=<nil>
    
    2022/11/04 21:38:46.597754 main.ERROR Unable to access 'goofys-testing-us-west-2': NoCredentialProviders: no valid providers in chain. Deprecated.
    	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
    2022/11/04 21:38:46.597767 main.FATAL Mounting file system: Mount: initialization failed
    
    [email protected]:/$ goofys -o allow_other --debug_fuse --profile=gcs -f --endpoint https://storage.googleapis.com  goofys-testing-us-west-2 aperturedb/db/images
    2022/11/04 21:41:02.043650 s3.ERROR code=NoCredentialProviders msg=no valid providers in chain. Deprecated.
    	For verbose messaging see aws.Config.CredentialsChainVerboseErrors, err=<nil>
    
    2022/11/04 21:41:02.043751 main.ERROR Unable to access 'goofys-testing-us-west-2': NoCredentialProviders: no valid providers in chain. Deprecated.
    	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
    2022/11/04 21:41:02.043764 main.FATAL Mounting file system: Mount: initialization failed
    

    What am I missing?

    opened by luisremis 0
Releases(v0.24.0)
  • v0.24.0(Apr 4, 2020)

    A bunch of small fixes all over the place, xattr should work better, fixed writing to the wrong path if we mounted with a prefix and we read the directory where a new file was concurrently being created, fixed some inode caching behavior, also improved handling of azure blob in storage accounts with hierarchical namespace enabled. Also fixed mounting azure blob using only environment variables.

    Built with go 1.13.1

    Twitter Follow me on Twitter.

    Github Release

    Source code(tar.gz)
    Source code(zip)
    goofys(22.21 MB)
  • v0.23.1(Nov 17, 2019)

    I accidentally uploaded the 0.23.0 binary from right before I made the tag, so the version string was wrong. This release has no changes except with the correct version string.

    Built with go 1.13.1

    Twitter Follow me on Twitter.

    Github Release

    Source code(tar.gz)
    Source code(zip)
    goofys(22.18 MB)
  • v0.23.0(Nov 13, 2019)

  • v0.22.0(Nov 2, 2019)

    Lots of big and small fixes. Fixed a short read that can happen for files around 20MB. Fixed writing to files when the same process also fork/exec other processes. Correctly invalidate list results when they've been removed out of band on the server side. Fixed cases when we can omit some files from being returned in list. Improved retry in Azure Datalake v2. Release binary no longer built with cgo.

    Thanks to @dotslash for his many contributions!

    Built with go 1.13.1

    Twitter Follow me on Twitter.

    Github Release

    Source code(tar.gz)
    Source code(zip)
    goofys(22.79 MB)
  • v0.21.0(Aug 21, 2019)

    Goofys now have support for for Azure Blob Storage, Data Lake Gen1 and Data Lake Gen2! Also renaming smallish directories (with <= 1000 children in the subtree) is also supported now. A few locking issues were fixed. Thanks @t2y, @vager, and @gaul for their contributions!

    Also, as someone who was born and grew up in Hong Kong but now lives in America, I sympathize with the struggle for more liberties for everyone living there and in the rest of China. It's a long battle that won't end with extradition law or the 5 demands. At the same time it's also important to remember Hong Kong became the place it is now because it welcomed people, including my parents, from all over China and showed them there's a different way of living. I am confident that through peaceful protests and never give up on finding common grounds everyone can come out of this ahead. 🇭🇰

    Built with go 1.12.5

    Twitter Follow me on Twitter.

    Github Release

    Source code(tar.gz)
    Source code(zip)
    goofys(28.26 MB)
  • v0.20.0(Jun 10, 2019)

    It's been a year and a half since the last release so this release is huge. Sorry I've been lazy about cutting new releases and I will try not to make these huge releases again. There were a couple deadlock and panic fixes, and listing now uses listobjects v2 which should be much faster for large buckets. Most importantly we now have a logo!

    There are lots of new and exciting things coming this summer and I can't wait for the next million downloads!

    Built with go 1.12.5

    Twitter Follow me on Twitter.

    Github Release

    Source code(tar.gz)
    Source code(zip)
    goofys(15.49 MB)
  • v0.19.0(Jan 9, 2018)

    Supports writing files > 5MB to Google Cloud Storage. Fixed a crash with minio. A couple other small fixes including some directory mtime changes, although more of the latter is probably needed soon.

    Bumped version to 0.19.0 from 0.0.18 to be more consistent with catfs versioning.

    Built with go 1.9.2

    Twitter Follow me on Twitter.

    Github Release

    Source code(tar.gz)
    Source code(zip)
    goofys(13.31 MB)
  • v0.0.18(Oct 25, 2017)

    Fixed a read corruption that can happen when we get network errors. Fixed a deadlock that can happen under memory pressure. Fixed a couple panics that can happen on rename or read errors. Fixed regression for Ceph support and also write access to buckets that are public-read (use --profile default). Goofys should also use somewhat less memory now with huge buckets. A couple other misc bug fixes as well.

    Built with go 1.8.3

    Twitter Follow me on Twitter.

    Github Release

    Source code(tar.gz)
    Source code(zip)
    goofys(13.23 MB)
  • v0.0.17(Sep 5, 2017)

  • v0.0.16(Aug 17, 2017)

    Fixed a bug with returning files as directories during cached readdir. A couple xattr fixes. Also experimental integration with catfs.

    You can now run goofys' benchmark suite in docker: https://github.com/kahing/goofys/#runbenchmark

    Built with go 1.8.3

    Twitter Follow me on Twitter.

    Github Release

    Source code(tar.gz)
    Source code(zip)
    goofys(13.21 MB)
  • v0.0.15(Aug 10, 2017)

    Even faster find than 0.0.14 by doing smarter caching (2+x faster than 0.0.14 and 5+x faster than 0.0.13). Also fixed a race introduced in the previous version. This optimization only kicks in when goofys detects application is sequentially reading a directory tree (either breath-first or depth-first).

    Built with go 1.8.3

    Twitter Follow me on Twitter.

    Github Release

    Source code(tar.gz)
    Source code(zip)
    goofys(13.19 MB)
  • v0.0.14(Aug 8, 2017)

    Extended attributes support and much faster readdir over smallish directory trees (find). Readdir result is now cached according to --type-cache-ttl. New --no-implicit-dir option to reduce ListObjects requests. Small POSIX tweaks.

    Thanks @andrewgaul, @hit1943, @jtwang83, and @khoan for contributing during this release period!

    Built with go 1.8.3

    Twitter Follow me on Twitter.

    Github Release

    Source code(tar.gz)
    Source code(zip)
    goofys(13.17 MB)
  • v0.0.13(Jun 28, 2017)

  • v0.0.12(Jun 4, 2017)

    Correct release binary for 0.0.11. The changelog is the same.

    Mostly a bunch of bug fixes. Fixed a couple read hangs and worked around applications that don't check the return value of read(2). Upgraded aws-sdk-go to 1.8.25 for better compatibility and listing error handling. Better usability with bucket mount prefix and mount error displays.

    Built with go 1.8.1

    Github Release

    Source code(tar.gz)
    Source code(zip)
    goofys(13.11 MB)
  • v0.0.11(May 18, 2017)

    PLEASE READ: removed the binary from the release as I accidentally uploaded the wrong one. Will make a new release when I travel to somewhere with better wifi.

    Mostly a bunch of bug fixes. Fixed a couple read hangs and worked around applications that don't check the return value of read(2). Upgraded aws-sdk-go to 1.8.25 for better compatibility and listing error handling. Better usability with bucket mount prefix and mount error displays.

    Built with go 1.8.1

    Github Release

    Source code(tar.gz)
    Source code(zip)
  • v0.0.10(Feb 16, 2017)

    Added a --cheap option to reduce S3 operation cost at a slight expense of directory lookup performance. Fixed a bug with reading blobs with Content-Encoding set to gzip. Better compatibility with Amplidata. Switched to a newer version of s3proxy for testing so tests should run faster as well.

    goofys (v0.0.9) is now available via Homebrew so OS X users should have an easier time installing it.

    Built with go 1.7.5

    Github Release

    Source code(tar.gz)
    Source code(zip)
    goofys(10.72 MB)
  • v0.0.9(Nov 3, 2016)

  • v0.0.8(Sep 15, 2016)

  • v0.0.7(Aug 25, 2016)

  • v0.0.6(Jun 1, 2016)

    Allow mounting only objects under a prefix with bucket:prefix notation. Improved compatibility with more non-AWS S3 systems. Optionally set object content type base on their extensions (--use-content-type). Allow true anonymous access to buckets that allow it. And a couple crash bugs squashed.

    Built with go 1.6.2

    Github All Releases

    Source code(tar.gz)
    Source code(zip)
    goofys(13.45 MB)
  • v0.0.5(Feb 27, 2016)

  • v0.0.4(Feb 10, 2016)

    Support for v2 signing added which enabled goofys to work with ceph-s3 and other S3 based object stores. Better read performance by using multiple streams. Reduced idle memory usage. Better S3 region detection and explicit override (--region). AWS profiles support (--profile). Fixed inode cache TTL and other misc bug fixes.

    Thank @andrewgaul and @bedge for their contributions!

    Github All Releases

    Source code(tar.gz)
    Source code(zip)
    goofys(12.27 MB)
  • v0.0.3(Nov 19, 2015)

    Support for mounting via fstab added. Fixed a problem with new files sometimes not showing up, and a problem with touch returning success even when it failed. Also fixed a crash with inode caching.

    Binary is built with aws-sdk-go v1.0.0

    Github All Releases

    Source code(tar.gz)
    Source code(zip)
    goofys(11.81 MB)
  • v0.0.2(Oct 30, 2015)

    Fixed a read data corruption bug, some performance improvements, and sprinkled a little more POSIX dust. Otherwise mostly more benchmark automation and cleanups. Thanks @andrewgaul and @lrowe for their contributions, especially @lrowe for all the investigations!

    Github All Releases

    Source code(tar.gz)
    Source code(zip)
    goofys(11.16 MB)
  • v0.0.1(Oct 16, 2015)

Owner
Ka-Hing Cheung
All things systems, storage, and free software. Previously at @riverbed, @gridstore, @bouncestorage, @etleap, and @cloudflare.
Ka-Hing Cheung
High quality cloud service emulators for local development stacks

emulators High quality Google Cloud service emulators for local development stacks Why? At FullStory, our entire product and backend software stack ru

Engineering at FullStory 92 Dec 14, 2022
An edge-native container management system for edge computing

SuperEdge is an open source container management system for edge computing to manage compute resources and container applications in multiple edge regions. These resources and applications, in the current approach, are managed as one single Kubernetes cluster. A native Kubernetes cluster can be easily converted to a SuperEdge cluster.

SuperEdge 881 Dec 29, 2022
cloud-native local storage management system

Open-Local是由多个组件构成的本地磁盘管理系统,目标是解决当前 Kubernetes 本地存储能力缺失问题。通过Open-Local,使用本地存储会像集中式存储一样简单。

null 325 Dec 30, 2022
GoDrive: A cloud storage system similar to Dropbox or Google Drive, with resilient

Cloud Storage Service Author: Marisa Tania, Ryan Tjakrakartadinata Professor: Matthew Malensek See project spec here: https://www.cs.usfca.edu/~mmalen

Ryan G Tjakrakartadinata 2 Dec 7, 2021
Sample apps and code written for Google Cloud in the Go programming language.

Google Cloud Platform Go Samples This repository holds sample code written in Go that demonstrates the Google Cloud Platform. Some samples have accomp

Google Cloud Platform 3.7k Jan 9, 2023
GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go

GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go Overview GeeseFS allows you to mount an S3 bucket as a file sys

Yandex.Cloud 329 Jan 1, 2023
Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go

Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go Overview Goofys allows you to mount an S3 bucket as a filey system. It's a

Ka-Hing Cheung 4.5k Jan 8, 2023
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).

中文 English 愿景:为用户提供最简单、可靠、高效的分布式文件系统。 go-fastdfs是一个基于http协议的分布式文件系统,它基于大道至简的设计理念,一切从简设计,使得它的运维及扩展变得更加简单,它具有高性能、高可靠、无中心、免维护等优点。 大家担心的是这么简单的文件系统,靠不靠谱,可不

小张 3.3k Jan 8, 2023
🔑A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout (LSM+WAL) similar to Riak.

bitcask A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout

James Mills 10 Sep 26, 2022
Amazon ECS Container Agent: a component of Amazon Elastic Container Service

Amazon ECS Container Agent The Amazon ECS Container Agent is a component of Amazon Elastic Container Service (Amazon ECS) and is responsible for manag

null 0 Dec 28, 2021
JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is an open-source POSIX file system built on top of Redis and object storage (e.g. Amazon S3), designed and optimized for cloud native environ

Juicedata, Inc 7.2k Jan 2, 2023
JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is an open-source POSIX file system built on top of Redis and object storage

Juicedata, Inc 7.2k Jan 5, 2023
JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native

Juicedata, Inc 7.2k Jan 4, 2023
JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native

Juicedata, Inc 7.2k Jan 1, 2023
the pluto is a gateway new time, high performance, high stable, high availability, easy to use

pluto the pluto is a gateway new time, high performance, high stable, high availability, easy to use Acknowledgments thanks nbio for providing low lev

mobus 2 Sep 19, 2021
Squzy - is a high-performance open-source monitoring, incident and alert system written in Golang with Bazel and love.

Squzy - opensource monitoring, incident and alerting system About Squzy - is a high-performance open-source monitoring and alerting system written in

Squzy 468 Dec 12, 2022
A POSIX-compliant AWK interpreter written in Go

GoAWK: an AWK interpreter written in Go AWK is a fascinating text-processing language, and somehow after reading the delightfully-terse The AWK Progra

Ben Hoyt 1.6k Dec 31, 2022
cyhone 149 Dec 28, 2022