golang NFSv3 server

Overview

Golang Network File Server

NFSv3 protocol implementation in pure Golang.

Current Status:

  • Minimally tested
  • Mounts, read-only and read-write support

Usage

The most interesting demo is currently in example/osview.

Start the server go run ./example/osview ..

The local folder at . will be the initial view in the mount. mutations to metadata or contents will be stored purely in memory and not written back to the OS. When run, this demo will print the port it is listening on.

The mount can be accessed using a command similar to mount -o port=,mountport= -t nfs localhost:/mount (For Mac users)

or

mount -o port=,mountport=,nfsvers=3,noacl,tcp -t nfs localhost:/mount (For Linux users)

API

The NFS server runs on a net.Listener to export a file system to NFS clients. Usage is structured similarly to many other golang network servers.

import (
   	"github.com/go-git/go-billy/v5/memfs"

	nfs "github.com/willscott/go-nfs"
	nfshelper "github.com/willscott/go-nfs/helpers"
)

listener, _ := net.Listen("tcp", ":0")
fmt.Printf("Server running at %s\n", listener.Addr())

mem := memfs.New()
f, err := mem.Create("hello.txt")
f.Write([]byte("hello world"))
f.Close()

handler := nfshelper.NewNullAuthHandler(mem)
cacheHelper := nfshelper.NewCachingHandler(handler)
nfs.Serve(listener, cacheHelper)

Notes

  • Ports are typically determined through portmap. The need for running portmap (which is the only part that needs a privileged listening port) can be avoided through specific mount options. e.g. mount -o port=n,mountport=n -t nfs host:/mount /localmount

  • This server currently uses billy to provide a file system abstraction layer. There are some edges of the NFS protocol which do not translate to this abstraction.

    • NFS expects access to an inode or equivalent unique identifier to reference files in a file system. These are considered opaque identifiers here, which means they will not work as expected in cases of hard linking.
    • The billy abstraction layer does not extend to exposing uid and gid ownership of files. If ownership is important to your file system, you will need to ensure that the os.FileInfo meets additional constraints. In particular, the Sys() escape hatch is queried by this library, and if your file system populates a syscall.Stat_t concrete struct, the ownership specified in that object will be used.
  • Relevant RFCS: 5531 - RPC protocol, 1813 - NFSv3, 1094 - NFS

Comments
  • mount error

    mount error

    hi: 1 start example osview like this: ./osview /public 2045 2 use mount like this: mount -t nfs -o port=2045,mountport=2045,nfsvers=3,tcp -o nolock 10.243.88.235:/public/ /share/ 3 the output at server console:

    2021/09/14 20:11:49 request: RPC #4228535329 (nfs.Null) 2021/09/14 20:11:49 request: RPC #4245312545 (nfs.GetAttr) 2021/09/14 20:11:49 call to 0x5dc6a0 failed: Invalid file handle

    How can I do above the problem... help me~~~ thank you My English is terrible, please forgive me...

    opened by twoholes 8
  • ReadDirPlus doesn't return all pages when directory has too much content

    ReadDirPlus doesn't return all pages when directory has too much content

    To reproduce:

    step 1 - get content

    cd /home/git
    git clone https://github.com/Tieqiang/dc-heacth
    ls dc-heacth/src/main/java/com/dchealth/VO | wc -l
    # expected output:  65
    

    step 2 - run osview server

    go run ./example/osview /home/git
    

    step 3 - mount nfs and list dir

    mkdir /tmp/git
    mount -o port=50184,mountport=50184 -t nfs localhost:/ /tmp/git
    ls /tmp/git/dc-heacth/src/main/java/com/dchealth/VO | wc -l
    # actual output is 27, not the whole 65
    

    After #25 was merged I'm no longer seeing stale cookie errors, but still not all pages are retrieved.

    opened by liiri 4
  • Folder content is not shown if there 100 or more catalogs in one catalog

    Folder content is not shown if there 100 or more catalogs in one catalog

    Easy to reproduce like:

    for i in range(1,100):
        os.makedirs(f'broken_catalog/sub_catalog_{i:03}', exist_ok=True)
    

    broken_catalog content will be empty for a NFS client, a similar issue exists for many files in one catalog.

    opened by AleksandrIakhnev 4
  • go-nfs not taking into account file permissions

    go-nfs not taking into account file permissions

    Hey @willscott,

    First of all, thank you for this awesome project.

    I'm trying to containerize this project for my demos. I have it working but at this time it doesn't honor file permissions present on the filesystem.

    On the readme you have this:

    The billy abstraction layer does not extend to exposing uid and gid ownership of files. If ownership is important to your file system, you will need to ensure that the os.FileInfo meets additional constraints. In particular, the Sys() escape hatch is queried by this library, and if your file system populates a syscall.Stat_t concrete struct, the ownership specified in that object will be used.

    I don't fully understand what you mean here, I believe my filesystem populates the uid and gid for every object, not sure what I'm doing wrong.

    I'm using the example in example/osnfs

    Thanks!

    opened by mvazquezc 4
  • Dependency Issue

    Dependency Issue

    Hi

    I am getting:

    go: github.com/willscott/[email protected] requires
    	github.com/willscott/[email protected] requires
    	go.polydawn.net/[email protected]: unrecognized import path "go.polydawn.net/go-timeless-api": https fetch: Get "https://go.polydawn.net/go-timeless-api?go-get=1": dial tcp 107.170.61.58:443: connect: connection refused
    

    Probably go.polydawn.net/* needs to be replaced with github.com/polydawn/*

    opened by shammishailaj 3
  • Mount(ubuntu 18.04): portmap query failed: RPC: Program not registered

    Mount(ubuntu 18.04): portmap query failed: RPC: Program not registered

    server: go run ./example/helloworld

    Server running at [::]:19815
    

    client: sudo mount -o port=19815,vers=3,tcp,noacl -t nfs localhost:/ /mnt/nfs -v

    mount.nfs: timeout set for Fri Jul  2 13:45:04 2021
    mount.nfs: trying text-based options 'port=19815,vers=3,tcp,noacl,addr=127.0.0.1'
    mount.nfs: prog 100005, trying vers=3, prot=6
    mount.nfs: portmap query failed: RPC: Program not registered
    mount.nfs: requested NFS version or transport protocol is not supported
    
    
    opened by aesirteam 2
  • Bug in readdirplus causing all files in a directory to be the same

    Bug in readdirplus causing all files in a directory to be the same

    There is a bug on this line: https://github.com/willscott/go-nfs/blob/master/nfs_onreaddirplus.go#L73

    handle := userHandle.ToHandle(fs, append(p, c.Name()))
    

    This code is called in a loop, looping over the result from ReadDir(). The problem here is the usage of append()

    p is a []string that comes from converting the handle to a previously stored []string.

    Each time the loop runs, append(p, ...) is called to either:

    1. create a new slice with p copied
    2. simply putting ... into existing p slice because it's big enough, return a slice that points to same memory as p but a bigger length

    If we happened to run into the second case, what happens is:

    1. we create a new handle for a file path, add to handle cache
    2. we proceed to create a second handle for a second file path, add to handle cache
    3. but the second file path and the first file path points to the same memory, and now you have 2 different file handles (uuid in this implementation) pointing to the same path

    When a NFS client list the directory, it gets 2 different handles, but they read the same file.

    opened by ziyan 2
  • off-by-one issues with dir cookies

    off-by-one issues with dir cookies

    There was an issue in accounting for the implicit . and .. entries in directory listings that was part of the issue in #26

    With this fix, I get the correct output on a linux machine. The described symptoms in #26 imply there may be a secondary issue in mac interactions, since it the disjoint of 27 vs 65 entries is likely that the subsequent pages of entries are not correctly fetched at all, not simply the off-by-two paging issue this commit fixes.

    opened by willscott 2
  • Chown errors

    Chown errors

    When I use the osnfs example and try to chown a file from the client mount, I get an error it's not supported.

    expected:

    chown works

    actual:

    client error:

    chown root hi.txt
    chown: hi.txt: RPC struct is bad
    

    server error:

    No file for lookup of ._hi.txt
    2021/03/03 13:26:51 call to 0x1125460 failed: No such file or directory
    2021/03/03 13:26:51 request: RPC #3742974177 (nfs.SetAttr)
    2021/03/03 13:26:51 call to 0x112edc0 failed: lchown /Users/source/repos/github.com/dummy/hi.txt: operation not permitted
    
    opened by chrisdostert 2
  • osnfs example errors with

    osnfs example errors with "RPC struct is bad" on writing to file

    Running the osnfs example and writing to a file in the mounted directory leads to an error "RPC struct is bad"

    this is on macOS big sur. There's many errors in the logs of osnfs but I don't know which are applicable.

    BTW, thanks for the awesome lib!

    opened by chrisdostert 2
  • Bump github.com/go-git/go-billy/v5 from 5.1.0 to 5.2.0

    Bump github.com/go-git/go-billy/v5 from 5.1.0 to 5.2.0

    Bumps github.com/go-git/go-billy/v5 from 5.1.0 to 5.2.0.

    Release notes

    Sourced from github.com/go-git/go-billy/v5's releases.

    v5.2.0

    Changelog

    • osfs: js/wasm implementation based on menfs
    Commits
    • 42ead41 osfs: fix js implementation based on menfs
    • 43a34ba github action: update versions and support js testing
    • 8d1f328 Merge pull request #15 from paralin/gopherjs-compat
    • 36058ce osfs: add js shim
    • See full diff in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • Make the cookie verifier stable across multiple readdirplus calls

    Make the cookie verifier stable across multiple readdirplus calls

    At least one NFS client seems to expect the nfs verifier cookie to be stable accross more than one call as the current implemenentation does.

    There are two options, the default one being using the stat call to the dir to get the modification time-stamp and use that as a verifier.

    The advantage is that doesn't need to calculate any checksum, the downside is, that it breaks memfs, which returns always now() as the timestamp.

    opened by fwiesel 3
  • Incompatibility with knfs ubuntu 5.15.0-47-generic

    Incompatibility with knfs ubuntu 5.15.0-47-generic

    Apparently there is an incompatibility between the linux kernel as a client and go-nfs as a server: Steps to reproduce:

    Script to reproduce the problem:

    if test ! -d testdirs; then
      mkdir -p testdirs
      for ((i=0;i<1024;++i)); do
        mkdir testdirs/glance_$(uuidgen)
      done
    fi
    
    go run example/osview/main.go  testdirs/ 12049 &
    sudo mount -o port=12049,mountport=12049,nfsvers=3,noacl,tcp,sec=none -t nfs localhost:/  /mnt
    
    ls /mnt
    

    The last command will hang and needs to be interrupted.

    Doing the same with ganesha.nfsd seems to work.

    Looking with at a tcpdump shows, (now all identifier as tcpdump uses them), that the kernel is always sending with nfs.cookie3=256 a nfs.verifier from a prior readdirplus (the one with entry 122-136). Then checksum then doesn't match, and NFS3_ERR_BAD_COOKIE is returned. And everything starts from the beginning.

    I checked the file-handles and file-ids in the reply with entry 122-136, and do not match the one, so I cannot see any sensible reason why the kernel is behaving that way.

    It looks to me that the answer from go-nfs is correct, but assuming compatibility is desired I raise the issue here, even if it is presumably a bug in the kernel (hash collision over the cookie id?).

    opened by fwiesel 1
  • build on windows 10 fail

    build on windows 10 fail

    Hello,I clone go-nfs on windows 10,then execute go run ./example/osview .,but it failed,the errors as follows:

    $ go run ./example/osview .
    go: downloading github.com/willscott/memphis v0.0.0-20201122065000-f2beb41b6be3
    go: downloading github.com/go-git/go-billy/v5 v5.3.1
    go: downloading github.com/hashicorp/golang-lru v0.5.4
    go: downloading github.com/rasky/go-xdr v0.0.0-20170124162913-1a41d1a06c93
    go: downloading github.com/willscott/go-nfs-client v0.0.0-20200605172546-271fa9065b33
    go: downloading github.com/polydawn/rio v0.0.0-20201122020833-6192319df581
    go: downloading github.com/warpfork/go-errcat v0.0.0-20180917083543-335044ffc86e
    go: downloading github.com/polydawn/go-timeless-api v0.0.0-20201121022836-7399661094a6
    go: downloading github.com/polydawn/refmt v0.0.0-20190807091052-3d65705ee9f1
    # github.com/willscott/memphis
    C:\Users\LENOVO\go\pkg\mod\github.com\willscott\[email protected]\deferred.go:20:31: undefined:
    syscall.Stat_t
    C:\Users\LENOVO\go\pkg\mod\github.com\willscott\[email protected]\deferred.go:23:21: undefined:
    createTime
    C:\Users\LENOVO\go\pkg\mod\github.com\willscott\[email protected]\deferred.go:57:30: undefined:
    syscall.Stat_t
    C:\Users\LENOVO\go\pkg\mod\github.com\willscott\[email protected]\deferred.go:60:18: undefined:
    createTime
    
    opened by jackyzjk 1
  • Support golang io/fs interface

    Support golang io/fs interface

    This library currently makes use of the billy v5 filesystem interface as the expected content that the driving program passes it to serve over NFS.

    This interface is imperfect, although that is mostly due to rough edges in golang itself - fs.FileInfo's Sys as an escape hatch to system specific stat structs containing owner/group information on unix permissioned files is awkward to say the least, and the billy interface does not integrate the Change interface particularly well, making the requirements to support a read-write filesystem for this library non-obvious.

    Golang 1.16 introduces the io/fs interface pattern for thinking about embedded files and filesystem. This uses a similar interface pattern as billy's Change, for interface-based detection of optional filesystem features. The initial 1.16 interfaces do not include any of the write or mutation portion of filesystems, though there is some indication that we may expect that to come eventually. They also do not provide a mechanism for resolving symlinks (no lstat or Readlink).

    We should decide and then support either a io/fs.FS -> billy.Filesystem shim or native io/fs.FS.

    opened by willscott 5
  • auth helper for authenticated use

    auth helper for authenticated use

    currently, the repo has only a NullAuthHelper.

    To support multi-user user cases, it would be useful to have a credential auth implementation to allow user-views of a mount.

    ref: #17

    enhancement 
    opened by willscott 0
Owner
Will
Will
server-to-server sync application, written in go/golang.

svcpy: server to server copy a basic server-to-server copy application. on a single binary, it can be a server or a client. example usage: on the serv

Mert Akengin 0 Nov 4, 2021
Pape-server - A small server written in golang to serve a random wallpaper.

pape-server I like to inject custom CSS themes into a lot of websites and electron apps, however browsers don't let websites access local disk through

null 0 Dec 31, 2021
A Language Server Protocol (LSP) server for Jsonnet

Jsonnet Language Server Warning: This project is in active development and is likely very buggy. A Language Server Protocol (LSP) server for Jsonnet.

Jack Baldry 17 Apr 29, 2022
The server-pubsub is the main backend of DATAVOC project that manages all the other web-server modules of the same project such as the processor

server-pubsub The server-pubsub is the main backend of DATAVOC project that manages all the other web-server modules of the same project such as the p

null 0 Dec 3, 2021
Server and client implementation of the grpc go libraries to perform unary, client streaming, server streaming and full duplex RPCs from gRPC go introduction

Description This is an implementation of a gRPC client and server that provides route guidance from gRPC Basics: Go tutorial. It demonstrates how to u

Joram Wambugu 0 Nov 24, 2021
Cert bound sts server - Certificate Bound Tokens using Security Token Exchange Server (STS)

Certificate Bound Tokens using Security Token Exchange Server (STS) Sample demonstration of Certificate Bound Tokens acquired from a Security Token Ex

null 0 Jan 2, 2022
Echo-server - An HTTP echo server designed for testing applications and proxies

echo-server An HTTP echo server designed for testing applications and proxies. R

Erik Cavalcanti 5 Jul 14, 2022
Broadcast-server - A simple Go server that broadcasts any data/stream

broadcast A simple Go server that broadcasts any data/stream usage data You can

Zack 55 Sep 24, 2022
Videos2gether-server - Server for the Realtime video streaming app Videos2Gether

Videos Together server Server source code for the https://videos2gether.com Arch

Tiago Taquelim 0 Jan 9, 2022
JPRQ Customizer is a customizer that helps to use the JPRQ server code and make it compatible with your own server with custom subdomain and domain

JPRQ Customizer is a customizer that helps to use the JPRQ server code and make it compatible with your own server with custom subdomain and domain.You can upload the generated directory to your web server and expose user localhost to public internet. You can use this to make your local machine a command center for your ethical hacking purpose ;)

Abir Ghosh 1 Jan 19, 2022
Envoy-eds-server - Envoy EDS server is a working Envoy Discovery Service implementation

envoy-eds-server Intro Envoy EDS server is a working Envoy Discovery Service imp

Radu Crisan 2 Apr 2, 2022
Http-server - A HTTP server and can be accessed via TLS and non-TLS mode

Application server.go runs a HTTP/HTTPS server on the port 9090. It gives you 4

Vedant Pareek 0 Feb 3, 2022
Server - Dupman server written in Go

server dupman server written in Go Requirements Go (>=1.17) Installation Usage C

dupman 9 Feb 22, 2022
“Dear Port80” is a zero-config TCP proxy server that hides SSH connection behind a HTTP server!

Dear Port80 About The Project: “Dear Port80” is a zero-config TCP proxy server that hides SSH connection behind a HTTP server! +---------------------

Abbas Gheydi 6 Jun 29, 2022
Simple mDNS client/server library in Golang

mdns Simple mDNS client/server library in Golang. mDNS or Multicast DNS can be used to discover services on the local network without the use of an au

HashiCorp 903 Sep 26, 2022
golang tcp server

TCPServer Package tcp_server created to help build TCP servers faster. Install package go get -u github.com/firstrow/tcp_server Usage: NOTICE: OnNewMe

null 419 Sep 27, 2022
A LWM2M Client and Server implementation (For Go/Golang)

Betwixt - A LWM2M Client and Server in Go Betwixt is a Lightweight M2M implementation written in Go OMA Lightweight M2M is a protocol from the Open Mo

Zubair Hamed 54 May 28, 2022
High-performance PHP application server, load-balancer and process manager written in Golang

RoadRunner is an open-source (MIT licensed) high-performance PHP application server, load balancer, and process manager. It supports running as a serv

Spiral Scout 6.7k Sep 25, 2022