go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).

Overview

中文 English

愿景:为用户提供最简单、可靠、高效的分布式文件系统。

logo

go-fastdfs是一个基于http协议的分布式文件系统,它基于大道至简的设计理念,一切从简设计,使得它的运维及扩展变得更加简单,它具有高性能、高可靠、无中心、免维护等优点。

大家担心的是这么简单的文件系统,靠不靠谱,可不可以用于生产环境?答案是肯定的,正因为简单所以高效,因为简单所以稳定。如果你担心功能,那就跑单元测试,如果担心性能,那就跑压力测试,项目都自带了,跑一跑更放心^_^。

注意:使用前请认真阅读使用文档视频教程

  • 支持curl命令上传
  • 支持浏览器上传
  • 支持HTTP下载
  • 支持多机自动同步
  • 支持断点下载
  • 支持配置自动生成
  • 支持小文件自动合并(减少inode占用)
  • 支持秒传
  • 支持跨域访问
  • 支持一键迁移(搬迁)
  • 支持异地备份(特别是小文件1M以下)
  • 支持并行体验
  • 支持断点续传(tus)
  • 支持docker部署
  • 支持自监控告警
  • 支持图片缩放
  • 支持google认证码
  • 支持自定义认证
  • 支持集群文件信息查看
  • 使用通用HTTP协议
  • 无需专用客户端(支持wget,curl等工具)
  • 类fastdfs
  • 高性能 (使用leveldb作为kv库)
  • 高可靠(设计极其简单,使用成熟组件)
  • 无中心设计(所有节点都可以同时读写)

优点

  • 无依赖(单一文件)
  • 自动同步
  • 失败自动修复
  • 按天分目录方便维护
  • 支持不同的场景
  • 文件自动去重
  • 支持目录自定义
  • 支持保留原文件名
  • 支持自动生成唯一文件名
  • 支持浏览器上传
  • 支持查看集群文件信息
  • 支持集群监控邮件告警
  • 支持小文件自动合并(减少inode占用)
  • 支持秒传
  • 支持图片缩放
  • 支持google认证码
  • 支持自定义认证
  • 支持跨域访问
  • 极低资源开销
  • 支持断点续传(tus)
  • 支持docker部署
  • 支持一键迁移(从其他系统文件系统迁移过来)
  • 支持异地备份(特别是小文件)
  • 支持并行体验(与现有的文件系统并行体验,确认OK再一键迁移)
  • 支持token下载 token=md5(file_md5+timestamp)
  • 运维简单,只有一个角色(不像fastdfs有三个角色Tracker Server,Storage Server,Client),配置自动生成
  • 每个节点对等(简化运维)
  • 所有节点都可以同时读写

极速体验,只需一分钟

启动服务器(已编译,下载

./fileserver #注意:线上使用请使用项目的control文件进行管理,直接运行,关闭终端会退出。

命令上传

curl -F [email protected] http://10.1.xx.60:8080/group1/upload

WEB上传(浏览器打开)

http://yourserver ip:8080 注意:不要使用127.0.0.1上传

使用文档

视频教程

如果你觉得本项目不错,请点击项目顶部的 star 按钮关注本项目

QQ交流群:964274270(go-fastdfs技术交流群)

同时要说明的是,该群是一个学习交流群,如果是程序相关问题,请直接提交issues,不接受邮件求助、微信求助和QQ私信求助

Issues
  • 文件服务器是怎样扩容的?

    文件服务器是怎样扩容的?

    文件服务器的扩容是怎样实现的呢?比如现在已经配置了三个peer节点,这三个节点对等,会自动同步文件,实现了文件服务器的高可用。但发现有一天文件服务器磁盘快满了,我应该怎样扩容呢? 扩容方案: 1)增加服务器,新增加的服务器跟现有peer节点之间数据不再同步,而是跟现有peer节点间形成2个分片式集群。也就是可以形成多个分片,每个分片数据都不相同,每个分片都可以有多个对等节点,形成高可用。这种实现方式有点类似于solr。 2)在现有节点上挂载新空间。利用存量对等节点,给每个节点都分配新的存储空间。这种方式有点类似于TFS。

    opened by chengyuansen 9
  • 还是组文件删除的问题~~

    还是组文件删除的问题~~

    经测试: auth_url不配置的时候,组内的两台机都能删除。 auth_url配置了,有一台不能删除,fileserver.log报错如下:

    2021-04-30 11:37:34 [ERR] [fileserver.go:1009] [main.(*Server).CheckAuth] main.JsonResult.Status: ReadString: expects " or n, but found 4, error found in #10 byte of ...|"status":400,"error"|..., bigger context ...|mestamp":"2021-04-30T03:37:34.122+0000","status":400,"error":"Bad Request","message":"Required Strin|...

    请问解决方法,谢谢~~

    opened by lovelygz 5
  • Can you dynamically add nodes?

    Can you dynamically add nodes?

    Hello,

    I am looking for a good alternative to IPFS in Golang for a P2P project that would need to support many thousands of Peers and I am wondering how easily go-fastdfs scales if nodes are dynamically added and removed as in a P2P operation.

    Also, it is possible to "pin" files to a specific node and still be accessible to those files from other nodes?

    Any help would be greatly appreciated. Thanks in advance

    opened by lonnietc 4
  • 使用hutool的http工具进行上传时,返回request Content-Type isn't multipart/form-data

    使用hutool的http工具进行上传时,返回request Content-Type isn't multipart/form-data

    按照文档上的代码使用hutool进行http上传,但是给我返回这个提示 request Content-Type isn't multipart/form-data

    下面是代码: ` byte[] bytes = FileUtil.readBytes(filePath); File file = new File(filePath);

        InputStream inputStream = new ByteArrayInputStream(bytes);
        InputStreamResource isr = new InputStreamResource(inputStream, file.getName());
       
    
        //声明参数集合
        HashMap<String, Object> paramMap = new HashMap<String, Object>();
        //文件
        paramMap.put("file", isr);
        //输出
        paramMap.put("output", "json");
        //自定义路径
        //paramMap.put("path", "image");
        //场景
        paramMap.put("scene", "register");
        //上传
        String result = HttpUtil.post(MessageFormat.format("{0}/upload", address), paramMap);`
    

    使用okhttp3进行上传(代码复制的文档中的示例代码)时,有时候正常,有时候提示

    (String)"(error) if you want to upload fast md5 is require,and if you want to upload file,you must use post method

    想问下这个情况的大概原因

    opened by jethrozz 4
  • 当跨域与验证auth_token都使用时上传问题

    当跨域与验证auth_token都使用时上传问题

    opened by ghost 4
  • 断点上传返回地址错误

    断点上传返回地址错误

    在使用TUS断点上传时返回地址没有端口

    位置TusClient.createUpload
    String urlStr = connection.getHeaderField("Location");
    

    docker-compose.yml

    version: "3.7"
    
    services:
      fastdfs0:
        container_name: fastdfs0
        image: sjqzhang/go-fastdfs
        volumes:
          - ./data0:/data
        environment:
          GO_FASTDFS_DIR: /data
        networks:
          fastdfs:
            ipv4_address: 172.30.0.2
    
      fastdfs1:
        container_name: fastdfs1
        image: sjqzhang/go-fastdfs
        volumes:
          - ./data1:/data
        environment:
          GO_FASTDFS_DIR: /data
        networks:
          fastdfs:
            ipv4_address: 172.30.0.3
    
      fastdfs2:
        container_name: fastdfs2
        image: sjqzhang/go-fastdfs
        volumes:
          - ./data2:/data
        environment:
          GO_FASTDFS_DIR: /data
        networks:
          fastdfs:
            ipv4_address: 172.30.0.4
    
      fastdfs-nginx:
        container_name: fastdfs-nginx
        image: nginx
        volumes:
         - ./nginx:/etc/nginx/conf.d
        depends_on:
         - fastdfs0
         - fastdfs1
         - fastdfs2
        networks:
          fastdfs:
            ipv4_address: 172.30.0.250
        ports:
         - "8080:80"
    
    networks:
      fastdfs:
         driver: bridge
         ipam:
           driver: default
           config:
             - subnet: 172.30.0.0/16
    

    nginx default.conf

    upstream group1{
      server 172.30.0.2:8080;
      server 172.30.0.3:8080;
      server 172.30.0.4:8080;
      ip_hash;
    }
    
    log_format  upstream  '$remote_addr - $remote_user [$time_local] "$request" '
                '$status $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for" '
                '$upstream_addr $request_time $upstream_response_time ';
    
    client_max_body_size 100M;
    
    server {
        listen       80;
        server_name  localhost;
    
        #charset koi8-r;
        access_log  /var/log/nginx/access.log  upstream;
    
        location /{
          proxy_set_header    Host $host;
          proxy_set_header    X-Real-IP $remote_addr;
          proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
    
          proxy_pass http://group1;
        }
    
        location /group1{
          proxy_set_header    Host $host;
          proxy_set_header    X-Real-IP $remote_addr;
          proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
    
          proxy_pass http://group1;
        }
    
        #location / {
        #    root   /usr/share/nginx/html;
        #    index  index.html index.htm;
        #}
    
        #error_page  404              /404.html;
    
        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    
        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}
    
        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}
    
        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }
    
    
    opened by ukid 4
  • 文件去重可能出现问题?

    文件去重可能出现问题?

    测试时候发现不是必然发生,但是概率不低。 就是多次上传相同文件,按照去重,应该返回的都是第一次上传的文件信息,但是有时候却返回的是无法打开文件失败,路径指向的是最新的地址。 open files/video/20190226/18/08/8/test.mp4: no such file or directory

    使用的是sha1去重,搭建两台服务器,使用nginx做负载均衡。

    opened by jianbiaoxu 4
  • 首次启动无权限报错

    首次启动无权限报错

    在CentOS7中,首次启动报错如下

    panic: open file path: conf/cfg.json error: open conf/cfg.json: permission denied
    
    
    goroutine 1 [running]:
    main.ParseConfig(0x99d8f6, 0xd)
    	/root/fileserver.go:422 +0x76b
    main.init.0()
    	/root/fileserver.go:2801 +0x222
    main.init()
    	<autogenerated>:1 +0x12b
    

    启动用户为普通账户,非root账户

    解决方法也简单,首次启动时,生成的目录给属主有x权限即可,即

    chmod u+x conf
    chmod u+x files
    

    上传文件不存在,也是由于目录权限导致 ./files/default无权限导致

    建议代码中所有os.mkdir权限全部改为0775

    opened by vinsonzou 4
  • 还是文件丢失问题

    还是文件丢失问题

    今天又试了试,我甚至重新部署了一遍,去重功能也关了,早上 在files/0709/chaxiao8ml/870路径下上传了870张照片, 在files/0709/chaxiaoml/844路径下上传了844张照片, 在files/0709/test路径下上传了1张照片,调用statAPI统计fileCount一共1715张,一切都很正常, 但是下午一看,870张照片变成了824张,844张照片变成了808张,更离谱的是调用statAPI统计fileCount变成了1721张,不少反多?这些数据都是刚测出来,搞不懂为什么https://github.com/sjqzhang/go-fastdfs/issues/293

    opened by OTLCLoud 3
  • Tus的Concatenation扩展问题

    Tus的Concatenation扩展问题

    360截图18061214184075 StatusCode: 404, ReasonPhrase: 'Not Found', Version: 1.1, Content: System.Net.Http.HttpConnection+HttpConnectionResponseContent, Headers:\r\n{\r\n Access-Control-Allow-Headers: Authorization, Content-Type, Depth, User-Agent, X-File-Size, X-Requested-With, X-Requested-By, If-Modified-Since, X-File-Name, X-File-Type, Cache-Control, Origin\r\n Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, DELETE\r\n Access-Control-Allow-Origin: *\r\n Access-Control-Expose-Headers: Authorization\r\n Tus-Resumable: 1.0.0\r\n X-Content-Type-Options: nosniff\r\n Date: Mon, 24 May 2021 13:01:18 GMT\r\n Content-Length: 17\r\n Content-Type: text/plain; charset=utf-8\r\n}

    opened by bidw77 3
  • reconstruct project and add go mod support

    reconstruct project and add go mod support

    1、add go mod support 2、reconstruct project 3、add github workflows 4、add Makefile 5、change tusd import path 6、change gopsutil import path 7、add .gitignore file 8、add cmd

    // run on mac
    $ ./fileserver_darwin                                                                                                                                                                                                                                                                                                                               [email protected]
    2021/05/17 18:02:10 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
    Usage:
      fileserver [command]
    
    Available Commands:
      doc         Run doc server
      help        Help about any command
      server      Run fastdfs server
      version     version
    
    Flags:
      -h, --help   help for fileserver
    
    Use "fileserver [command] --help" for more information about a command.
    
    opened by busyfree 3
  • 上传编码后的中文文件名后,访问返回的地址404

    上传编码后的中文文件名后,访问返回的地址404

    1、上传编码后的中文文件名,本地文件存储为编码后的文件名 2、返回的url也是编码后的,但发现访问URL时,会解码URI,导致无法访问到 fileserver.go:285

    if fullpath, err = url.PathUnescape(fullpath); err != nil {
    		log.Error(err)
    }
    
    opened by BlitheStudy 0
  • 指定存储文件夹,上传相同名字的图片会被最开始的图片所覆盖

    指定存储文件夹,上传相同名字的图片会被最开始的图片所覆盖

    屏幕快照 2021-06-09 下午9 52 35
    • 指定存储文件夹后->上传 icon_bank_check.png文件后 ->再次上传icon_bank_check.png文件
    • 第二个icon_bank_check 文件名为 0_icon_bank_check.png,但是返回的结果是依然是icon_bank_check.png

    返回的结果依然是

    obj.Url: http://localhost:8080/group1/testimg/icon_bank_check.png?name=icon_bank_check.png&download=1
    
    opened by yyccQQu 1
Releases(v1.4.3)
Owner
小张
擅长系统架构, 对高可用高并发有比较深的理解,对 devops也有较深认识。
小张
Open Source Continuous File Synchronization

Goals Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers. We strive to fulfill the goals belo

The Syncthing Project 45.4k Jun 26, 2022
Plik is a scalable & friendly temporary file upload system ( wetransfer like ) in golang.

Want to chat with us ? Telegram channel : https://t.me/plik_root_gg Plik Plik is a scalable & friendly temporary file upload system ( wetransfer like

root.gg 1k Jun 28, 2022
Fast, dependency-free, small Go package to infer the binary file type based on the magic numbers signature

filetype Small and dependency free Go package to infer file and MIME type checking the magic numbers signature. For SVG file type checking, see go-is-

Tom 1.5k Jun 23, 2022
GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go

GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go Overview GeeseFS allows you to mount an S3 bucket as a file sys

Yandex.Cloud 258 Jun 21, 2022
High Performance, Kubernetes Native Object Storage

MinIO Quickstart Guide MinIO is a High Performance Object Storage released under Apache License v2.0. It is API compatible with Amazon S3 cloud storag

High Performance, Kubernetes Native Object Storage 33.7k Jun 21, 2022
Provide an upload endpoint that stores files on pinata and returns a json response with the uploaded file pinata url

Purpose Build a template repository to get to coding as quickly as possible, by starting from a common template which follows the guidelines here Feat

Dathan Vance Pattishall 0 Dec 30, 2021
Upgit - Upgit helps you simply upload any file to your Github repository and then get a raw URL for it

Upgit - Upgit helps you simply upload any file to your Github repository and then get a raw URL for it

null 266 Jun 16, 2022
Merge some files in a path with custom extension in golang

goMerge Merge some files in a path with custom extension in golang Feature Walk in a path with a goroutine using parallelism Merge files with custom e

Javad Rajabzade 4 Apr 26, 2022
A virtual file system for small to medium sized datasets (MB or GB, not TB or PB). Like Docker, but for data.

AetherFS assists in the production, distribution, and replication of embedded databases and in-memory datasets. You can think of it like Docker, but f

mya 8 Feb 9, 2022
Download an upload large files to Google Drive (API v3)

gdriver gdriver is a command-line tool, written in Go, used for uploading and downloading large personal files from Google Drive (API v3). The tool pr

Marcin Tojek 66 Apr 2, 2022
upload to gdrive include shared drive

gdrive_uploader upload to gdrive include shared drive How to get go get github.com/heat1024/gdrive_uploader About credential You must set your client

ryu 3 Jun 8, 2022
A simply upload fileserver for fun

Filebus A simply upload fileserver for fun Build Binary: go build -o filebus . Docker docker build . -t n0vad3v/filebus:lastest Usage Filebus will b

Nova Kwok 0 Dec 14, 2021
A PDF document generator with high level support for text, drawing and images

GoFPDF document generator Package go-pdf/fpdf implements a PDF document generator with high level support for text, drawing and images. Features UTF-8

null 140 Jun 22, 2022
Ghostinthepdf - This is a small tool that helps to embed a PostScript file into a PDF

This is a small tool that helps to embed a PostScript file into a PDF in a way that GhostScript will run the PostScript code during the

Emil Lerner 123 Jun 18, 2022
A small tool for sending a single file to another machine

file-traveler A small tool for sending a single file to another machine. Build g

Vence Lam 1 Dec 28, 2021
Abstract File Storage

afs - abstract file storage Please refer to CHANGELOG.md if you encounter breaking changes. Motivation Introduction Usage Matchers Content modifiers S

Viant, Inc 186 Jun 19, 2022
File uploader with support for multiple hosts and progress reporting written in Go.

go-upload File uploader with support for multiple hosts and progress reporting written in Go. Windows, Linux, macOS and Android binaries Usage Upload

null 15 Jun 3, 2022
Bigfile -- a file transfer system that supports http, rpc and ftp protocol https://bigfile.site

Bigfile ———— a file transfer system that supports http, rpc and ftp protocol 简体中文 ∙ English Bigfile is a file transfer system, supports http, ftp and

null 225 Jun 13, 2022
Dragonfly is an intelligent P2P based image and file distribution system.

Dragonfly Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in o

dragonflyoss 5.8k Jun 30, 2022