Bigfile -- a file transfer system that supports http, rpc and ftp protocol https://bigfile.site

Overview

Bigfile ———— a file transfer system that supports http, rpc and ftp protocol 简体中文English


Bigfile is a file transfer system, supports http, ftp and rpc protocol. Designed to provide a file management service and give developers more help. At the bottom, bigfile splits the file into small pieces of 1MB, the same slice will only be stored once.

In fact, we built a file organization system based on the database. Here you can find familiar files and folders.

Since the rpc and http protocols are supported, those languages supported by grpc and other languages can be quickly accessed.

More detailed documents can be found here

Features

  • Support HTTP(s) protocol

    • Support rate limit by ip
    • Support cors
    • Support to avoid replay attack
    • Support to validate parameter signature
    • Support Http Single Range Protocol
  • Support FTP(s) protocol

  • Support RPC protocol

  • Support to deploy by docker

  • Provide document with English and Chinese

Install Bigfile

There are kinds of ways to install Bigfile, you can find more detailed documentation here English 简体中文

Start Bigfile

  1. generate certificates

bigfile rpc:make-cert

  1. start server

bigfile multi:server

This will print some information as follows:

    [2019/09/19 15:38:32.817] 56628 DEBUG  bigfile http service listening on: https://0.0.0.0:10985
    [2019/09/19 15:38:32.818] 56628 DEBUG   Go FTP Server listening on 2121
    [2019/09/19 15:38:32.819] 56628 DEBUG  bigfile rpc service listening on: tcp://[::]:10986

HTTP Example (Token Create)

package main

import (
	"fmt"
	"io/ioutil"
	libHttp "net/http"
	"strings"
	"time"

	"github.com/bigfile/bigfile/databases/models"
	"github.com/bigfile/bigfile/http"
)

func main() {
	appUid := "42c4fcc1a620c9e97188f50b6f2ab199"
	appSecret := "f8f2ae1fe4f70b788254dcc991a35558"
	body := http.GetParamsSignBody(map[string]interface{}{
		"appUid":         appUid,
		"nonce":          models.RandomWithMD5(128),
		"path":           "/images/png",
		"expiredAt":      time.Now().AddDate(0, 0, 2).Unix(),
		"secret":         models.RandomWithMD5(44),
		"availableTimes": -1,
		"readOnly":       false,
	}, appSecret)
	request, err := libHttp.NewRequest(
		"POST", "https://127.0.0.1:10985/api/bigfile/token/create", strings.NewReader(body))
	if err != nil {
		fmt.Println(err)
		return
	}
	request.Header.Set("Content-Type", "application/x-www-form-urlencoded")
	resp, err := libHttp.DefaultClient.Do(request)
	if err != nil {
		fmt.Println(err)
		return
	}
	if bodyBytes, err := ioutil.ReadAll(resp.Body); err != nil {
		fmt.Println(err)
		return
	} else {
		fmt.Println(string(bodyBytes))
	}
}

RPC Example (Token Create)

package main

import (
	"crypto/tls"
	"crypto/x509"
	"fmt"
	"io/ioutil"

	"google.golang.org/grpc"
	"github.com/bigfile/bigfile/rpc"
	"google.golang.org/grpc/credentials"
)

func createConnection() (*grpc.ClientConn, error) {
	var (
		err           error
		conn          *grpc.ClientConn
		cert          tls.Certificate
		certPool      *x509.CertPool
		rootCertBytes []byte
	)
	if cert, err = tls.LoadX509KeyPair("client.pem", "client.key"); err != nil {
		return nil, err
	}

	certPool = x509.NewCertPool()
	if rootCertBytes, err = ioutil.ReadFile("ca.pem"); err != nil {
		return nil, err
	}

	if !certPool.AppendCertsFromPEM(rootCertBytes) {
		return nil, err
	}

	if conn, err = grpc.Dial("192.168.0.103:10986", grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{
		Certificates: []tls.Certificate{cert},
		RootCAs:      certPool,
	}))); err != nil {
		return nil, err
	}
	return conn, err
}

func main() {
	var (
		err  error
		conn *grpc.ClientConn
	)

	if conn, err = createConnection(); err != nil {
		fmt.Println(err)
		return
	}
	defer conn.Close()
	grpcClient := rpc.NewTokenCreateClient(conn)
    	fmt.Println(grpcClient.TokenCreate(context.TODO(), &rpc.TokenCreateRequest{
    		AppUid:    "42c4fcc1a620c9e97188f50b6f2ab199",
    		AppSecret: "f8f2ae1fe4f70b788254dcc991a35558",
    }))
}

FTP Example

ftp-example

FOSSA Status Get it from the Snap Store

You might also like...
A rudimentary go program that allows you to mount a mongo database as a FUSE file system

This is a rudimentary go program that allows you to mount a mongo database as a

Gokrazy mkfs: a program to create an ext4 file system on the gokrazy perm partition

gokrazy mkfs This program is intended to be run on gokrazy only, where it will c

The best HTTP Static File Server, write with golang+vue
The best HTTP Static File Server, write with golang+vue

gohttpserver Goal: Make the best HTTP File Server. Features: Human-friendly UI, file uploading support, direct QR-code generation for Apple & Android

ZipFly: a golang HTTP server that streams a ZIP file from a list of URLs extracted from a JSON manifest

ZipFly, streaming files as a ZIP like a 🚀 ZipFly is a golang HTTP server that s

QueryCSV enables you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to a CSV file
QueryCSV enables you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to a CSV file

QueryCSV enable you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to CSV file

A basic file server automatically generates self certificates and serves the given folder.

A basic file server automatically generates self certificates and serves the given folder.

File uploader with support for multiple hosts and progress reporting written in Go.
File uploader with support for multiple hosts and progress reporting written in Go.

go-upload File uploader with support for multiple hosts and progress reporting written in Go. Windows, Linux, macOS and Android binaries Usage Upload

Provide an upload endpoint that stores files on pinata and returns a json response with the uploaded file pinata url

Purpose Build a template repository to get to coding as quickly as possible, by starting from a common template which follows the guidelines here Feat

Extract profiles and tasks information from CSV file

Footsite-Bot ideas from jw6602 Extract profiles and tasks information from CSV f

Comments
  • 生成证书出错

    生成证书出错

    Describe the bug A clear and concise description of what the bug is.

    • go version: no
    • bigfile version (or commit ref): no
    • operating system: no
    • mysql version no
    • docker image version: v1.0.2

    Description

    运行docker run -v $(pwd):/certs bigfile/bigfile rpc:make-cert --save-to /certs命令时没有生成证书

    Screenshots

    image

    good first issue 
    opened by fnoopv 8
  • Add license scan report and status

    Add license scan report and status

    Your FOSSA integration was successful! Attached in this PR is a badge and license report to track scan status in your README.

    Below are docs for integrating FOSSA license checks into your CI:

    opened by fossabot 1
  • Error 1264: Out of range value for column 'size' at row 1

    Error 1264: Out of range value for column 'size' at row 1

    Describe the bug

    FTP 中移动文件时,会爆错

    • go version: 1.12
    • bigfile version (or commit ref): v1.0.7
    • operating system: MacOS?
    • mysql version: 5.7.21
    • cpu arch: amd64
    • If you use Bigfile in Docker, docker version:
     docker version
    Client: Docker Engine - Community
     Version:           19.03.2
     API version:       1.40
     Go version:        go1.12.8
     Git commit:        6a30dfc
     Built:             Thu Aug 29 05:26:49 2019
     OS/Arch:           darwin/amd64
     Experimental:      false
    
    Server: Docker Engine - Community
     Engine:
      Version:          19.03.2
      API version:      1.40 (minimum version 1.12)
      Go version:       go1.12.8
      Git commit:       6a30dfc
      Built:            Thu Aug 29 05:32:21 2019
      OS/Arch:          linux/amd64
      Experimental:     true
     containerd:
      Version:          v1.2.6
      GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
     runc:
      Version:          1.0.0-rc8
      GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
     docker-init:
      Version:          0.18.0
      GitCommit:        fec3683
    

    Description

    移动文件或同时上传多个文件时会报错

    Screenshots

    image

    bug 
    opened by gamelife1314 0
  • 下载源码编译报错

    下载源码编译报错

    Describe the bug A clear and concise description of what the bug is.

    • go version: 1.16
    • bigfile version (or commit ref): v1.0.5?
    • operating system: MacOS
    • mysql version: 5.7.x
    • cpu arch: amd64
    • If you use Bigfile in Docker, docker version:
     
    
    
    ## Description
    
    Package GraphicsMagickWand was not found in the pkg-config search path.
    Perhaps you should add the directory containing `GraphicsMagickWand.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'GraphicsMagickWand' found
    pkg-config: exit status 1
    
    
    ## Screenshots
    
    bug 
    opened by LOFTERART 0
Releases(v1.0.10)
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).

中文 English 愿景:为用户提供最简单、可靠、高效的分布式文件系统。 go-fastdfs是一个基于http协议的分布式文件系统,它基于大道至简的设计理念,一切从简设计,使得它的运维及扩展变得更加简单,它具有高性能、高可靠、无中心、免维护等优点。 大家担心的是这么简单的文件系统,靠不靠谱,可不

小张 3.3k Nov 23, 2022
Dragonfly is an intelligent P2P based image and file distribution system.

Dragonfly Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in o

dragonflyoss 6k Nov 26, 2022
File system event notification library on steroids.

notify Filesystem event notification library on steroids. (under active development) Documentation godoc.org/github.com/rjeczalik/notify Installation

Rafal Jeczalik 773 Nov 30, 2022
Pluggable, extensible virtual file system for Go

vfs Package vfs provides a pluggable, extensible, and opinionated set of file system functionality for Go across a number of file system types such as

C2FO 204 Nov 21, 2022
Cross-platform file system notifications for Go.

File system notifications for Go fsnotify utilizes golang.org/x/sys rather than syscall from the standard library. Ensure you have the latest version

fsnotify 7.7k Dec 1, 2022
Plik is a scalable & friendly temporary file upload system ( wetransfer like ) in golang.

Want to chat with us ? Telegram channel : https://t.me/plik_root_gg Plik Plik is a scalable & friendly temporary file upload system ( wetransfer like

root.gg 1.1k Nov 21, 2022
File system for GitHub

HUBFS · File System for GitHub HUBFS is a read-only file system for GitHub and Git. Git repositories and their contents are represented as regular dir

Bill Zissimopoulos 1.6k Nov 25, 2022
A virtual file system for small to medium sized datasets (MB or GB, not TB or PB). Like Docker, but for data.

AetherFS assists in the production, distribution, and replication of embedded databases and in-memory datasets. You can think of it like Docker, but f

mya 8 Feb 9, 2022
GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go

GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go Overview GeeseFS allows you to mount an S3 bucket as a file sys

Yandex.Cloud 309 Nov 21, 2022
Encrypted File System in Go

Getting Started: Setup the environment: Install GoLang: $ sudo apt update $ sudo apt upgrade $ sudo apt install libssl-dev gcc pkg-config $ sudo apt

Lucky Verma 0 Apr 30, 2022