Tarserv serves streaming tar files from filesystem snapshots.

Overview

tarserv

A collection of tools that allow serving large datasets from local filesystem snapshots.

It is meant for serving big amounts of data to shell scripted receivers.

Process:

  1. Take snapshot of directory containing data "/var/datadir/.snapshot12345".
  2. Create build index of the snapshot and store it into index directory:
    $ createindex /var/index/snapshot12345.taridx /var/datadir/.snapshot12345
  3. Serve snapshots via HTTP: $ tarserv -l 127.0.0.1:8080 -i /var/index/
  4. Download snapshots via HTTP: $ curl http://127.0.0.1:8080/snapshot12345/data.tar

Features:

  • The tar files are created on the fly from the index file and filesystem snapshot/directory.
  • Downloading is resilient, it allows range requests and so can be efficiently restarted.
  • Furthermore the tar files produced contain a file ".version" that contains the source snapshot name.
  • If given the "lastfile" GET parameter, the served snapshot will start with the file named by the parameter.
  • All contained paths are normalized to current directory "./".
Owner
Aurora
Scaling Solution for Ethereum - Built on NEAR
Aurora
Vaala archive is a tar archive tool & library optimized for lots of small files.

?? Vaar ?? Vaala archive is a tar archive tool & library optimized for lots of small files. Written in Golang, vaar performs operations in parallel &

Qing Moy 14 Apr 4, 2022
Read a tar file contents using go1.16 io/fs abstraction

go-tarfs Read a tar file contents using go1.16 io/fs abstraction Usage ⚠️ go-tarfs needs go>=1.16 Install: go get github.com/nlepage/go-tarfs Use: pac

Nicolas Lepage 17 Mar 26, 2022
A basic file server automatically generates self certificates and serves the given folder.

A basic file server automatically generates self certificates and serves the given folder.

Ahmet ÖZER 3 Oct 30, 2021
A Go io/fs filesystem implementation for reading files in a Github gists.

GistFS GistFS is an io/fs implementation that enables to read files stored in a given Gist. Requirements This module depends on io/fs which is only av

Jean Hadrien Chabran 124 May 1, 2022
A Go filesystem package for working with files and directories

Stowage A Go filesystem package for working with files and directories, it features a simple API with support for the common files and directories ope

null 19 May 28, 2021
Grep archive search in any files on the filesystem, in archive and even inner archive.

grep-archive Grep archive search for string in any files on the filesystem, in archive and even inner archive. Supported archive format are : Tar Form

Michel Prunet 0 Jan 26, 2022
s3fs provides a S3 implementation for Go1.16 filesystem interface.

S3 FileSystem (fs.FS) implementation.Since S3 is a flat structure, s3fs simulates directories by using prefixes and "/" delim. ModTime on directories is always zero value.

Jacek Szwec 130 May 17, 2022
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Steve Francia 4.4k May 17, 2022
A package to allow one to concurrently go through a filesystem with ease

skywalker Skywalker is a package to allow one to concurrently go through a filesystem with ease. Features Concurrency BlackList filtering WhiteList fi

Will Dixon 79 May 6, 2022
Takes an input http.FileSystem (likely at go generate time) and generates Go code that statically implements it.

vfsgen Package vfsgen takes an http.FileSystem (likely at go generate time) and generates Go code that statically implements the provided http.FileSys

null 943 Apr 29, 2022
memfs: A simple in-memory io/fs.FS filesystem

memfs: A simple in-memory io/fs.FS filesystem memfs is an in-memory implementation of Go's io/fs.FS interface. The goal is to make it easy and quick t

Peter Sanford 60 May 21, 2022
A Small Virtual Filesystem in Go

This is a virtual filesystem I'm coding to teach myself Go in a fun way. I'm documenting it with a collection of Medium posts that you can find here.

Alyson 31 Apr 18, 2022
CRFS: Container Registry Filesystem

CRFS: Container Registry Filesystem Discussion: https://github.com/golang/go/issues/30829 Overview CRFS is a read-only FUSE filesystem that lets you m

Google 1.2k May 19, 2022
Encrypted overlay filesystem written in Go

An encrypted overlay filesystem written in Go. Official website: https://nuetzlich.net/gocryptfs (markdown source). gocryptfs is built on top the exce

null 2.3k May 20, 2022
Go filesystem implementations for various URL schemes

hairyhenderson/go-fsimpl This module contains a collection of Go filesystem implementations that can discovered dynamically by URL scheme. All filesys

Dave Henderson 223 May 14, 2022
filesystem for golang

filesystem filesystem for golang installation go get github.com/go-component/filesystem import import "github.com/go-component/filesystem" Usage sup

null 4 Jul 9, 2021
A set of io/fs filesystem abstractions and utilities for Go

A set of io/fs filesystem abstractions and utilities for Go Please ⭐ this project Overview This package provides io/fs interfaces for: Cloud providers

null 8 Mar 24, 2022
Split text files into gzip files with x lines

hakgzsplit split lines of text into multiple gzip files

Luke Stephens (hakluke) 7 Apr 29, 2022