A simple daemon which will watch files on your filesystem, mirror them to MFS, automatically update related pins, and update related IPNS keys.

Overview

ipfs-sync

Go Reference

Note: This software is very young. If you discover any bugs, please report them via the issue tracker.

ipfs-sync is a simple daemon which will watch files on your filesystem, mirror them to MFS, automatically update related pins, and update related IPNS keys, so you can always access your directories from the same address. You can use it to sync your documents, photos, videos, or even a website!

Buy Me A Coffee

Installation

Binary

If you're on an Arch based distro, ipfs-sync is available on the AUR.

Binaries are available on the releases page for other distros.

Source

You need go installed, with a working GOPATH and $GOPATH/bin should be added to your $PATH to execute the command.

go install github.com/TheDiscordian/ipfs-sync

Usage

The only required parameter is dirs, which can be specified in the config file, or as an argument. The ID parameter is simply a unique idenifier for you to remember, the IPNS key will be generated using this ID.

Usage of ipfs-sync:
  -basepath string
        relative MFS directory path (default "/ipfs-sync/")
  -config string
        path to config file to use
  -copyright
        display copyright and exit
  -db string
        path to file where db should be stored (example: "/home/user/.ipfs-sync/hashes.db")
  -dirs value
        set the dirs to monitor in json format like: [{"ID":"Example1", "Dir":"/home/user/Documents/"},{"ID":"Example2", "Dir":"/home/user/Pictures/"}]
  -endpoint string
        node to connect to over HTTP (default "http://127.0.0.1:5001")
  -ignore value
        set the suffixes to ignore (default: ["kate-swp", "swp", "part"])
  -ignorehidden
        ignore anything prefixed with "."
  -sync duration
        time to sleep between IPNS syncs (ex: 120s) (default 10s)

ipfs-sync can be setup and used as a service. Simply point it to a config file, and restart it whenever the config is updated. An example config file can be found at config.json.sample.

Example

Getting started is simple. The only required field is dirs, so if we wanted to sync a folder, we'd simply run:

ipfs-sync -dirs '[{"ID":"ExampleID", "Dir":"/home/user/Documents/ExampleFolder/"}]'
2021/02/12 18:03:38 ipfs-sync starting up...
2021/02/12 18:03:38 ExampleID not found, generating...
2021/02/12 18:03:38 Adding file to /ipfs-sync/ExampleFolder/index.html ...
2021/02/12 18:04:40 ExampleID loaded: k51qzi5uqu5dlpvinw1zhxzo4880ge5hg9tp3ao4ye3aujdru9rap2h7izk5lm

This command will first check if there's a key named ExampleID in ipfs-sync's namespace, if not, it'll generate and return one. In this example, it synced a simple website to k51qzi5uqu5dlpvinw1zhxzo4880ge5hg9tp3ao4ye3aujdru9rap2h7izk5lm. As you add/remove/change files in the directory now, they'll be visible live at that address.

Comments
  • Filestore cleanup is too slow

    Filestore cleanup is too slow

    ~~Maybe we can speed it up by only removing offset 0 references, see if that fixes filestore errors just as well, and we can let the gc clean up the rest later.~~

    performance 
    opened by TheDiscordian 8
  • Multiple hashes for same file

    Multiple hashes for same file

    I made a directory called website and added 3 files in it - file1, file2 ad file3

    I added some text to them.

    I used ipfs-sync to add them.

    Then I removed file1 and edited file2 to add one more line.

    This is the output I am getting after that:

    (base) [email protected]:~/APPS/ipfs-sync-linux64$ ipfs key list self ipfs-sync.mywebsite

    (base) [email protected]:~/APPS/ipfs-sync-linux64$ ipfs pin ls --type=all QmbNTPp3wgU4QDxjsJ9v7aNBm3grwkHfno57YjA3Gj8buG indirect Qme4y8z5nxZd3g4uXPyzUWjthDXoDjTwsPPRseNE6qzETC recursive QmejyB5JSYNMcJeQbXuPj4W1DM23DxsWrU42JQFqy3Z7Xe indirect QmQmtJC2QmhpZiX3cp1PTK3AL4xjbF25jDyJ6RPjqXN7AV indirect QmYCJD2Fp7PMN4xoYeGNz8wDzYtPaB1tuBD8XszmzTg5mq indirect QmYNc7WdiByB2Li6717UgAcdRXjnMSbDX8FHJLwVWChzcA indirect

    (base) [email protected]:~/APPS/ipfs-sync-linux64$ ipfs cat QmbNTPp3wgU4QDxjsJ9v7aNBm3grwkHfno57YjA3Gj8buG This is file 2 This is a second edit

    (base) [email protected]:~/APPS/ipfs-sync-linux64$ ipfs cat Qme4y8z5nxZd3g4uXPyzUWjthDXoDjTwsPPRseNE6qzETC Error: this dag node is a directory

    (base) [email protected]:~/APPS/ipfs-sync-linux64$ ipfs cat QmejyB5JSYNMcJeQbXuPj4W1DM23DxsWrU42JQFqy3Z7Xe Manuj: No Output - seems like file1 is gone but its hash is still pinned?

    (base) [email protected]:~/APPS/ipfs-sync-linux64$ ipfs cat QmQmtJC2QmhpZiX3cp1PTK3AL4xjbF25jDyJ6RPjqXN7AV This is file 2 This is a second edit

    (base) [email protected]:~/APPS/ipfs-sync-linux64$ ipfs cat QmYCJD2Fp7PMN4xoYeGNz8wDzYtPaB1tuBD8XszmzTg5mq This is file 3

    (base) [email protected]:~/APPS/ipfs-sync-linux64$ ipfs cat QmYNc7WdiByB2Li6717UgAcdRXjnMSbDX8FHJLwVWChzcA This is file 3

    Notice that hash QmbNTPp3wgU4QDxjsJ9v7aNBm3grwkHfno57YjA3Gj8buG and QmQmtJC2QmhpZiX3cp1PTK3AL4xjbF25jDyJ6RPjqXN7AV are both showing the contents of file2

    and QmYCJD2Fp7PMN4xoYeGNz8wDzYtPaB1tuBD8XszmzTg5mq and QmYNc7WdiByB2Li6717UgAcdRXjnMSbDX8FHJLwVWChzcA are showing the contents of file 3!

    Using the latest version available in release.

    help wanted needs investigation 
    opened by manujchandra 7
  • Filestore (nocopy) breaks horribly with quick renames

    Filestore (nocopy) breaks horribly with quick renames

    Currently filestore can break horribly if we do quick renames. For example:

    1. Download file to synced folder with a browser, file will be named Unconfirmed.xyz or something.
    2. After download completes, it'll rename to the desired filename

    What happens?

    Well filestore will initially store a reference for Unconfirmed.xyz before it gets totally removed from the blockstore, the rename occurs. The causes filestore to ignore the abspath passed in the header on the second add, and instead lookup the hash, and use the abspath used for Unconfirmed.xyz, which no longer exits. This causes errors trying to do just about anything.

    How do we fix it?

    On pin/update and sometimes files/cp we can get an error:

    failed to get block for bafkreidjs72a6h2quuhq7fl4q4aefqxdrioestsci2uoyvns3duzl4nv2q: open /home/user/Documents/IPFS Websites/DiscoNet/documents/riichi_mahjong/Unconfirmed 642720.crdownload: no such file or directory

    If we run a block/rm on the CID above, the error is resolved, we simply need to re-add the effected file after, then attempt to update the pin again.

    Not totally clear on how we locate the effected file in the pin/update scenario, I suppose we'll just have to hope that doesn't happen, and if it does, a user can re-add the file by simply renaming it to something, then back again. So for this scenario, we'll just remove the block, and hope filestore sorts itself out.

    bug 
    opened by TheDiscordian 4
  • Syncing randomly ends with

    Syncing randomly ends with "Killed"

    While trying to sync a fairly large (7GB, 46 file) directory, the process ends in the middle of syncing with a SIGKILL. Note that this was an own compiled executable created with GOOS=linux GOARCH=arm64 go build on a different system.

    needs investigation 
    opened by ravenclaw900 3
  • Estuary Support

    Estuary Support

    Using Estuary users could store data long-term for free using the Filecoin network. I think this would be a nice alternative to #42 (but don't worry, both issues are accepted).

    enhancement 
    opened by TheDiscordian 2
  • panic: runtime error: invalid memory address or nil pointer dereference

    panic: runtime error: invalid memory address or nil pointer dereference

    Last login: Tue Jun  1 12:48:29 on ttys000
    [email protected] ~ % go install github.com/TheDiscordian/[email protected]
    go: downloading github.com/TheDiscordian/ipfs-sync v0.5.0
    go: downloading github.com/fsnotify/fsnotify v1.4.9
    go: downloading github.com/cespare/xxhash v1.1.0
    go: downloading github.com/syndtr/goleveldb v1.0.0
    go: downloading gopkg.in/yaml.v2 v2.4.0
    go: downloading golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9
    go: downloading github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
    [email protected] ~ % brew reinstall libexec
    Error: No available formula or cask with the name "libexec".
    [email protected] ~ % ipfs-sync -config $HOME/.ipfs-sync.yaml -db $HOME/.ipfs-sync.db
    2021/06/01 13:13:16 ipfs-sync starting up...
    2021/06/01 13:13:16 Loading config file /Users/williamdavis/.ipfs-sync.yaml
    2021/06/01 13:13:16 Starting up ipfs-sync  ...
    2021/06/01 13:13:16 Starting watchdog...
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x2 addr=0x20 pc=0x10110dda8]
    
    goroutine 1 [running]:
    main.filePathWalkDir.func1(0x14000152800, 0x1f, 0x0, 0x0, 0x1012188f8, 0x140001175f0, 0x0, 0x169)
    	/Users/williamdavis/go/pkg/mod/github.com/!the!discordian/[email protected]/main.go:305 +0x28
    path/filepath.WalkDir(0x14000152800, 0x1f, 0x140006a5b40, 0x1011a23e0, 0x16)
    	/opt/homebrew/Cellar/go/1.16.4/libexec/src/path/filepath/path.go:463 +0x70
    main.filePathWalkDir(0x14000152800, 0x1f, 0x30, 0x140001175c0, 0x14000080480, 0x0, 0x0)
    	/Users/williamdavis/go/pkg/mod/github.com/!the!discordian/[email protected]/main.go:304 +0x58
    main.HashDir(0x14000152800, 0x1f, 0x101173500, 0x1, 0x0, 0x2)
    	/Users/williamdavis/go/pkg/mod/github.com/!the!discordian/[email protected]/db.go:100 +0x30
    main.WatchDog()
    	/Users/williamdavis/go/pkg/mod/github.com/!the!discordian/[email protected]/main.go:658 +0x648
    main.main()
    	/Users/williamdavis/go/pkg/mod/github.com/!the!discordian/[email protected]/main.go:751 +0x108
    [email protected] ~ % go unnstall github.com/TheDiscordian/[email protected]
    go unnstall: unknown command
    Run 'go help' for usage.
    [email protected] ~ % go uninstall github.com/TheDiscordian/[email protected]
    go uninstall: unknown command
    Run 'go help' for usage.
    [email protected] ~ % go help
    Go is a tool for managing Go source code.
    
    Usage:
    
    	go <command> [arguments]
    
    The commands are:
    
    	bug         start a bug report
    	build       compile packages and dependencies
    	clean       remove object files and cached files
    	doc         show documentation for package or symbol
    	env         print Go environment information
    	fix         update packages to use new APIs
    	fmt         gofmt (reformat) package sources
    	generate    generate Go files by processing source
    	get         add dependencies to current module and install them
    	install     compile and install packages and dependencies
    	list        list packages or modules
    	mod         module maintenance
    	run         compile and run Go program
    	test        test packages
    	tool        run specified go tool
    	version     print Go version
    	vet         report likely mistakes in packages
    
    Use "go help <command>" for more information about a command.
    
    Additional help topics:
    
    	buildconstraint build constraints
    	buildmode       build modes
    	c               calling between Go and C
    	cache           build and test caching
    	environment     environment variables
    	filetype        file types
    	go.mod          the go.mod file
    	gopath          GOPATH environment variable
    	gopath-get      legacy GOPATH go get
    	goproxy         module proxy protocol
    	importpath      import path syntax
    	modules         modules, module versions, and more
    	module-get      module-aware go get
    	module-auth     module authentication using go.sum
    	packages        package lists and patterns
    	private         configuration for downloading non-public code
    	testflag        testing flags
    	testfunc        testing functions
    	vcs             controlling version control with GOVCS
    
    Use "go help <topic>" for more information about that topic.
    
    [email protected] ~ % go list
    go: go.mod file not found in current directory or any parent directory; see 'go help modules'
    [email protected] ~ % cd /Users/williamdavis/go/pkg/mod/github.com/\!the\!discordian/[email protected]
    [email protected] [email protected] % go list
    github.com/TheDiscordian/ipfs-sync
    [email protected] [email protected] % brew install systemd
    Updating Homebrew...
    ==> Auto-updated Homebrew!
    Updated 3 taps (homebrew/cask-versions, homebrew/core and homebrew/cask).
    ==> Updated Formulae
    Updated 2 formulae.
    
    systemd: Linux is required for this software.
    libcap: Linux is required for this software.
    Error: Unsatisfied requirements failed this build.
    [email protected] [email protected] % brew install systemd -f
    systemd: Linux is required for this software.
    libcap: Linux is required for this software.
    Error: Unsatisfied requirements failed this build.
    [email protected] [email protected] % cd
    [email protected] ~ % go install github.com/TheDiscordian/[email protected]
    go: downloading github.com/TheDiscordian/ipfs-sync v0.5.0
    go: downloading github.com/fsnotify/fsnotify v1.4.9
    go: downloading github.com/cespare/xxhash v1.1.0
    go: downloading github.com/syndtr/goleveldb v1.0.0
    go: downloading gopkg.in/yaml.v2 v2.4.0
    go: downloading golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9
    go: downloading github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
    [email protected] ~ % ipfs-sync
    2021/06/01 13:17:36 ipfs-sync starting up...
    2021/06/01 13:17:36 dirs field is required as flag, or in config (ex: {"ID":"UniqueIdentifier", "Dir":"/path/to/dir/to/sync/", "Nocopy": false}).
    [email protected] ~ % ipfs-sync -config $HOME/.ipfs-sync.yaml          
    2021/06/01 13:17:46 ipfs-sync starting up...
    2021/06/01 13:17:46 Loading config file /Users/williamdavis/.ipfs-sync.yaml
    2021/06/01 13:17:47 Starting up ipfs-sync  ...
    2021/06/01 13:17:47 Starting watchdog...
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x2 addr=0x20 pc=0x101109da8]
    
    goroutine 1 [running]:
    main.filePathWalkDir.func1(0x1400015a800, 0x1f, 0x0, 0x0, 0x1012148f8, 0x140000123c0, 0x0, 0x169)
    	/Users/williamdavis/go/pkg/mod/github.com/!the!discordian/[email protected]/main.go:305 +0x28
    path/filepath.WalkDir(0x1400015a800, 0x1f, 0x14000165b40, 0x10119e3e0, 0x16)
    	/opt/homebrew/Cellar/go/1.16.4/libexec/src/path/filepath/path.go:463 +0x70
    main.filePathWalkDir(0x1400015a800, 0x1f, 0x30, 0x14000012390, 0x14000044900, 0x0, 0x0)
    	/Users/williamdavis/go/pkg/mod/github.com/!the!discordian/[email protected]/main.go:304 +0x58
    main.HashDir(0x1400015a800, 0x1f, 0x10116f500, 0x1, 0x0, 0x2)
    	/Users/williamdavis/go/pkg/mod/github.com/!the!discordian/[email protected]/db.go:100 +0x30
    main.WatchDog()
    	/Users/williamdavis/go/pkg/mod/github.com/!the!discordian/[email protected]/main.go:658 +0x648
    main.main()
    	/Users/williamdavis/go/pkg/mod/github.com/!the!discordian/[email protected]/main.go:751 +0x108
    [email protected] ~ % 
    
    bug good first issue 
    opened by unbeatable-101 2
  • Do less `files/mkdir` calls

    Do less `files/mkdir` calls

    Related to #28, currently we call files/mkdir for every single file updated/added. It'd speed up these operations a lot if we only did this once per directory.

    For context I just attempted to update 18k files totalling ~1.1GiB, it's taking over 12 hours on a powerful system with an NVME SSD. This is obviously not good enough.

    Each call can take 1-3s!!

    enhancement performance 
    opened by TheDiscordian 2
  • Remove subdirectories and files from DB upon deletion

    Remove subdirectories and files from DB upon deletion

    Currently if you delete a directory, or move it out of the watched folder onto the same drive, it won't delete the contents from the DB.

    So for example, you move the directory out. It gets removed from MFS. However you move it back, and it doesn't get re-added, as the DB decides all the files are already there.

    We're using LevelDB, and our keys are just paths, we should be able to filter all the relevant keys pretty easily, and remove them.

    bug 
    opened by TheDiscordian 2
  • could not resolve name

    could not resolve name

    When I tried to sync a directory. It returned error- ipfs-sync -dirs "[{"ID":"Example1","Dir":"/home/Users/Deepak Dash/Desktop/Test_go1"}]"

    cmd name/resolve?arg=k51qzi5uqu5djumqhkhcjt8q5yof4h97usq3ldzb3lg1iw8q3cdbojuo8egagj
    {"Message":"could not resolve name","Code":0,"Type":"error"}
    
    panic: runtime error: index out of range [2] with length 1
    
    goroutine 1 [running]:
    main.ResolveIPNS(0xc0001920c0, 0x3e, 0xc00011de78, 0x2)
            C:/Users/Deepak Dash/go/src/github.com/TheDiscordian/ipfs-sync/main.go:432 +0x367
    main.WatchDog()
            C:/Users/Deepak Dash/go/src/github.com/TheDiscordian/ipfs-sync/main.go:490 +0x80c
    main.main()
            C:/Users/Deepak Dash/go/src/github.com/TheDiscordian/ipfs-sync/main.go:634 +0x294
    

    When Checked I could see that http.client is nil. http.client = &{ 0

    Could you let me know if I have done something wrong ?

    question 
    opened by Deepak-dash007 2
  • Linux Binary GLIBC_2.32 error with Ubuntu 20.04LTS

    Linux Binary GLIBC_2.32 error with Ubuntu 20.04LTS

    When: ./ipfs-sync

    it returns:

    ./ipfs-sync: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./ipfs-sync)

    I searched for this error and it seems very hard/impossible to have GLIBC_2.32 for Ubuntu 20.04, some users recommends to upgrade to 20.10 to have GLIBC_2.32 support. Since 20.10 is not LTS I don't use it for a server. Possibly we can have Ubuntu 20.04 support for the Linux binary? Thank you.

    bug 
    opened by jurjendevries 2
  • Make Pinning Optional

    Make Pinning Optional

    Pinning is added complexity that can break (#37), pinning should be optional as the data's already safe from GC because of MFS, which is the primary goal of pinning anyways.

    The feature must be optional though, as users may want to use ipfs-sync with a pinning service.

    enhancement performance 
    opened by TheDiscordian 1
  • Wishlist: Make more IPFS options available

    Wishlist: Make more IPFS options available

    @TheDiscordian this sounds really great!

    I really like to test this out, but I would need a couple of options, so I thought I could maybe post a wishlist 🥺

    Ability to

    • set blake2b-256 as hash algorithm
    • select rabin as chunker (with min/max/avg)
    • use only CIDv1 (not sure if that's not already the case)
    • use raw-leaves

    Additionally, I think I would need a way to trigger the whole sync process manually, as I would fetch a lot of data via rsync in irregular intervals. After the rsync process has completed its work I would like to notify ipfs-sync to start pushing the changes and terminate afterwards (or just pause the sync and wait for the next signal).

    So maybe the easiest way would be to have it listen to SIGHUP or something like that. So I can send a signal vial kill to start syncing.

    opened by RubenKelevra 0
  • Use variables in `.ipfs-sync.yaml`

    Use variables in `.ipfs-sync.yaml`

    I tried to pass the variables using env-cmd, but it didn't work, maybe there's a workaround? I want to pass the variable from my .env directly into the .yaml like ${{ env.FOLDER_TO_SYNC }}

    opened by T-Damer 0
  • error: fsnotify queue overflow

    error: fsnotify queue overflow

    I think it just kept accumulating these because it had to finish adding the other millions of files first. The best way would be to just keep these in one place and de-duplicate these fsnotify notifications before trying to process them.

    Also ipfs-sync tends to take up to 30GB of ram a lot of times.

    opened by RomeSilvanus 0
  • Ensure In-Use Blocks Aren't Removed

    Ensure In-Use Blocks Aren't Removed

    #38 was solved by calculating the CID of a file, then removing all the blocks related to that file upon error, then (if necessary), re-adding the file. I believe this shouldn't be an issue as files are re-added, but I don't think they're re-added in every scenario. Either way, this could be a tricky problem to solve, and I'm opening this issue because it likely isn't solved.

    filestore 
    opened by TheDiscordian 0
  • [Bug] Loop when files are being actively written to and FSNotify triggers

    [Bug] Loop when files are being actively written to and FSNotify triggers

    Should check for file activity or if file is open in another process before proceeding with add process

    Error message:

    2021/10/08 22:43:38 fsnotify event: "files/713.7z": WRITE 2021/10/08 22:43:38 Adding file from files/713.7z to /ipfs-sync/files/713.7z ... 2021/10/08 22:43:38 Generating file headers... 2021/10/08 22:43:38 Doing add request... 2021/10/08 22:43:38 File hash: bafkreih 2021/10/08 22:43:38 Removing existing file (if any)... 2021/10/08 22:43:38 Adding file to mfs path: /ipfs-sync/files/713.7z 2021/10/08 22:43:38 AddFile reply: bafkreih 2021/10/08 22:43:38 fsnotify event: "files/813.7z": WRITE 2021/10/08 22:43:38 Adding file from files/813.7z to /ipfs-sync/files/813.7z ... 2021/10/08 22:43:38 Generating file headers... 2021/10/08 22:43:38 Doing add request... 2021/10/08 22:43:38 File hash: bafkreih 2021/10/08 22:43:38 Removing existing file (if any)... 2021/10/08 22:43:38 Adding file to mfs path: /ipfs-sync/files/813.7z 2021/10/08 22:43:38 AddFile reply: bafkreih 2021/10/08 22:43:38 fsnotify event: "files/713.7z": WRITE 2021/10/08 22:43:38 Adding file from files/713.7z to /ipfs-sync/files/713.7z ... 2021/10/08 22:43:38 Generating file headers... 2021/10/08 22:43:38 Doing add request... 2021/10/08 22:43:38 File hash: bafkreihdw 2021/10/08 22:43:38 Removing existing file (if any)... 2021/10/08 22:43:38 Adding file to mfs path: /ipfs-sync/files/713.7z 2021/10/08 22:43:38 AddFile reply: bafkreih 2021/10/08 22:43:38 fsnotify event: "files/819.7z": WRITE 2021/10/08 22:43:38 Adding file from files/819.7z to /ipfs-sync/files/819.7z ...

    performance 
    opened by RomeSilvanus 0
Releases(v0.7.0)
Owner
I design and create software.
null
ChangeTower is intended to help you watch changes in webpages and get notified of any changes written in Go

ChangeTower is intended to help you watch changes in webpages and get notified of any changes written in Go

The Cats 34 Nov 17, 2022
BRUS - Parses your web server (e.g. nginx) log files and checks with GreyNoise how much noise your website is exposed to.

BRUS bbbbbb rrrrrr u u sssss b b r r u u s bbbbbb rrrrrr u u sssss b b r r u u s bbbbbb r r

dubs3c 1 May 29, 2022
mirror to https://android.googlesource.com/kernel/configs

Android Kernel Configs How are kernel config settings typically stored? When building the Linux kernel for a particular platform one usually begins by

null 0 Dec 8, 2021
Scraping medium blogs to make them loadable with shitty internet and have a pleasant reading experience

Unmedium This project is still WIP We all know medium right? A bunch of JS, wast

Marcel Schramm 2 Mar 20, 2022
Golog is a logger which support tracing and other custom behaviors out of the box. Blazing fast and simple to use.

GOLOG Golog is an opinionated Go logger with simple APIs and configurable behavior. Why another logger? Golog is designed to address mainly two issues

Damiano Petrungaro 37 Oct 2, 2022
A simple web service for storing text log files

logpaste A minimalist web service for uploading and sharing log files. Run locally go run main.go Run in local Docker container The Docker container a

Michael Lynch 257 Nov 21, 2022
Parametrized JSON logging library in Golang which lets you obfuscate sensitive data and marshal any kind of content.

Noodlog Summary Noodlog is a Golang JSON parametrized and highly configurable logging library. It allows you to: print go structs as JSON messages; pr

Gyoza Tech 42 Oct 27, 2022
System resource usage profiler tool which regularly takes snapshots of the memory and CPU load of one or more running processes so as to dynamically build up a profile of their usage of system resources.

Vegeta is a system resource usage tracking tool built to regularly take snapshots of the memory and CPU load of one or more running processes, so as to dynamically build up a profile of their usage of system resources.

Kartik 8 Jan 16, 2022
With this package you can create your own syslog server with your own handlers for different kind of syslog messages

Using this library you can easy implement your own syslog server that: Can listen on multiple UDP ports and unix domain sockets. Can pass parsed syslo

Michał Derkacz 67 Nov 9, 2022
List files and their creation, modification and access time on android

andfind List files and their access, modification and creation date on a Android

Tek 2 Jan 5, 2022
Cloudinsight Agent is a system tool that monitors system processes and services, and sends information back to your Cloudinsight account.

Cloudinsight Agent 中文版 README Cloudinsight Agent is written in Go for collecting metrics from the system it's running on, or from other services, and

cloudinsight-backup 368 Nov 3, 2022
Go package for reading from continously updated files (tail -f)

Go package for tail-ing files A Go package striving to emulate the features of the BSD tail program. t, err := tail.TailFile("/var/log/nginx.log", tai

HPE Helion 2.5k Nov 27, 2022
Lumberjack is a Go package for writing logs to rolling files.

Lumberjack is a Go package for writing logs to rolling files.

Go微服务 4 Feb 24, 2022
Peimports - based on golang's debug/pe this package gives quick access to the ordered imports of pe files with ordinal support

This code is almost entirely derived from the Go standard library's debug/pe package. It didn't provide access to ordinal based entries in the IAT and

Mike Wiacek 0 Jan 5, 2022
A helper tool to work with profile.proto (pprof) files

qpprof qpprof complements the pprof tool. Commands Use qpprof command --help to get more information. Flat aggregation Alternative flat aggregations a

Iskander (Alex) Sharipov 53 Sep 15, 2022
A version control system to manage large files.

ArtiVC ArtiVC (Artifacts Version Control) is a handy command-line tool for data versioning on cloud storage. With only one command, it helps you neatl

InfuseAI 218 Nov 10, 2022
A tool to list and diagnose Go processes currently running on your system

gops gops is a command to list and diagnose Go processes currently running on your system. $ gops 983 980 uplink-soecks go1.9 /usr/local/bin/u

Google 5.9k Nov 18, 2022
Monitor your network and internet speed with Docker & Prometheus

Stand-up a Docker Prometheus stack containing Prometheus, Grafana with blackbox-exporter, and speedtest-exporter to collect and graph home Internet reliability and throughput.

Jeff Geerling 1.2k Nov 18, 2022