Open Source Continuous File Synchronization

Overview

Syncthing


Latest Linux & Cross Build Latest Windows Build Latest Mac Build MPLv2 License CII Best Practices Go Report Card

Goals

Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers. We strive to fulfill the goals below. The goals are listed in order of importance, the most important one being the first. This is the summary version of the goal list - for more commentary, see the full Goals document.

Syncthing should be:

  1. Safe From Data Loss

    Protecting the user's data is paramount. We take every reasonable precaution to avoid corrupting the user's files.

  2. Secure Against Attackers

    Again, protecting the user's data is paramount. Regardless of our other goals we must never allow the user's data to be susceptible to eavesdropping or modification by unauthorized parties.

  3. Easy to Use

    Syncthing should be approachable, understandable and inclusive.

  4. Automatic

    User interaction should be required only when absolutely necessary.

  5. Universally Available

    Syncthing should run on every common computer. We are mindful that the latest technology is not always available to any given individual.

  6. For Individuals

    Syncthing is primarily about empowering the individual user with safe, secure and easy to use file synchronization.

  7. Everything Else

    There are many things we care about that don't make it on to the list. It is fine to optimize for these values, as long as they are not in conflict with the stated goals above.

Getting Started

Take a look at the getting started guide.

There are a few examples for keeping Syncthing running in the background on your system in the etc directory. There are also several GUI implementations for Windows, Mac and Linux.

Docker

To run Syncthing in Docker, see the Docker README.

Vote on features/bugs

We'd like to encourage you to vote on issues that matter to you. This helps the team understand what are the biggest pain points for our users, and could potentially influence what is being worked on next.

Getting in Touch

The first and best point of contact is the Forum. There is also an IRC channel, #syncthing on freenode (with a web client), for talking directly to developers and users. If you've found something that is clearly a bug, feel free to report it in the GitHub issue tracker.

Building

Building Syncthing from source is easy, and there's a guide that describes it for both Unix and Windows systems.

Signed Releases

As of v0.10.15 and onwards release binaries are GPG signed with the key D26E6ED000654A3E, available from https://syncthing.net/security.html and most key servers.

There is also a built in automatic upgrade mechanism (disabled in some distribution channels) which uses a compiled in ECDSA signature. macOS binaries are also properly code signed.

Documentation

Please see the Syncthing documentation site.

All code is licensed under the MPLv2 License.

Issues
  • Filesystem Notification - Continuation

    Filesystem Notification - Continuation

    This is the continuation of plouj's PR #2807 on a branch in the syncthing repo for ease of maintenance. Refer to that PR for general information what this is about and commits previous to https://github.com/syncthing/syncthing/commit/1d424d84609a40d2a11ac8e06cb87aac6b130629.

    frozen-due-to-age pr-merged 
    opened by imsodin 184
  • Support for file encryption (e.g. non-trusted servers)

    Support for file encryption (e.g. non-trusted servers)

    So I have had a look at BitTorrent sync, syncthing and alternatives and what I always wondered about was the possibility to not only sync between resources I own and trust, but also external resources/servers which I do NOT trust with my data, up to a certain extent.

    One way to do this is using ecryptfs or encfs, but this has many obvious downsides: it is not an interoperable solution (only works on Linux), the files are actually stored in encrypted form on the disk (even if the resource is trusted and this is not necessary, for instance because of the file system being encrypted already), etc.

    What I propose is somehow configuring nodes which are only sent the files in an encrypted format, with all file contents (and potentially file/directory names as well; or even permissions) being encrypted. This way, if I want to store my private files on a fast server in a datacenter to access them from anywhere, I could do this with syncthing without essentially giving up ownership of those files. I could also prevent that particular sync node from being allowed/able to make any changes to the files without me noticing.

    I realize that this requires a LOT of additional effort, but it would be a killer feature that seems to not be available in any other "private cloud" solution so far. What are your thoughts on this feature?

    EDIT: BitTorrent sync mentions a feature like this in their API docs: "Encryption secret API users can generate folder secrets with encrypted peer support. Encryption secrets are read-only. They make Sync data encrypted on the receiver’s side. Recipients can sync files, but they can’t see file content, and they can’t modify the files. Encryption secrets come in handy if you need to sync to an untrusted location." (from http://www.bittorrent.com/intl/de/sync/developers/api)

    enhancement 
    opened by Natanji 143
  • Inotify support

    Inotify support

    To notice changes more quickly.

    enhancement 
    opened by jpjp 116
  • all: Store pending devices and folders in database (fixes #7178)

    all: Store pending devices and folders in database (fixes #7178)

    Purpose

    As discussed in #5758 and mentioned on the forum, storing the pending (offered from remote but not yet added locally) devices and folders in the XML configuration is not a nice and scalable design. Instead, the information should live in the database, properly structured, and made available over dedicated API endpoints.

    This is also ground work and practice to finally approach an acceptable implementation of the prototype in #5758, extending the concept to other devices we have heard about in ClusterConfig messages. That is deliberately saved for another PR though.

    Testing

    All code paths have undergone extensive manual testing within an existing setup of four instances (one compiled from this PR). Especially regarding the clean-up of pending entries upon config changes, both online through the POST /rest/system/config REST API and offline by modifying the XML while Syncthing is not running. Notifications on the GUI look as before, API endpoints have been verified with curl.

    ~~Unit tests would be rather complicated and were discussed as not strictly needed considering how low the importance of the handled information is.~~ Unit tests included, with a few basic test cases.

    Documentation

    The envisioned API for this stuff is described in https://github.com/syncthing/docs/pull/498.

    The commit messages contain much of the rationale for each change, so I'd be happy to keep them intact instead of squash-merging in a single commit. If desired, I can rebase to clean out obvious fixup commits and end up with a nice patch series of self-contained changes.

    Breaking Changes For The User

    Existing <pendingDevice> / <pendingFolder> entries in the XML config are not carried over to the database. The corresponding notifications will disappear until the next connection attempt with the offering device. It has been discussed that the low importance of this information does not warrant a separate logic for config --> database info migrations. One possibility to minimize disruption would be to just keep the XML entries intact for some releases and hold back on commit 8ae24612397fea6182cfb3d6dcce592c78c6df5d until then.

    opened by acolomb 109
  • Option to not sync mtime

    Option to not sync mtime

    Trying to sync my android device and my NAS(armv5) both using syncthing v0.10.0:

    On the ntfs mount of the NAS the file /my/directory/.syncthing.example.test is created then I get:

    puller: final: chtimes /my/directory/.syncthing.example.test: invalid argument

    File /my/directory/example.test is not created. Syncthing continues with the next file but the same error happens for all the files in the directory.

    enhancement frozen-due-to-age 
    opened by otrag 108
  • GUI display of global changes

    GUI display of global changes

    Purpose

    This PR adds a button to the device section of the GUI that the user can click on to see all filesystem changes that originated/occurred on that machine. That is to say, not changed copied from the network. This should help people discover more easily what computer made a certain change that was propagated in larger P2P swarms.

    Screenshots

    image

    frozen-due-to-age 
    opened by nrm21 107
  • Folder isn't making progress

    Folder isn't making progress

    All nodes are on v0.10.2 GUI says

    9:35:58: Folder "default" isn't making progress - check logs for possible root cause. Pausing puller for 1m0s.
    9:36:41: Folder "PIA" isn't making progress - check logs for possible root cause. Pausing puller for 1m0s.
    

    In the log I see the "hash mismatch" info but the file is not changed during pull. It was probably the whole night in that state but i don't have more logs, I should probably increase the size... logs from the pulling node: http://alex-graf.de/public/st/no-progress.tar.gz

    Restarting also does not fix it

    opened by alex2108 104
  • lib/connections: Add KCP support (fixes #804)

    lib/connections: Add KCP support (fixes #804)

    This is mostly for benchmarks and testing.

    Seems to connect, and not crash immediately, which is good news.

    Known issues:

    1. No timeouts as part of the protocol, so we usually end up with a dead connection for 5-7.5 minutes until it times out. This could be addressed by always using the new connection that we get, as long as the priority is right
    2. Has some weirdness around closing connections. See https://github.com/xtaci/kcp-go/issues/18

    The diff is massive due to deps, so I suggest you pull the PR locally and diff inside the lib and cmd/syncthing dirs.

    frozen-due-to-age pr-merged 
    opened by AudriusButkevicius 91
  • gui: add advance config port mapping to gui

    gui: add advance config port mapping to gui

    Purpose

    https://github.com/syncthing/syncthing/issues/4824

    Gives the user the opportunity to display a link to web gui remote machines. The user can choose the port and set if the link will display.

    Testing

    I just check to see if the link was shown depending on the setting.

    Screenshots

    Screen Shot 2020-09-29 at 11 13 53 PM Screen Shot 2020-09-29 at 11 34 37 PM

    Documentation

    https://github.com/syncthing/docs/pull/573

    Authorship

    Your name and email will be added automatically to the AUTHORS file based on the commit metadata.

    opened by rjpruitt16 91
  • Memory and CPU usage is prohibitively high

    Memory and CPU usage is prohibitively high

    I've been testing syncthing across 3 machines, a laptop with 8GB of RAM and two NAS-style servers with 1GB and 2GB of RAM. My repositories have the following sizes:

    • 19 items, 536 KiB
    • 83471 items, 16.2 GiB
    • 181482 items, 387 GiB

    To sync these three repositories syncthing 0.9.0 uses a bit over 700MB of RAM and while syncing continuously pegs the CPU at 150% on all nodes.

    While the CPU usage I could manage during the initial sync the memory usage is simply too high. A typical NAS server like the two I have holds >4TB of storage. At the current level of usage that would require ~8GB of memory just for syncthing.

    Without looking at the code I assume an index is being kept in memory for all the repository contents. 700MB is 2.6kb per item on disk, which seems way too high. The index should only really need to store filename, parent, permissions, size and timestamp. On average (depends on filename sizes) that should only be 50-60 bytes per item which would only be 13MB. Moving that index to disk would also make a lot more sense.

    I assume the CPU usage is hashing lots of files. There I assume using librsync might be a better option.

    enhancement frozen-due-to-age 
    opened by pedrocr 86
  • chore: Quiet numerous DeepSource reported issues

    chore: Quiet numerous DeepSource reported issues

    Purpose

    Quiet DeepSource issues reported in https://deepsource.io/gh/syncthing/syncthing/run/1f029f7e-c9e4-40a1-9b59-d1a8a4d39f1f/go/

    Testing

    All tests pass.

    Comments

    I chose to quiet things like

    func (fs *errorFilesystem) Chmod(name string, mode FileMode) error { return fs.err }
    

    by prefixing it with:

    // skipcq: RVV-B0012 : parameter 'name' seems to be unused, consider removing or renaming it as _
    

    as I didn't want to rewrite someone else's code as:

    func (fs *errorFilesystem) Chmod(_ string, _ FileMode) error { return fs.err }
    

    without direction.

    opened by rasa 5
  • lib/model, lib/protocol: Don't generate encrypted filenames that are reserved on Windows (fixes #7808)

    lib/model, lib/protocol: Don't generate encrypted filenames that are reserved on Windows (fixes #7808)

    Purpose

    Avoid creating reserved filenames by trivial escaping.

    Testing

    I don't know? Hence this is draft. How do we test this?

    Badness

    This handles the practicalities, but not really all the metadata. If we already have announced a file under a given bad name, the next update will use a better name, but the old metadata will be around forever, causing a conflict if a new trusted device is set up and receives all metadata from the untrusted device. There will be two entries representing the same trusted-side file.

    opened by calmh 1
  • build(deps): bump github.com/lib/pq from 1.10.1 to 1.10.3

    build(deps): bump github.com/lib/pq from 1.10.1 to 1.10.3

    Bumps github.com/lib/pq from 1.10.1 to 1.10.3.

    Release notes

    Sourced from github.com/lib/pq's releases.

    v1.10.3

    • implement ConnPrepareContext/StmtQueryContext/StmtExecContext interfaces (context.Cancel() now ends connections)
    • Avoid type assertion to the same type
    • Fix build for illumos and solaris

    v1.10.2

    • fix TimeTZ with second offsets
    • fix GOOS compilation
    Commits
    • 756b4d7 Merge pull request #1053 from EnergySRE/patch-1
    • 8667c6b Merge pull request #1047 from michaelshobbs/feature/context-interfaces
    • 9fa33e2 implement ConnPrepareContext/StmtQueryContext/StmtExecContext interfaces
    • 2140507 Merge pull request #1054 from michaelshobbs/feature/gh-actions
    • e10fdd5 remove travis ci artifacts
    • 7da16d9 mv certs Makefile to certs dir and add explanation
    • b81abde update goimports formatting for go1.17
    • 1c85910 implement gh actions workflow
    • 6ede22e Clarify maintenance mode
    • 9e747ca Merge pull request #1044 from xhit/fix-build-solaris-illumos
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 0
  • build(deps): bump github.com/shirou/gopsutil/v3 from 3.21.4 to 3.21.8

    build(deps): bump github.com/shirou/gopsutil/v3 from 3.21.4 to 3.21.8

    Bumps github.com/shirou/gopsutil/v3 from 3.21.4 to 3.21.8.

    Commits
    • 5268453 Merge pull request #1126 from tklauser/sysconf-update
    • b764840 Update github.com/tklauser/go-sysconf to v0.3.9
    • 0d0659a Merge pull request #1122 from secDre4mer/master
    • 595a629 Merge pull request #1119 from tbarker25/process-fixes-1
    • 9248140 Wait for server connection to be established before checking
    • bc46619 Minor cleanups motivated by staticcheck warnings.
    • 5ce887d Make sure that Test_AllProcesses_cmdLine doesn't ignore failures.
    • 34cdfa2 Test_Connections currently fails intermittently on Linux (and maybe
    • d07af87 chore: Drop PROCESS_QUERY_INFORMATION support
    • f86a042 Merge pull request #1117 from tklauser/update-go-sysconf
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 0
  • notify: File mode change events not emitted on MacOS/FSEvents watcher

    notify: File mode change events not emitted on MacOS/FSEvents watcher

    Technically an issue in syncthing/notify. It seems like it doesn't generate events when changing file mode, which means permissions don't get synced after running chmod until the next scan.

    I've done a tiny bit of digging and it looks like those events should be generated by FSEvents. Thefsnotify package, which uses kqueue instead of FSEvents, does seem to emit chmod events.

    Version: Syncthing v1.18.1-1 macos OS: macOS BigSur x64 11.5.2

    Repro:

    • Have a synced folder
    • Create a file
    • Change file permissions using chmod

    Result:

    • File permissions are not synced until next scan or until file is modified e.g by touching it

    Expected:

    • File permissions will be synced after fsWatchDelay
    bug needs-triage 
    opened by yyogo 0
  • Connection not dropped and sync stalled when sharing folder first, and only then adding password

    Connection not dropped and sync stalled when sharing folder first, and only then adding password

    1. On Device 1, share a folder with Device 2. image image
    2. Only now, set a password for the folder on Device 1. image
    3. On Device 2, try to add the folder. The type is correctly detected and automatically set to Receive Encrypted. image
    4. On Device 2, the folder has been added and is marked as "Up to Date". On Device 1, the files are stuck in sync though. The connection itself has not been dropped after adding the folder, as should normally happen when adding a new Receive Encrypted folder. image image

    The workaround is to manually disconnect and reconnect the two devices, which will then force the folder to sync.

    Edit:

    I've misclicked the issue type. It should be labelled as a "bug" :flushed:.

    enhancement needs-triage 
    opened by tomasz1986 0
  • lib/osutil: Don't fsync directories in Windows (fixes #7918)

    lib/osutil: Don't fsync directories in Windows (fixes #7918)

    Windows supports fsync for files only, so don't even try to use it on directories. This eliminates fsync failed errors in model debug logs.

    Signed-off-by: Tomasz Wilczyński [email protected]

    opened by tomasz1986 4
  • ability to decrypt specific or all staggered files

    ability to decrypt specific or all staggered files

    Syncthing version: v1.18.1 Browser: Firefox 91.0.2

    Right now with staggered file versioning enabled on an untrusted encrypted node, decrypting just decrypts the lastest version, instead of all versions, potentially because the file names get mangled. An ability to decrypt specific files or all the staggered versions of a file would be helpful.

    bug needs-triage 
    opened by coldbootinitheart 0
  • lib/tlsutil: Allocate UnionedConnection in one go

    lib/tlsutil: Allocate UnionedConnection in one go

    Another micro-optimization: a UnionedConnection takes at most two allocations to set up, but it can be done in one.

    opened by greatroar 0
  • Custom relay - Excessive DNS lookups

    Custom relay - Excessive DNS lookups

    Does your log mention database corruption? No

    Include required information

    Please be sure to include at least:

    • which version of Syncthing and what operating system you are using - 1.18.1, Linux amd64, running in LXC containers (Proxmox)
    • browser and version, if applicable n/a
    • what happened, From Pi-hole's DNS query logs, I can see heaps of lookups to myrelay.example.com
    • what you expected to happen instead, and Reasonable caching of name lookups within Syncthing

    any steps to reproduce the problem.

    1. Fresh minimal LXC container running Debian, install syncthing from official apt repo
    2. Make sure /etc/resolv.conf in the container is pointing to a Pi-hole instance (Or any DNS server that lets you see the query logs)
    3. Set up a relay and assign a domain name e.g. myrelay.example.com, reasonable TTL e.g. 1 hour
    4. In Sync Protocal Listen Addresses, enter the relay - relay://myrelay.example.com:12345/?id=...
    5. Look at the Pi-hole query log - Many lookups of myrelay.example.com
    bug needs-triage 
    opened by GitHubGeek 11
Releases(v1.18.2)
Pluggable, extensible virtual file system for Go

vfs Package vfs provides a pluggable, extensible, and opinionated set of file system functionality for Go across a number of file system types such as

C2FO 122 Sep 7, 2021
A Go filesystem package for working with files and directories

Stowage A Go filesystem package for working with files and directories, it features a simple API with support for the common files and directories ope

null 19 May 28, 2021
Dragonfly is an intelligent P2P based image and file distribution system.

Dragonfly Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in o

dragonflyoss 5.7k Sep 9, 2021
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Steve Francia 4k Sep 14, 2021
Easily create & extract archives, and compress & decompress files of various formats

archiver Introducing Archiver 3.1 - a cross-platform, multi-format archive utility and Go library. A powerful and flexible library meets an elegant CL

Matt Holt 3.3k Sep 12, 2021
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).

中文 English 愿景:为用户提供最简单、可靠、高效的分布式文件系统。 go-fastdfs是一个基于http协议的分布式文件系统,它基于大道至简的设计理念,一切从简设计,使得它的运维及扩展变得更加简单,它具有高性能、高可靠、无中心、免维护等优点。 大家担心的是这么简单的文件系统,靠不靠谱,可不

小张 2.8k Sep 13, 2021
Run a command when files change

Reflex Reflex is a small tool to watch a directory and rerun a command when certain files change. It's great for automatically running compile/lint/te

Caleb Spare 2.4k Sep 11, 2021
CRFS: Container Registry Filesystem

CRFS: Container Registry Filesystem Discussion: https://github.com/golang/go/issues/30829 Overview CRFS is a read-only FUSE filesystem that lets you m

Google 1.2k Sep 4, 2021
Embed files into a Go executable

statik statik allows you to embed a directory of static files into your Go binary to be later served from an http.FileSystem. Is this a crazy idea? No

Jaana Dogan 3.3k Sep 14, 2021
Fast, dependency-free, small Go package to infer the binary file type based on the magic numbers signature

filetype Small and dependency free Go package to infer file and MIME type checking the magic numbers signature. For SVG file type checking, see go-is-

Tom 1.4k Sep 14, 2021
Optimized compression packages

compress This package provides various compression algorithms. zstandard compression and decompression in pure Go. S2 is a high performance replacemen

Klaus Post 2.4k Sep 15, 2021
Encrypted overlay filesystem written in Go

An encrypted overlay filesystem written in Go. Official website: https://nuetzlich.net/gocryptfs (markdown source). gocryptfs is built on top the exce

null 1.9k Sep 15, 2021
Abstract File Storage

afs - abstract file storage Please refer to CHANGELOG.md if you encounter breaking changes. Motivation Introduction Usage Matchers Content modifiers S

Viant, Inc 155 Sep 10, 2021
Plik is a scalable & friendly temporary file upload system ( wetransfer like ) in golang.

Want to chat with us ? Telegram channel : https://t.me/plik_root_gg Plik Plik is a scalable & friendly temporary file upload system ( wetransfer like

root.gg 727 Sep 5, 2021
The best HTTP Static File Server, write with golang+vue

gohttpserver Goal: Make the best HTTP File Server. Features: Human-friendly UI, file uploading support, direct QR-code generation for Apple & Android

Sound Sun 1.3k Sep 9, 2021
gsheet is a CLI tool (and Golang package) for piping csv data to and from Google Sheets

gsheet Table of Contents Introduction Why? Installation Authentication and Authorization What about OAuth authentication? CLI Usage Sheet commands Dri

chris 13 Aug 5, 2021
Go filesystem implementations for various URL schemes

hairyhenderson/go-fsimpl This module contains a collection of Go filesystem implementations that can discovered dynamically by URL scheme. All filesys

Dave Henderson 132 Aug 19, 2021
File system for GitHub

HUBFS · File System for GitHub HUBFS is a read-only file system for GitHub and Git. Git repositories and their contents are represented as regular dir

Bill Zissimopoulos 141 Sep 5, 2021
goelftools is library written in Go for parsing ELF file.

goelftools goelftools is library written in Go for parsing ELF file. This library is inspired by pyelftools and rbelftools. Motivation The motivation

null 19 Aug 21, 2021