Open Source Continuous File Synchronization

Overview

Syncthing


Latest Linux & Cross Build Latest Windows Build Latest Mac Build MPLv2 License CII Best Practices Go Report Card

Goals

Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers. We strive to fulfill the goals below. The goals are listed in order of importance, the most important one being the first. This is the summary version of the goal list - for more commentary, see the full Goals document.

Syncthing should be:

  1. Safe From Data Loss

    Protecting the user's data is paramount. We take every reasonable precaution to avoid corrupting the user's files.

  2. Secure Against Attackers

    Again, protecting the user's data is paramount. Regardless of our other goals we must never allow the user's data to be susceptible to eavesdropping or modification by unauthorized parties.

  3. Easy to Use

    Syncthing should be approachable, understandable and inclusive.

  4. Automatic

    User interaction should be required only when absolutely necessary.

  5. Universally Available

    Syncthing should run on every common computer. We are mindful that the latest technology is not always available to any given individual.

  6. For Individuals

    Syncthing is primarily about empowering the individual user with safe, secure and easy to use file synchronization.

  7. Everything Else

    There are many things we care about that don't make it on to the list. It is fine to optimize for these values, as long as they are not in conflict with the stated goals above.

Getting Started

Take a look at the getting started guide.

There are a few examples for keeping Syncthing running in the background on your system in the etc directory. There are also several GUI implementations for Windows, Mac and Linux.

Docker

To run Syncthing in Docker, see the Docker README.

Vote on features/bugs

We'd like to encourage you to vote on issues that matter to you. This helps the team understand what are the biggest pain points for our users, and could potentially influence what is being worked on next.

Getting in Touch

The first and best point of contact is the Forum. There is also an IRC channel, #syncthing on freenode (with a web client), for talking directly to developers and users. If you've found something that is clearly a bug, feel free to report it in the GitHub issue tracker.

Building

Building Syncthing from source is easy, and there's a guide that describes it for both Unix and Windows systems.

Signed Releases

As of v0.10.15 and onwards release binaries are GPG signed with the key D26E6ED000654A3E, available from https://syncthing.net/security.html and most key servers.

There is also a built in automatic upgrade mechanism (disabled in some distribution channels) which uses a compiled in ECDSA signature. macOS binaries are also properly code signed.

Documentation

Please see the Syncthing documentation site.

All code is licensed under the MPLv2 License.

Comments
  • Filesystem Notification - Continuation

    Filesystem Notification - Continuation

    This is the continuation of plouj's PR #2807 on a branch in the syncthing repo for ease of maintenance. Refer to that PR for general information what this is about and commits previous to https://github.com/syncthing/syncthing/commit/1d424d84609a40d2a11ac8e06cb87aac6b130629.

    frozen-due-to-age pr-merged 
    opened by imsodin 184
  • Support for file encryption (e.g. non-trusted servers)

    Support for file encryption (e.g. non-trusted servers)

    So I have had a look at BitTorrent sync, syncthing and alternatives and what I always wondered about was the possibility to not only sync between resources I own and trust, but also external resources/servers which I do NOT trust with my data, up to a certain extent.

    One way to do this is using ecryptfs or encfs, but this has many obvious downsides: it is not an interoperable solution (only works on Linux), the files are actually stored in encrypted form on the disk (even if the resource is trusted and this is not necessary, for instance because of the file system being encrypted already), etc.

    What I propose is somehow configuring nodes which are only sent the files in an encrypted format, with all file contents (and potentially file/directory names as well; or even permissions) being encrypted. This way, if I want to store my private files on a fast server in a datacenter to access them from anywhere, I could do this with syncthing without essentially giving up ownership of those files. I could also prevent that particular sync node from being allowed/able to make any changes to the files without me noticing.

    I realize that this requires a LOT of additional effort, but it would be a killer feature that seems to not be available in any other "private cloud" solution so far. What are your thoughts on this feature?

    EDIT: BitTorrent sync mentions a feature like this in their API docs: "Encryption secret API users can generate folder secrets with encrypted peer support. Encryption secrets are read-only. They make Sync data encrypted on the receiver’s side. Recipients can sync files, but they can’t see file content, and they can’t modify the files. Encryption secrets come in handy if you need to sync to an untrusted location." (from http://www.bittorrent.com/intl/de/sync/developers/api)

    enhancement 
    opened by Natanji 143
  • all: Store pending devices and folders in database (fixes #7178)

    all: Store pending devices and folders in database (fixes #7178)

    Purpose

    As discussed in #5758 and mentioned on the forum, storing the pending (offered from remote but not yet added locally) devices and folders in the XML configuration is not a nice and scalable design. Instead, the information should live in the database, properly structured, and made available over dedicated API endpoints.

    This is also ground work and practice to finally approach an acceptable implementation of the prototype in #5758, extending the concept to other devices we have heard about in ClusterConfig messages. That is deliberately saved for another PR though.

    Testing

    All code paths have undergone extensive manual testing within an existing setup of four instances (one compiled from this PR). Especially regarding the clean-up of pending entries upon config changes, both online through the POST /rest/system/config REST API and offline by modifying the XML while Syncthing is not running. Notifications on the GUI look as before, API endpoints have been verified with curl.

    ~~Unit tests would be rather complicated and were discussed as not strictly needed considering how low the importance of the handled information is.~~ Unit tests included, with a few basic test cases.

    Documentation

    The envisioned API for this stuff is described in https://github.com/syncthing/docs/pull/498.

    The commit messages contain much of the rationale for each change, so I'd be happy to keep them intact instead of squash-merging in a single commit. If desired, I can rebase to clean out obvious fixup commits and end up with a nice patch series of self-contained changes.

    Breaking Changes For The User

    Existing <pendingDevice> / <pendingFolder> entries in the XML config are not carried over to the database. The corresponding notifications will disappear until the next connection attempt with the offering device. It has been discussed that the low importance of this information does not warrant a separate logic for config --> database info migrations. One possibility to minimize disruption would be to just keep the XML entries intact for some releases and hold back on commit 8ae24612397fea6182cfb3d6dcce592c78c6df5d until then.

    frozen-due-to-age 
    opened by acolomb 109
  • Option to not sync mtime

    Option to not sync mtime

    Trying to sync my android device and my NAS(armv5) both using syncthing v0.10.0:

    On the ntfs mount of the NAS the file /my/directory/.syncthing.example.test is created then I get:

    puller: final: chtimes /my/directory/.syncthing.example.test: invalid argument

    File /my/directory/example.test is not created. Syncthing continues with the next file but the same error happens for all the files in the directory.

    enhancement frozen-due-to-age 
    opened by otrag 108
  • GUI display of global changes

    GUI display of global changes

    Purpose

    This PR adds a button to the device section of the GUI that the user can click on to see all filesystem changes that originated/occurred on that machine. That is to say, not changed copied from the network. This should help people discover more easily what computer made a certain change that was propagated in larger P2P swarms.

    Screenshots

    image

    frozen-due-to-age 
    opened by nrm21 107
  • Folder isn't making progress

    Folder isn't making progress

    All nodes are on v0.10.2 GUI says

    9:35:58: Folder "default" isn't making progress - check logs for possible root cause. Pausing puller for 1m0s.
    9:36:41: Folder "PIA" isn't making progress - check logs for possible root cause. Pausing puller for 1m0s.
    

    In the log I see the "hash mismatch" info but the file is not changed during pull. It was probably the whole night in that state but i don't have more logs, I should probably increase the size... logs from the pulling node: http://alex-graf.de/public/st/no-progress.tar.gz

    Restarting also does not fix it

    opened by alex2108 104
  • gui: add advance config port mapping to gui

    gui: add advance config port mapping to gui

    Purpose

    https://github.com/syncthing/syncthing/issues/4824

    Gives the user the opportunity to display a link to web gui remote machines. The user can choose the port and set if the link will display.

    Testing

    I just check to see if the link was shown depending on the setting.

    Screenshots

    Screen Shot 2020-09-29 at 11 13 53 PM Screen Shot 2020-09-29 at 11 34 37 PM

    Documentation

    https://github.com/syncthing/docs/pull/573

    Authorship

    Your name and email will be added automatically to the AUTHORS file based on the commit metadata.

    frozen-due-to-age 
    opened by rjpruitt16 91
  • lib/connections: Add KCP support (fixes #804)

    lib/connections: Add KCP support (fixes #804)

    This is mostly for benchmarks and testing.

    Seems to connect, and not crash immediately, which is good news.

    Known issues:

    1. No timeouts as part of the protocol, so we usually end up with a dead connection for 5-7.5 minutes until it times out. This could be addressed by always using the new connection that we get, as long as the priority is right
    2. Has some weirdness around closing connections. See https://github.com/xtaci/kcp-go/issues/18

    The diff is massive due to deps, so I suggest you pull the PR locally and diff inside the lib and cmd/syncthing dirs.

    frozen-due-to-age pr-merged 
    opened by AudriusButkevicius 91
  • Memory and CPU usage is prohibitively high

    Memory and CPU usage is prohibitively high

    I've been testing syncthing across 3 machines, a laptop with 8GB of RAM and two NAS-style servers with 1GB and 2GB of RAM. My repositories have the following sizes:

    • 19 items, 536 KiB
    • 83471 items, 16.2 GiB
    • 181482 items, 387 GiB

    To sync these three repositories syncthing 0.9.0 uses a bit over 700MB of RAM and while syncing continuously pegs the CPU at 150% on all nodes.

    While the CPU usage I could manage during the initial sync the memory usage is simply too high. A typical NAS server like the two I have holds >4TB of storage. At the current level of usage that would require ~8GB of memory just for syncthing.

    Without looking at the code I assume an index is being kept in memory for all the repository contents. 700MB is 2.6kb per item on disk, which seems way too high. The index should only really need to store filename, parent, permissions, size and timestamp. On average (depends on filename sizes) that should only be 50-60 bytes per item which would only be 13MB. Moving that index to disk would also make a lot more sense.

    I assume the CPU usage is hashing lots of files. There I assume using librsync might be a better option.

    enhancement frozen-due-to-age 
    opened by pedrocr 86
  • /rest/system/status returns empty response during startup

    /rest/system/status returns empty response during startup

    Syncthing version 0.12.20 (although it's been around since at least the beginning of February).

    During startup, the /rest/system/status endpoint can return an empty response (seems to happen more on some devices than others).

    The headers returned are:

      Access-Control-Allow-Origin: *
      Pragma: no-cache
      X-Syncthing-Id: IDIDIID-IDIDID-IDIDID-IDIDID-IDIDID-IDIDID-IDIDID-IDIDID
      X-Syncthing-Version: v0.12.20
      Cache-Control: no-store, no-cache, max-age=0
      Date: Sat, 19 Mar 2016 22:40:58 GMT
      Content-Length: 0
      Content-Type: application/json; charset=utf-8
      Expires: Sat, 19 Mar 2016 22:40:58 GMT
    

    Of particular interest is the Content-Length: 0. I had a quick look through the code but couldn't see why it might be doing this. I fetch the /rest/system/version resource milliseconds afterwards and that returns the correct data.

    Thanks!

    frozen-due-to-age 
    opened by canton7 82
  • Use protocol buffers instead of custom XDR

    Use protocol buffers instead of custom XDR

    Not done, not for merging, just for review and discussion.

    This changes our serialization code to use protocol buffers instead of our custom code. Todo/done:

    • [X] Protocol structs should be protobuf
    • [X] Database structs should be protobuf
    • [x] Local discovery structs should be protobuf by humans?
    • [x] Clean up bitfields vs booleans (as per Audrius stuff)
    • [x] Remove currently unused fields as these can be added painlessly in the future

    Changes I think we should not do:

    • The global discovery protocol. No need, might be nice to have human readable, no real reason to cause the disturbance.
    • The relay protocol, as far as it concerns speaking to a relay. No reason to break it currently and leaving it unchanged lets us reap the benefits of the deployed infrastructure.

    Some changes that were necessary but unexpected or non-obvious:

    • ~~The protobuf serializer uses a Size() method which conflicted with our Size fields on BlockInfo. I've renamed it to Length.~~
    • I've added a Size field instead of CachedSize to FileInfo and let this be serialized everywhere. It means we get it for free in the TruncatedFileInfo instead of having to extract it from the block list.
    • ~~The TruncatedFileInfo type is "officialized" and moved into the protocol package, but it's still mostly used by the database. I'm not sure of this, maybe it can be undone and moved back to db as we have some protobufs stuff there as well (but didn't when I first made this change)...~~
    • The version vector type Vector []Counter definition doesn't work with protobuf so I had to make it a struct which resulted in a lot of tedious changes everywhere, but it's conceptually the same.
    • I added FileName() and FileLength() methods to the interface that covers both FileInfo and FileInfoTruncated so we can get the name or size of the interface without having to type assert to the underlying type.
    • I threw out the conversion from 0.12 db:s. We'll need full rescan for this change, but on the other hand it ought to be the last time ever.
    frozen-due-to-age 
    opened by calmh 79
  • Add arm64 architecture for syncthing/discosrv container

    Add arm64 architecture for syncthing/discosrv container

    First of all, thank you for this awesome project that I discovered this week. It is very simple to use and very well documented.

    I would like to deploy my own private discosrv on a RaspberryPi with the official image syncthing/discosrv. Unfortunately only amd64 architecture is available.

    Is it possible to support arm64 architecture for discosrv images ? I think that arm64 is a popular platform for this kind of little self-hosted service.

    enhancement needs-triage 
    opened by LoHub 0
  • LAN detection via subnetmask

    LAN detection via subnetmask

    IPv6 LAN connections currently show up as WAN connections because they use public IPv6 addresses. We should check if the other address is in the same subnet to determine if it's a LAN connection.

    Apart from the wrong connection type in the UI, this might also affect bandwidth limitation logic.

    https://forum.syncthing.net/t/towards-better-connection-type-icons/19333/23

    enhancement needs-triage 
    opened by bt90 3
  • Add possibility to exclude files/directories from versioning

    Add possibility to exclude files/directories from versioning

    The versioning system in SyncThing is great, but sometimes one does not want to version everything. In my case it is a Thunderbird profile folder that (for backup purposes) is sync'd as part of a larger parent directory. For that directory and (most of) its subdirectory, versioning is in place. But for the TB Profile directory this results in a lot of space being consumed. Perhaps not the best reason for this feature, but it is a reason... That feature would be that you can specify (in some way) that some directories need not by versioned, although sync'd. I see roughly two ways of dealing with this: either a file like .stignore containing regex patterns, or the (probably) more simpler approach of allowing the user to create a directory inside the .stversions directory with a marker signalling that that everything in that directory needs not be versioned. Something like:

    syncdir--+
             |-.stversions--+
             |              |-dir_a--+
             |              |        |-file_1.221117...
             |              |        |-...
             |              |
             |              |-dir_b--+
             |              |        |-.stdisable_versions
             |              |   
             |              |-dir_c--+
             |              |        |-file_1.221021....
             ...            ...      ...
    

    I could see this happen in lib/versioner/util.go where archiveFile does a check if the directory to move the versioned file to exists and if not, creates it. That could be prepended by a check for the existence of .stdisable_versions in any of the parent directories. I have, unfortunately, not enough knowledge of the Go language to prepare a code block for this.

    enhancement needs-triage 
    opened by Frank071 1
  • docs: Separate synced file folder and config files folder mount point

    docs: Separate synced file folder and config files folder mount point

    Purpose

    The original document didn't clarify that the /var/syncthing folder contains both the configuration files folder and the synced files folder, which might cause users trying to pull synced files directly from /var/syncthing rather than /var/syncthing/Sync.

    opened by ResRipper 1
  • GUI editor for xattr filter patterns

    GUI editor for xattr filter patterns

    The per-folder xattr filter is a complex object which is not by itself editable in the advanced config editor, and there's nothing custom built for it in the normal settings editor, so it's currently unavailable except for hacking the config file manually.

    There should be an editor for it in the normal settings dialog, under the xattr enablement checkboxes. There is should be possible to add a list of patterns, each with an allow/deny checkbox.

    The default is to allow all attributes if there are no patterns configured; this should be indicated in the GUI. When there are any patterns at all, the default (for things not matched by any pattern) becomes deny; this should also be indicated, especially if the user just adds one deny-pattern -- we should then suggest adding a permit * pattern at the end, maybe even add it by default and allow it to be removed or edited, or something like that.

    enhancement 
    opened by calmh 1
  • Syncthing rescans whole folders when individual files change.

    Syncthing rescans whole folders when individual files change.

    Version: v1.20.3 on Archlinux 2022-06-26

    What happened

    I use syncthing to backup large directories on my laptop to a server. When watch mode is turned on, editing a file leads to syncthing rescanning the entire directory, which leads to considerable CPU usage and fan spin-up. This is especially problematic in situations where the noise might be disruptive. Of these directories is my projects folder. Editing a project leads to my laptop taking flight.

    What I expected to happen

    Only the changed files being checked, instead of a full rescan of large directories.

    bug needs-triage 
    opened by winrg 1
Releases(v1.22.2-rc.3)
Owner
The Syncthing Project
The Syncthing Project
Magma is an open-source software platform that gives network operators an open, flexible and extendable mobile core network solution.

Connecting the Next Billion People Magma is an open-source software platform that gives network operators an open, flexible and extendable mobile core

Magma 1.4k Nov 26, 2022
Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator

Trickster is an HTTP reverse proxy/cache for http applications and a dashboard query accelerator for time series databases. Learn more below, and chec

null 1.8k Nov 21, 2022
Open source 5G core network base on 3GPP R15

What is free5GC The free5GC is an open-source project for 5th generation (5G) mobile core networks. The ultimate goal of this project is to implement

free5GC 1.5k Nov 21, 2022
Squzy - is a high-performance open-source monitoring, incident and alert system written in Golang with Bazel and love.

Squzy - opensource monitoring, incident and alerting system About Squzy - is a high-performance open-source monitoring and alerting system written in

Squzy 469 Nov 13, 2022
Project Kebe is the open-source Snap Store implementation.

Introduction Kebe intends to be a full replacement for the Snap Store. Quickstart Once you have an environment setup (for instance using https://githu

Free To Compute 25 Nov 9, 2022
CDN for Open Source, Non-commercial CDN management

CDN Control Official Website: https://cluckcdn.buzz Documentation (Traditional Chinese): https://cluckcdn.buzz/docs/ 简体中文 README: README_CN.md Please

CluckCDN 2 Dec 20, 2021
Headscale - An open source, self-hosted implementation of the Tailscale control server

Headscale - An open source, self-hosted implementation of the Tailscale control server

Juan Font 9.3k Nov 27, 2022
Apache Traffic Control is an Open Source implementation of a Content Delivery Network

Apache Traffic Control Apache Traffic Control is an Open Source implementation of a Content Delivery Network. Documentation Intro CDN Basics Traffic C

The Apache Software Foundation 833 Nov 18, 2022
mysshw - a free and open source ssh cli client soft.

mysshw install go version <= 1.16.* use go get go get -u github.com/cnphpbb/mysshw go version >= 1.17.* use go install go install github.com/cnphpbb/

cnphpbb 2 Dec 16, 2021
gRelay is an open source project written in Go that provides the circuit break pattern with a relay idea behind.

gRELAY gRelay is an open source project written in Go that provides: Circuit Break ✔️ Circuit Break + Relay ✔️ Concurrecny Safe ✔️ Getting start Insta

null 31 Sep 30, 2022
An open source Pusher server implementation compatible with Pusher client libraries written in Go

Try browsing the code on Sourcegraph! IPÊ An open source Pusher server implementation compatible with Pusher client libraries written in Go. Why I wro

Hava 1 Aug 27, 2022
Hybridnet is an open source container networking solution, integrated with Kubernetes and used officially by following well-known PaaS platforms

Hybridnet What is Hybridnet? Hybridnet is an open source container networking solution, integrated with Kubernetes and used officially by following we

Alibaba 154 Nov 11, 2022
Openp2p - an open source, free, and lightweight P2P sharing network

It is an open source, free, and lightweight P2P sharing network. As long as any device joins in, you can access them anywhere

null 184 Nov 27, 2022
Scout is a standalone open source software solution for DIY video security.

scout Scout is a standalone open source software solution for DIY video security. https://www.jonoton-innovation.com Features No monthly fees! Easy In

null 8 Oct 25, 2022
NebulaChat - Open source mtproto server written in golang

NebulaChat - Open source mtproto server written in golang open source mtproto server implemented in golang with compatible telegram client. Introduce

Teamgram 1.1k Nov 22, 2022
Open-source platform to request any SSP like Bidswitch or Xandr.

The project goal is to provide an unique program to contact every SSP without know the differences between all of them.

null 1 Apr 7, 2022
Kasen - Yet-another open-source CMS for scanlators

Kasen Oh no, a yet-another open-source CMS for scanlators. Anyways... The back-e

rs 11 Nov 22, 2022