2.9. Understanding Synchronization¶
This article describes the mechanisms Syncthing uses to bring files in sync on a high level.
Files are divided into blocks. The blocks are currently fixed size, 128 KiB, except the last one in the file which may be smaller. Each file is sliced into a number of these blocks, and the SHA256 hash of each block is computed. This results in a block list containing the offset, size and hash of all blocks in the file.
To update a file, Syncthing compares the block list of the current version of the file to the block list of the desired version of the file. It then tries to find a source for each block that differs. This might be locally, if another file already has a block with the same hash, or it may be from another device in the cluster. In the first case the block is simply copied on disk, in the second case it is requested over the network from the other device.
When a block is copied or received from another device, its SHA256 hash is computed and compared with the expected value. If it matches the block is written to a temporary copy of the file, otherwise it is discarded and Syncthing tries to find another source for the block.
Syncthing detects changes to files by scanning. By default this happens
every 60 seconds, but this can be changed per folder. Increasing the scan
interval uses less resources and is useful for example on large folders that
change infrequently. To make sure that not all folders are rescanned at the same
time, the actual scan interval is a random time between 3/4 and 5/4 of the given
syncthing-inotify can also be used, which tells
Syncthing to scan changed files when changes are detected, thus reducing the
need for periodic scans.
During a rescan the existing files are checked for changes to their modification time, size or permission bits. The file is “rehashed” if a change is detected based on those attributes, that is a new block list is calculated for the file. It is not possible to know which parts of a file have changed without reading the file and computing new SHA256 hashes for each block.
Changes that were detected and hashed are transmitted to the other devices after each rescan.
Syncthing keeps track of several versions of each file - the version that it currently has on disk, called the local version, the versions announced by all other connected devices, and the “best” (usually the most recent) version of the file. This version is called the global version and is the one that each device strives to be up to date with.
This information is kept in the index database, which is stored in the
configuration directory and called
index-vx.y.z.db (for some version
x.y.z which may not be exactly the version of Syncthing you’re running).
When new index data is received from other devices Syncthing recalculates which version for each file should be the global version, and compares this to the current local version. When the two differ, Syncthing needs to synchronize the file. The block lists are compared to build a list of needed blocks, which are then requested from the network or copied locally, as described above.
2.9.4. Temporary Files¶
Syncthing never writes directly to a destination file. Instead all changes are made to a temporary copy which is then moved in place over the old version. If an error occurs during the copying or syncing, such as a necessary block not being available, the temporary file is kept around for up to a day. This is to avoid needlessly requesting data over the network.
The temporary files are named
original-filename.ext is the destination filename. The temporary file is
normally hidden. If the temporary file name would be too long due to the
addition of the prefix and extra extension, a hash of the original file name
is used instead of the actual original file name.