Debugging Syncthing

A file isn’t syncing

If you wonder why a given file is out of sync, or you suspect it’s caused by a bug, we need to collect some information. You’ll need the name of the file and the folder ID it’s in.

To begin with, check the GUI on the receiving side. It may be listed under “Failed items”, with a cause for the failure. Secondly, check the logs on the receiving side. If the file has failed to sync this will be mentioned in the logs even if it is not currently visible in the GUI. (If you are debugging this together with someone on the forum, post screenshots of the GUI and the logs in question.)

If nothing relevant showed up so far, or the question is why the file is considered out of sync to begin with in a receive-only setup, etc., we can look closer at what Syncthing knows about the file. You’ll need to do these steps on both the receiving side and sending side. We will need to use the Syncthing CLI for this.

First, an extract of database metadata for the file, taked by the command:

syncthing debug database-file $folderID $fileName

The $folderID placeholder represents the folder ID (e.g. abcd-1234) and fileName is the name of the file, including any directories, relative to the folder root. Make sure to use quotes around the file name if it includes spaces.

Second, details of the file information for the file in question, taked by the command:

syncthing cli debug file $folderID $fileName

Use the same folder ID and file name as above.

Post the output from both of these, verbatim, indicating which is the receiving side and which is the sending side.

The folder counts seem incorrect

If the number of files in sync, out of sync, in total, etc. seem to be incorrect, we can inspect the raw counts in the database. Run the following command on both the sending and receiving sides and post the output:

syncthing debug database-counts $folderID

The $folderID placeholder represents the folder ID (e.g. abcd-1234).

Debug logs

There’s a lot that happens behind the covers, and Syncthing is generally quite silent about it. A number of environment variables can be used to set the logging to verbose for various parts of the program, and to enable profiling.

Under Unix (including Mac) the easiest way to run Syncthing with an environment variable set is to prepend the variable to the command line. I.e:

$ STTRACE=model syncthing

On windows, it needs to be set prior to running Syncthing.

C:\> set STTRACE=model
C:\> syncthing

Environment variables

STTRACE

Used to increase the debugging verbosity in specific facilities, generally mapping to a Go package. Enter a comma-separated string of facilities to trace: api,beacon. Optionally, a log level can be given per facility to specify something other than DEBUG: api:WARN,beacon:ERR, potentially overriding a global --log-level adjustment.

The valid facility strings are listed below; additionally, syncthing serve --help always outputs the most up-to-date list.

api

REST API

beacon

Multicast and broadcast discovery

config

Configuration loading and saving

connections

Connection handling

db/sqlite

SQLite database

dialer

Dialing connections

discover

Remote device discovery

events

Event generation and logging

fs

Filesystem access

main

Main package

model

The root hub

nat

NAT discovery and port mapping

pmp

NAT-PMP discovery and port mapping

protocol

The BEP protocol

relay/client

Relay client

scanner

File change detection and hashing

stun

STUN functionality

syncthing

Main run facility

upgrade

Binary upgrades

upnp

UPnP discovery and port mapping

ur

Usage reporting

versioner

File versioning

watchaggregator

Filesystem event watcher

STLOCKTHRESHOLD

Used for debugging internal deadlocks; sets debug sensitivity. Use only under direction of a developer.

STVERSIONEXTRA

Add extra information to the version string in logs and the version line in the GUI. Can be set to the name of a wrapper or tool controlling syncthing to communicate this to the end user.

GOMAXPROCS

Set the maximum number of CPU cores to use. Defaults to all available CPU cores.

GOGC

Percentage of heap growth at which to trigger GC. Default is 100. Lower numbers keep peak memory usage down, at the price of CPU usage (i.e. performance).

LOGGER_DISCARD

Hack to completely disable logging, for example when running benchmarks. Set to any nonempty value to use it.

Stepping with breakpoints

If you like to step through the running program, build a non-optimized binary and run with https://github.com/derekparker/delve.

Follow these steps:

$ go run build.go -debug-binary build
$ STNODEFAULTFOLDER=1 STNOUPGRADE=1  STNORESTART=1 dlv --listen=:2345 --headless=true --api-version=2 exec ./syncthing -- --home=./_test_config --no-browser

For installing and using delve itself see: