Debugging Syncthing¶
There’s a lot that happens behind the covers, and Syncthing is generally quite silent about it. A number of environment variables can be used to set the logging to verbose for various parts of the program, and to enable profiling.
Enabling any of the STTRACE facilities will also change the log format to
include microsecond timestamps and file names plus line numbers. This
can be used to enable this extra information on the normal logging
level, without enabling any debugging: STTRACE=somethingnonexistent
for example.
Under Unix (including Mac) the easiest way to run Syncthing with an environment variable set is to prepend the variable to the command line. I.e:
$ STTRACE=model syncthing
On windows, it needs to be set prior to running Syncthing.
C:\> set STTRACE=model
C:\> syncthing
Environment Variables¶
- STTRACE
Used to increase the debugging verbosity in specific or all facilities, generally mapping to a Go package. Enabling any of these also enables microsecond timestamps, file names plus line numbers. Enter a comma-separated string of facilities to trace.
syncthing --help
always outputs an up-to-date list. The valid facility strings are:- Main and operational facilities:
- config
Configuration loading and saving.
- db
The database layer.
- main
Main package.
- model
The root hub; the largest chunk of the system. File pulling, index transmission and requests for chunks.
- scanner
File change detection and hashing.
- versioner
File versioning.
- Networking facilities:
- beacon
Multicast and broadcast UDP discovery packets: Selected interfaces and addresses.
- connections
Connection handling.
- dialer
Dialing connections.
- discover
Remote device discovery requests, replies and registration of devices.
- nat
NAT discovery and port mapping.
- pmp
NAT-PMP discovery and port mapping.
- protocol
The BEP protocol.
- relay
Relay interaction (
strelaysrv
).- upnp
UPnP discovery and port mapping.
- Other facilities:
- fs
Filesystem access.
- events
Event generation and logging.
- http
REST API.
- sha256
SHA256 hashing package (this facility currently unused).
- stats
Persistent device and folder statistics.
- sync
Mutexes. Used for debugging race conditions and deadlocks.
- upgrade
Binary upgrades.
- walkfs
Filesystem access while walking.
- all
All of the above.
- STBLOCKPROFILE
Write block profiles to
block-$pid-$timestamp.pprof
every 20 seconds.- STCPUPROFILE
Write a CPU profile to
cpu-$pid.pprof
on exit.- STDEADLOCKTIMEOUT
Used for debugging internal deadlocks; sets debug sensitivity. Use only under direction of a developer.
- STLOCKTHRESHOLD
Used for debugging internal deadlocks; sets debug sensitivity. Use only under direction of a developer.
- STGUIADDRESS
Override GUI listen address. Equivalent to passing
--gui-address
.- STGUIAPIKEY
Override the API key needed to access the GUI / REST API. Equivalent to passing
--gui-apikey
.- STGUIASSETS
Directory to load GUI assets from. Overrides compiled in assets. Useful for developing webgui, commonly use
STGUIASSETS=gui bin/syncthing
.- STHEAPPROFILE
Write heap profiles to
heap-$pid-$timestamp.pprof
each time heap usage increases.- STNODEFAULTFOLDER
Don’t create a default folder when starting for the first time. This variable will be ignored anytime after the first run. Equivalent to the
--no-default-folder
flag.- STNORESTART
Equivalent to the
--no-restart
flag.- STNOUPGRADE
Disable automatic upgrades. Equivalent to the
--no-upgrade
flag.- STPROFILER
Set to a listen address such as “127.0.0.1:9090” to start the profiler with HTTP access, which then can be reached at http://localhost:9090/debug/pprof. See
go tool pprof
for more information.- STPERFSTATS
Write running performance statistics to
perf-$pid.csv
. Not supported on Windows.- STRECHECKDBEVERY
Time before folder statistics (file, dir, … counts) are recalculated from scratch. The given duration must be parseable by Go’s
time.ParseDuration
. If missing or not parseable, the default value of 1 month is used. To force recalculation on every startup, set it to1s
.- STGCINDIRECTEVERY
Sets the time interval in between database garbage collection runs. The given duration must be parseable by Go’s
time.ParseDuration
.- GOMAXPROCS
Set the maximum number of CPU cores to use. Defaults to all available CPU cores.
- GOGC
Percentage of heap growth at which to trigger GC. Default is 100. Lower numbers keep peak memory usage down, at the price of CPU usage (i.e. performance).
- LOGGER_DISCARD
Hack to completely disable logging, for example when running benchmarks. Set to any nonempty value to use it.
Stepping with breakpoints¶
If you like to step through the running program, build a non-optimized binary and run with https://github.com/derekparker/delve.
Follow these steps:
$ go run build.go -debug-binary build
$ STNODEFAULTFOLDER=1 STNOUPGRADE=1 STNORESTART=1 dlv --listen=:2345 --headless=true --api-version=2 exec ./syncthing -- --home=./_test_config --no-browser
For installing and using delve itself see:
VSCode (Microsoft): https://github.com/golang/vscode-go/blob/master/docs/debugging.md
GoLand (JetBrains): create remote run configuration and follow the two steps displayed