3.4. Debugging Syncthing

There’s a lot that happens behind the covers, and Syncthing is generally quite silent about it. A number of environment variables can be used to set the logging to verbose for various parts of the program, and to enable profiling.

Enabling any of the STTRACE facilities will also change the log format to include microsecond timestamps and file names plus line numbers. This can be used to enable this extra information on the normal logging level, without enabling any debugging: STTRACE=somethingnonexistent for example.

Under Unix (including Mac) the easiest way to run Syncthing with an environment variable set is to prepend the variable to the command line. I.e:

$ STTRACE=model syncthing

On windows, it needs to be set prior to running Syncthing.

C:\> set STTRACE=model
C:\> syncthing

3.4.1. Environment Variables

STTRACE

Used to increase the debugging verbosity in specific or all facilities, generally mapping to a Go package. Enabling any of these also enables microsecond timestamps, file names plus line numbers. Enter a comma-separated string of facilities to trace. syncthing -help always outputs an up-to-date list. The valid facility strings are:

Main and operational facilities:
config

Configuration loading and saving.

db

The database layer.

main

Main package.

model

The root hub; the largest chunk of the system. File pulling, index transmission and requests for chunks.

scanner

File change detection and hashing.

versioner

File versioning.

Networking facilities:
beacon

Multicast and broadcast UDP discovery packets: Selected interfaces and addresses.

connections

Connection handling.

dialer

Dialing connections.

discover

Remote device discovery requests, replies and registration of devices.

nat

NAT discovery and port mapping.

pmp

NAT-PMP discovery and port mapping.

protocol

The BEP protocol.

relay

Relay interaction (strelaysrv).

upnp

UPnP discovery and port mapping.

Other facilities:
fs

Filesystem access.

events

Event generation and logging.

http

REST API.

sha256

SHA256 hashing package (this facility currently unused).

stats

Persistent device and folder statistics.

sync

Mutexes. Used for debugging race conditions and deadlocks.

upgrade

Binary upgrades.

walkfs

Filesystem access while walking.

all

All of the above.

STBLOCKPROFILE

Write block profiles to block-$pid-$timestamp.pprof every 20 seconds.

STCPUPROFILE

Write a CPU profile to cpu-$pid.pprof on exit.

STDEADLOCKTIMEOUT

Used for debugging internal deadlocks; sets debug sensitivity. Use only under direction of a developer.

STDEADLOCKTHRESHOLD

Used for debugging internal deadlocks; sets debug sensitivity. Use only under direction of a developer.

STGUIASSETS

Directory to load GUI assets from. Overrides compiled in assets. Useful for developing webgui, commonly use STGUIASSETS=gui bin/syncthing.

STHASHING

Specify which hashing package to use. Defaults to automatic based on performance. Specify “minio” (compatibility) or “standard” for the default Go implementation.

STHEAPPROFILE

Write heap profiles to heap-$pid-$timestamp.pprof each time heap usage increases.

STNODEFAULTFOLDER

Don’t create a default folder when starting for the first time. This variable will be ignored anytime after the first run.

STNORESTART

Equivalent to the -no-restart flag. Disable the Syncthing monitor process which handles restarts for some configuration changes, upgrades, crashes and also log file writing (stdout is still written).

STNOUPGRADE

Disable automatic upgrades.

STPROFILER

Set to a listen address such as “127.0.0.1:9090” to start the profiler with HTTP access, which then can be reached at http://localhost:9090/debug/pprof. See go tool pprof for more information.

STPERFSTATS

Write running performance statistics to perf-$pid.csv. Not supported on Windows.

STRECHECKDBEVERY

Time before folder statistics (file, dir, … counts) are recalculated from scratch. The given duration must be parseable by GO’s time.ParseDuration. If missing or not parseable, the default value of 1 month is used. To force recalculation on every startup, set it to 0.

GOMAXPROCS

Set the maximum number of CPU cores to use. Defaults to all available CPU cores.

GOGC

Percentage of heap growth at which to trigger GC. Default is 100. Lower numbers keep peak memory usage down, at the price of CPU usage (i.e. performance).

3.4.2. Stepping with breakpoints

If you like to step through the running program, build a non-optimized binary and run with https://github.com/derekparker/delve.

Follow these steps:

$ go run build.go -debug-binary build
$ STNODEFAULTFOLDER=1 STNOUPGRADE=1  STNORESTART=1 dlv --listen=:2345 --headless=true --api-version=2 exec ./syncthing -- -home=./_test_config -no-browser

For installing and using delve itself see: