3.5. Automatic Crash Reporting for Developers¶
3.5.1. Collecting and Sending¶
Syncthing will normally run with one monitor process and one main process. The main process is the thing that is really “Syncthing”. The monitor process is responsible for reading the console output from the main process, restarting it if it exits, and reporting any crashes of the main process – when it’s allowed to do so.
No monitor process is used when Syncthing is started with the
--no-restart flag or the
STNORESTART environment variable. In these
cases there is also no crash reporting.
When the monitor process is running and detects a crash it creates a file
panic-$timestamp.log in the config directory and attempts to upload it
to the crash reporting server – if crash reporting is enabled. When a log
has been successfully reported it is renamed with the double file ending
.reported.log. Old crash logs are automatically removed after a while,
reported or not.
3.5.2. Report Format¶
A crash report is fundamentally a blob of plain UTF-8 text. It has a loose format, documented below. A report implicitly has a “report ID” which is the SHA-256 hash of the entire report text, in hex format.
The report consists of the following:
- One line containing the Syncthing version, exactly as reported by
syncthing --version. A leading timestamp and log level may be present but is ignored.
- Zero or more lines of plaintext data that is for human consumption only. The reports that Syncthing itself sends will have zero lines here, but one could include a report of what happened, log extracts, etc. here barring any privacy issues on the sender’s behalf.
- A line beginning with the words
Panic atfollowed by a timestamp in RFC3339 format.
- The panic backtrace as printed / formatted by the Go runtime.
Here is an example of a well formed but short report:
07:48:24 INFO: syncthing v1.1.4 "Erbium Earthworm" (go1.12.5 darwin-amd64) email@example.com 2019-05-21 20:36:38 UTC Panic at 2019-05-22T07:48:25+02:00 panic: interface conversion: *pfilter.FilteredConn is not net.Conn: missing method Read goroutine 106 [running]: github.com/syncthing/syncthing/lib/connections.(*quicListener).Serve(0xc000158000) /Users/jb/dev/github.com/syncthing/syncthing/lib/connections/quic_listen.go:74 +0x41b github.com/thejerf/suture.(*Supervisor).runService.func1(0xc0001c6690, 0xc000000000, 0x54b4728, 0xc000158000) /Usersfirstname.lastname@example.org+incompatible/supervisor.go:600 +0x47 created by github.com/thejerf/suture.(*Supervisor).runService /Usersemail@example.com+incompatible/supervisor.go:588 +0x5b
3.5.3. Wire Protocol¶
To upload a crash report we need three things:
- The data comprising the report as above,
- the SHA-256 hash of the report data, making up the report ID, and
- the base URL to send the report to.
The report URL is constructed by adding the report ID to the base URL. The
default base URL of
https://crash.syncthing.net/newcrash/ and the report
abcd1234 results in the URL
HEAD request is performed on the report URL. If this request
returns successfully (
200 OK) it means the server already has the report
ID in question. We do not need to upload it.
If the HEAD request returns
404 Not Found or another error we can
attempt to upload the report. This is done by a
PUT request to the same
URL with the report data as the body.