Automatic Crash Reporting for Developers¶
Collecting and Sending¶
Syncthing runs with one monitor process and one main process. The main process is the thing that is really “Syncthing”. The monitor process is responsible for reading the console output from the main process, restarting it if it exits, and reporting any crashes of the main process – when it’s allowed to do so.
When the monitor process detects a crash it creates a file
panic-$timestamp.log
in the config directory and attempts to upload it
to the crash reporting server – if crash reporting is enabled. When a log
has been successfully reported it is renamed with the double file ending
.reported.log
. Old crash logs are automatically removed after a while,
reported or not.
Report Format¶
A crash report is fundamentally a blob of plain UTF-8 text. It has a loose format, documented below. A report implicitly has a “report ID” which is the SHA-256 hash of the entire report text, in hex format.
The report consists of the following:
One line containing the Syncthing version, exactly as reported by
syncthing --version
. A leading timestamp and log level may be present but is ignored.Zero or more lines of plaintext data that is for human consumption only. The reports that Syncthing itself sends will have zero lines here, but one could include a report of what happened, log extracts, etc. here barring any privacy issues on the sender’s behalf.
A line beginning with the words
Panic at
followed by a timestamp in RFC3339 format.The panic backtrace as printed / formatted by the Go runtime.
Here is an example of a well formed but short report:
07:48:24 INFO: syncthing v1.1.4 "Erbium Earthworm" (go1.12.5 darwin-amd64) jb@kvin.kastelo.net 2019-05-21 20:36:38 UTC
Panic at 2019-05-22T07:48:25+02:00
panic: interface conversion: *pfilter.FilteredConn is not net.Conn: missing method Read
goroutine 106 [running]:
github.com/syncthing/syncthing/lib/connections.(*quicListener).Serve(0xc000158000)
/Users/jb/dev/github.com/syncthing/syncthing/lib/connections/quic_listen.go:74 +0x41b
github.com/thejerf/suture.(*Supervisor).runService.func1(0xc0001c6690, 0xc000000000, 0x54b4728, 0xc000158000)
/Users/jb/go/pkg/mod/github.com/thejerf/suture@v3.0.2+incompatible/supervisor.go:600 +0x47
created by github.com/thejerf/suture.(*Supervisor).runService
/Users/jb/go/pkg/mod/github.com/thejerf/suture@v3.0.2+incompatible/supervisor.go:588 +0x5b
Wire Protocol¶
To upload a crash report we need three things:
The data comprising the report as above,
the SHA-256 hash of the report data, making up the report ID, and
the base URL to send the report to.
The report URL is constructed by adding the report ID to the base URL. The
default base URL of https://crash.syncthing.net/newcrash/
and the report
ID abcd1234
results in the URL
https://crash.syncthing.net/newcrash/abcd1234
.
First a HEAD
request is performed on the report URL. If this request
returns successfully (200 OK
) it means the server already has the report
ID in question. We do not need to upload it.
If the HEAD request returns 404 Not Found
or another error we can
attempt to upload the report. This is done by a PUT
request to the same
URL with the report data as the body.