Periodic Profiling

Background

This article describes how to take a on-off CPU or memory profile in Syncthing. Doing so is occasionally useful, but the results will be tainted by the usage of the GUI to take the profile, and will only reflect the status at precisely the time the profile is taken. Generally it is more useful to collect profiles periodically over 24 to 48 hours, correlate them, and analyze the peak usages. Here we will describe a systematic way of doing so.

Prerequisites

  • We need to know the API key, and the API/GUI address if it is not localhost, of the Syncthing device to be profiled.

  • We need a Linux or similar hosts to run the profile collection script on. This host needs to have curl installed.

  • We need to be able to schedule the script using cron.

  • The Syncthing instance must have GUI debugging enabled.

Procedure

We will prepare a script that takes one CPU profile and one heap (memory) profile. We will run it manually once to ensure that it works. We will the use cron to schedule it to run hourly.

Enable GUI Debugging

Follow the instructions in the Syncthing documentation article to enable debugging.

Script

The following script should be saved on the Linux (or similar) host. The naming isn’t critical but for the purposes of this article we will assume /usr/local/bin/profile-syncthing.sh.

#!/bin/bash

apikey="ALob...EguN" # Get actual API key from settings
stamp=$(date +%s)

curl -skL -H "X-API-Key: $apikey" -o "cpu-$stamp.pprof" \
    http://localhost:8384/rest/debug/cpuprof

curl -skL -H "X-API-Key: $apikey" -o "heap-$stamp.pprof" \
    http://localhost:8384/rest/debug/heapprof

The API key on line three must be taken from the Syncthing device, either the GUI in the Settings dialog or the config file. Insert the full API key between the quotes. Create this script using your favourite editor and ensure that it has the execution bits set:

$ sudo chmod 755 /usr/local/bin/profile-syncthing.sh

Run the script once as a normal user. It should create two files called “cpu-(timestamp).pprof” and “heap-(timestamp).pprof” where the “(something)” is a numeric timestamp.

Cron

We will use cron to run the script once an hour. The precise process can differ slightly depending on your system and its setup. We recommend running the script as a normal user (not root). Typically a normal user can edit their crontab by running crontab -e. A line such as the following should be added:

15 *    * * *    /usr/local/bin/profile-syncthing.sh

This means that at fifteen minutes past the hour, run the script. The resulting profile files will saved to the user’s home directory. A different or more advanced schedule can be set as desired.