Use Syncthing with WebDAV to Sync Everywhere

2022-11-04 #org-mode #hosting

I have been using Syncthing with Caddy’s WebDAV to synchronize my Org Mode files across computers and mobiles for few weeks. I am satisfied with it and writing this post about the setup.

Why not just Syncthing?

My previous setup use solely Syncthing to synchronize them across computers and mobiles. While Syncthing works great on computers, it is problematic on mobiles because mobile can switch networks, have a limited battery, and have limited background sync capability (especially on iOS). Out-dated files or conflicts happen more often to me when I use a mobile device.

I noticed some Org Mode mobile apps (like “beorg” and “Orgzly”) support WebDAV sync. Hence, I tried to add a WebDAV entry point for my org files and see how it goes.

Overview

I run Syncthing and WebDAV both watching the same org directory on a tiny server, Raspberry Pi 3B. Computers and mobiles are connected with Syncthing and WebDAV respectively.

WebDAV Setup

I first tried Nextcloud but chose Caddy eventually to set up WebDAV.

Attempt #1: Nextcloud

My first attempt was using Nextcloud’s WebDAV, after searching “Syncthing WebDAV” and found this story.

I added the Syncthing’s local folder to Nextcloud via external storage so it could be accessed from WebDAV. It worked. It proved that Syncthing with WebDAV is a viable solution.

However, Nextcloud is too heavy for my Raspberry Pi 3. The latency is high enough to degrade the mobile experience. This could be a good solution for existing Nextcloud users with a faster hosting machine.

Attempt #2: Caddy WebDAV

Since my Raspberry Pi is already using Caddy as the web server, why not give Caddy’s WebDAV a shot? Caddy is written in Go and should have better performance.

The WebDAV module was not built-in and I need to built a new Caddy binary with xcaddy. Then, I updated the systemd service to run as the same user as syncthing to get the file permission right.

It worked and indeed ran so much faster. I am satisfied!

Here is the snippet from my Caddyfile WebDAV config.

my.webdav.host {

    basicauth * {
        <user> <hash>
    }

    root * /path/to/webdav

    @get method GET
    route {
        file_server @get {
            hide .git .gitignore .stfolder .stversions
            browse
        }

        webdav
    }
}

Android and iOS Setup

I use beorg on iOS and Orgzly on Android. Both of them support “WebDAV” with “Auto-sync”. “Auto-sync” performs a sync when a note is changed or when the app is resumed. This greatly reduces the chance of conflict, when compared with Syncthing.

In addition, I found beorg also comes with a LISP runtime (BiwaScheme) and allows users to customize the app in a way similar to Emacs. The customization is limited at the moment but it has the potential!

Syncthing Setup

I use most of the default configuration with two tweaks.

Tweak #1: Reduce File Watch Delay

I reduce the fsWatcherDelayS from 10s to 1s in all instances. This makes the synchronization happens more instantly and seamlessly between devices. For example, when I make a change on the phone, it updates the Raspberry Pi and synchronizes across other computers in a second.

Tweak #2: Restart Syncthing on Wakeup

When waking up from sleep, Syncthing can take quite a while to recover the connection.

Syncthing used to provide an option called restartOnWakeup to restart Syncthing on wakeup and avoid this long delay. However, this option has been removed in 2022. I replicated this behavior myself, by writing a script to restart Syncthing, and running it on wake up.

A Script to Restart Syncthing

Creates a script to hit the REST endpoint (/rest/system/restart) on the local Syncthing instance.

LOCAL_ENDPOINT=http://localhost:8384
LOCAL_API_KEY=<API_KEY>

echo "Restarting Local Syncthing"

curl --silent -H "X-API-Key: ${LOCAL_API_KEY}" \
     --connect-timeout 1 \
     -X POST \
     ${LOCAL_ENDPOINT}/rest/system/restart

Run the Script on Wakeup (MacOS)

For MacOS, uses sleepwatcher to detect wakeup event and execute the above script.

First, install and start sleepwatcher service.

# To install sleepwatcher
brew install sleepwatcher

# To start sleepwatcher now and restart at login
brew services start sleepwatcher

Second, creates another script ~/.wakeup, which will be executed by sleepwatcher on system wakeup.

/path/to/syncthing-restart.sh

Run the Script on Wakeup (Linux)

For Linux, creates a script under /usr/lib/systemd/system-sleep/, which will be executed both “pre” and “post” system sleep.

#!/bin/sh

case $1 in
    post)
        /path/to/syncthing-restart.sh
        ;;
esac

Credit to this blog post.

Run the Script on Wakeup (ChromeOS)

Unfortunately, the Linux systemd method does not work for ChromeOS via Crostini. My solution is to use a script to check the system timestamp every second. If the current timestamp and the last timestamp differ more than 3 seconds, then I consider it as a system wakeup.

#!/bin/bash

last=$(date +%s)

while true; do
  current=$(date +%s)

  if [[ $(($current - $last)) -ge 3 ]]; then
    echo "Detected wake up. Restarting Syncthing."
    /path/to/syncthing-restart.sh
  fi

  last=${current}
  sleep 1
done

Then, I create a systemd service (~/.config/systemd/user/sleep-watcher.service) to run this script in the background.

[Unit]
Description=Sleep Watcher Service

[Service]
Type=simple
StandardOutput=journal
ExecStart=/path/to/the/above/sleep-watcher.sh

[Install]
WantedBy=default.target

Once it is ready, reload systemd, start and enable the service.

systemctl --user daemon-reload
systemctl --user enable --now sleep-watcher

Conclusion

Syncthing works brilliant on computers and laptops with the tweaks mentioned above. WebDAV is simple and cheap to host. It is secure to use over the Internet when pairing with BasicAuth and HTTPS (both supported by Caddy). It is not a perfect synchronization tool but good enough to fill the mobile gap.