hoarder-dl

hoarder-dl is a download orchestration tool written in Go, hosted at https://vc.maxkaya.com/maxpeterkaya/hoarder-dl. It hoards data in bulk by handing each URL you feed it to the external download tool that fits it best, so the real downloading is done by mature programs such as yt-dlp and spot-dl rather than by hoarder-dl itself. You can use it as a one-shot command for a single URL or a whole file of URLs, or as a daemon that schedules jobs, reloads its config while running, and accepts new work over a unix socket.

Why hoarder-dl

Keeping a personal archive current usually means tracking many sources by hand, picking a download tool for each one, and queuing every download yourself. hoarder-dl combines the download sources with the proper tools and points them at the destinations you choose, so the data stays up to date without manually queuing anything or writing crontab schedules by hand. The scheduling, the tool selection, and the concurrency control all live in one config file that the daemon keeps in sync as you edit it.

Features

• A domain to tool map routes every URL to the right downloader, so a Spotify link goes to spot-dl, a YouTube link goes to yt-dlp, and any unknown domain falls back to the default tool you configure.
• A single global worker pool is shared across every job and every submission, which keeps concurrent downloads bounded so bandwidth and disk I/O are never overwhelmed by too many processes at once.
• Jobs run on cron schedules or repeat intervals, giving each input file or single URL a steady rhythm of refreshes without any outside scheduler.
• The config file is watched at runtime and reloaded as it changes, with new settings applied to new tasks while downloads already in flight finish under the settings they started with.
• Modules extend a download tool's invocation for deeper integration, so a Spotify playlist can sync incrementally through spotdl instead of being re-fetched on every run.
• A running daemon exposes a unix socket that the submit command talks to, which lets you queue additional URLs from a terminal at any moment without restarting anything.
• An annotated example config and a systemd unit ship with the project, so the daemon can start automatically with minimal setup.

Installation

From the Arch User Repository

On Arch Linux the package is published to the Arch User Repository. Install it with your AUR helper of choice such as yay -S hoarder-dl, which pulls the package page at https://aur.archlinux.org/packages/hoarder-dl and builds it from the tagged source rather than from a prebuilt binary. See the Arch wiki at https://wiki.archlinux.org/title/Arch_User_Repository for a complete introduction to installing packages from the AUR.

From the Forgejo package registry

Beyond the Arch User Repository, the project publishes its Arch packages to the package registry of the Forgejo instance at vc.maxkaya.com, so Arch Linux users can install and update hoarder-dl through the system package manager rather than through an AUR helper. Each tagged release uploads the packages for the x86_64, aarch64, and armv7h architectures to the repository at https://vc.maxkaya.com/api/packages/maxpeterkaya/arch/extras.

Point pacman at the repository by appending a repository section to /etc/pacman.conf, consulting the Arch wiki at https://wiki.archlinux.org/title/Pacman when you need more detail on repository configuration.

[maxpeterkaya.extras.vc.maxkaya.com]
SigLevel = Required
Server = https://vc.maxkaya.com/api/packages/maxpeterkaya/arch/$arch

Forgejo signs the repository database with its own key, so the key must be imported and trusted by pacman-key once before the repository is used.

curl -O https://vc.maxkaya.com/api/packages/maxpeterkaya/arch/repository.key
pacman-key --add repository.key
pacman-key --lsign-key maxpeterkaya@noreply.vc.maxkaya.com

Refresh the package database and install hoarder-dl as you would any other package.

pacman -Sy hoarder-dl

From source

Clone the repository and build the binary with the Go toolchain. The version information is injected through ldflags just as the release pipeline does, so a local build carries the same metadata and reports its version the same way.

git clone https://vc.maxkaya.com/maxpeterkaya/hoarder-dl.git
cd hoarder-dl
go build -ldflags "-s -w -X hoarder-dl/internal/build.Version=dev -X hoarder-dl/internal/build.Commit=$(git rev-parse HEAD)" -o hoarder-dl ./cmd/tool

Release binaries

Every tagged release publishes binaries for Linux, macOS, and the BSDs to the releases page of the repository at https://vc.maxkaya.com/maxpeterkaya/hoarder-dl/releases. Download the archive that matches your platform, extract it, and place the hoarder-dl binary somewhere on your PATH.

Configuration

The configuration lives at /etc/hoarder-dl/config.yaml when that file exists, falling back to ~/.config/hoarder-dl/config.yaml, or under $XDG_CONFIG_HOME/hoarder-dl/config.yaml when that environment variable is set. The --config flag on every command pins a specific file instead. The first time you run any command without a system config present, a complete annotated example config is written to the per-user location, so the file itself documents every field as you fill it in.

The top level keys set the worker pool, the proxy, the logging behavior, and the socket path. worker.num controls the size of the global worker pool that all jobs and submissions share. proxy.url supplies a proxy that is passed to the download tool when that tool supports one. logging.output and logging.error stream the download tool's standard output and standard error, and socket.path is the unix socket the daemon listens on.

tools.default names the download tool used when nothing more specific matches a URL. tools.map attributes domains to tools, and because matching walks up parent domains, a mapping for youtube.com also covers www.youtube.com and music.youtube.com. Each mapping may carry args passed to the tool and a max_concurrent cap on how many processes of that tool may run at once across all jobs, where a value of zero means unlimited. A mapping may also set its own proxy, which overrides the global proxy for matching URLs; the value off disables proxying for that tool entirely.

jobs is a list of scheduled work items. Each job schedules itself with either a cron occurrence or a repeat interval, but never both. A job pulls its URLs from a file with one URL per line or from a single URL. The delay spaces out downloads within a job, output.folder is the directory the download tool runs in, and download.tool with download.args override the domain resolution for that job when they are set.

version: dev-xxx
worker:
    num: 3
proxy:
    url: ""
logging:
    output: false
    error: false
socket:
    path: /home/user/.config/hoarder-dl/hoarder-dl.sock
tools:
    default: yt-dlp
    map:
        - domain: spotify.com
          tool: spot-dl
        - domain: open.spotify.com
          tool: spot-dl
        - domain: youtube.com
          tool: yt-dlp
          args: "--format bestaudio"
          max_concurrent: 2
        - domain: music.youtube.com
          tool: yt-dlp
jobs:
    - occurrence: "0 */6 * * *"
      input:
        file: /home/user/hoard/urls.txt
        delay: 5
      output:
        folder: /home/user/hoard/downloads
      download:
        tool: ""
        args: ""

Usage

run

The run command downloads a single URL or a whole file of URLs in one shot. Pass a URL as the argument or point --input at a file with one URL per line. The --tool flag overrides the download tool, and when it is omitted the domain map in the config picks the tool automatically. --workers sets the size of the local worker pool for the batch, --delay spaces out the URLs, --output sets the working directory, --args appends arguments to the tool, --proxy passes a proxy to the tool (or off to disable the global one), and --verbose streams the tool's output to the terminal.

hoarder-dl run "https://www.youtube.com/watch?v=abc"
hoarder-dl run --input urls.txt --workers 4 --output /data/downloads

daemon

The daemon command starts the long-running process that schedules jobs, watches the config file, and listens on the unix socket. It keeps running until it receives SIGINT or SIGTERM, at which point it drains any in-flight downloads and removes the socket file before exiting. It is normally managed by the systemd user unit described below, so it starts automatically rather than being launched by hand.

hoarder-dl daemon

submit

The submit command sends URLs to a running daemon over the unix socket. Give it a single URL as the argument or point --file at a file with one URL per line, and use --tool to force a specific download tool for the batch. The --socket flag overrides the socket path when the daemon is not using the default one.

hoarder-dl submit "https://open.spotify.com/track/abc"
hoarder-dl submit --file urls.txt

Daemon behavior

All jobs and all submitted URLs feed into one shared task queue, and the global worker pool draws from that queue. This design keeps the number of running goroutines and concurrent download processes bounded no matter how many jobs are configured, which protects bandwidth and disk I/O from being saturated by a spike of activity. A per tool semaphore enforces the max_concurrent cap, so the same downloader never runs more processes at once than the config allows.

The config watcher relies on fsnotify and reacts to writes on the config file after a short debounce that merges rapid edits into one reload. When the file changes, the daemon re-reads it and applies the new settings to new tasks only, so downloads that are already in flight finish with the settings they started under. The worker pool resizes to match worker.num, the scheduler re-registers every job from the new occurrence and interval values, and the per tool caps update without any restart.

Modules

Modules integrate a download tool more deeply with hoarder-dl by rewriting its command line before it runs. A module is keyed by the tool it applies to, so activating one is as simple as routing URLs to that tool in tools.map; the module then applies to every invocation from the CLI and the daemon alike.

spotdl

The spotdl module routes Spotify playlist, album, and artist links through spotdl's sync flow instead of a plain download. Each URL contributes its Spotify ID, and a per-ID save file named <id>.spotdl lives in the output folder. The first run creates the file with spotdl sync <url> --save-file <id>.spotdl, and every later run reuses it with spotdl sync <id>.spotdl, so spotdl only queries the Spotify and YouTube APIs once per list and afterwards downloads only newly added songs. Removed songs are kept on disk (--sync-without-deleting). Single-track URLs and non-Spotify URLs pass through as a regular download.

systemd

The daemon ships with a system level systemd unit named hoarder, installed to /usr/lib/systemd/system by the Arch package. It runs the daemon as root with the config at /etc/hoarder-dl/config.yaml. Enable and start it through the system systemd manager, and it launches the daemon at boot and keeps it running under the configured restart policy.

sudo systemctl daemon-reload
sudo systemctl enable --now hoarder

The unit sends its output to the system journal, so you can follow the daemon logs with journalctl -u hoarder -f. The unix socket is recreated in /run/hoarder-dl on every boot, so any user on the host can submit URLs with hoarder-dl submit without extra flags.

Building and releasing

Releases are produced with GoReleaser, which injects the version, commit, and build date through ldflags into the hoarder-dl/internal/build package. The same pipeline publishes the platform binaries, generates the checksums, and maintains the AUR source package, so a tagged release keeps the Arch package current without any extra work.

License

hoarder-dl is licensed under the GNU Affero General Public License version 3. See the LICENSE file in the repository for the full text.