Ian JohannesenSystems Engineer, SRE/DevOps, IP EngineerZola2026-04-16T15:30:00+00:00https://perlpimp.net/atom.xmlVisual State Engines for Opaque UIs2026-04-16T15:30:00+00:002026-04-16T15:30:00+00:00
Ian Johannesen
https://perlpimp.net/blog/visual-state-engines-opaque-uis/<p>The awkward thing about automating some modern browser applications is that, after a certain point, the browser stops being the application.</p>
<p>It is still there. You can still drive Chromium. You can still click buttons, wait for selectors, fill forms, and listen to network requests. But once the real interface has loaded into a canvas, a remote desktop stream, a game engine, a video surface, or a heavily custom renderer, the useful application state may no longer exist in the DOM in any meaningful way.</p>
<p>There is no selector for “the correct tab is active”. There is no structured event for “the list advanced after that click”. There is no API response that tells you whether the modal you care about appeared. There is just a rectangle of pixels.</p>
<p>That changes the shape of the automation problem.</p>
<p>This is not conventional DOM automation. It is closer to writing a state machine whose sensors are screenshots.</p>
<h2 id="the-wrong-shape">The Wrong Shape</h2>
<p>The tempting version of this kind of automation is a long imperative script:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">open browser
log in
click the navigation item
click the target tab
loop:
screenshot the row
click the action button
wait
</code></pre>
<p>That version is attractive because it matches how a human would describe the job. It is also fragile in exactly the way pixel-driven automation tends to be fragile. If login triggers a challenge, the script is lost. If an unexpected popup appears over the canvas, the script keeps clicking underneath it. If the click lands but the UI does not advance, the script has no idea whether the application is slow, the coordinate is wrong, or the screen is no longer where it thinks it is.</p>
<p>The working shape is usually not one big script. It is layered state engines.</p>
<p>One layer prepares the browser and account session. It knows about browser launch, navigation, login, consent banners, challenges, blocking popups, and whether the rendered app is ready. Another layer owns the opaque UI itself: canvas clicks, visual anchors, active tabs, list rows, button targeting, popup detection, and proof that a state transition happened. A third layer turns the whole thing into a runtime: scheduling, retries, safe stop/start, timeline events, artifacts, metrics, and operational recovery.</p>
<p>That separation matters. Browser preparation, visual navigation, and autonomous operation fail in different ways. If they are collapsed into one linear script, every failure looks like “the bot broke”. If they are separate state engines, each layer gets its own vocabulary for success, blockage, evidence, and recovery.</p>
<p>In practice, this means every step should answer four questions:</p>
<ul>
<li>What action am I about to take?</li>
<li>What evidence proves it worked?</li>
<li>What evidence means I am blocked?</li>
<li>What artifacts should I keep if I am wrong?</li>
</ul>
<p>Without those answers, the automation is mostly hoping.</p>
<h2 id="pixels-as-state">Pixels as State</h2>
<p>Once the useful interface is opaque, screenshots become the API.</p>
<p>The system has to infer state from evidence like this:</p>
<ul>
<li>Is the target panel visible?</li>
<li>Which tab or mode is active?</li>
<li>Is the list empty?</li>
<li>Where are the visible row bands?</li>
<li>Where is the action button inside the current row?</li>
<li>Did the screen change after the click?</li>
<li>Did a confirmation popup appear?</li>
<li>Does the next captured row match what the previous click should have produced?</li>
</ul>
<p>None of those are DOM questions. They are image questions.</p>
<p>The robust version is dynamic scanning. You crop meaningful regions and detect row bands by projection activity. You search for colored connected components that look like buttons. You compare before and after crops with visual change scores. You hash regions and compare signatures. You detect active tabs, empty states, and confirmation popups from image evidence.</p>
<p>This is resilient because it keeps asking the screen what actually happened. If a row is a few pixels taller than expected, dynamic row detection can still find it. If a button shifts, component detection can target the button instead of trusting a stale coordinate. If a click is swallowed, the before/after probes can say “no meaningful state change happened” instead of blindly moving on.</p>
<p>But dynamic scanning has a cost.</p>
<p>Every broad screenshot has to be captured by the browser, encoded, transported back to the automation process, decoded, cropped, scanned, hashed, and compared. In the hot path, that cost compounds quickly. The expensive part is not only the image algorithm. It is the screenshot pipeline around the algorithm.</p>
<p>That is easy to underestimate. You look at a helper called something like <code>visual_change_score</code> and think about CPU. But the real loop may be doing repeated screenshot captures, waiting between probes, and decoding images just to decide whether one click worked.</p>
<p>For a one-off debugging run, that may be fine. For a runtime that needs to repeat the same operation hundreds or thousands of times, it is the difference between a plausible service and a sluggish science project.</p>
<h2 id="calibrated-pixels">Calibrated Pixels</h2>
<p>The opposite approach is calibrated pixels.</p>
<p>Instead of searching the whole UI for a known tab, use a pinned click point. Instead of inferring active state from a broad region, sample a tiny swatch inside each tab body. Instead of rediscovering list geometry every time, configure the first row crop and row pitch. Model the coordinates against a known viewport and scale them to the runtime screenshot size.</p>
<p>This feels wrong at first. Hardcoded pixels are the sort of thing people make fun of in automation code, often for good reason. They are brittle. They depend on viewport, scale, layout, language, theme, and whatever the product designers changed this week.</p>
<p>But calibrated pixels are also extremely fast.</p>
<p>A fixed click point does not need image search. A tiny tab swatch is cheaper than analyzing a broad tab body. A known first-row crop is cheaper than rediscovering every candidate row. If the hot path is “open the next row and prove that it advanced”, small probes beat wide scans.</p>
<p>The trick is to make the brittleness operational rather than mysterious.</p>
<p>Calibrated systems need:</p>
<ul>
<li>a controlled viewport model;</li>
<li>explicit coordinate and region configuration;</li>
<li>debug artifacts that show what the automation saw;</li>
<li>timing metadata for slow or ambiguous transitions;</li>
<li>a fast recalibration path when the UI shifts.</li>
</ul>
<p>That makes a stale coordinate a normal maintenance event instead of a haunted failure. The coordinate is still brittle, but the system tells you which visual assumption broke and gives you the evidence needed to fix it.</p>
<h2 id="the-useful-compromise">The Useful Compromise</h2>
<p>The design I trust most is the hybrid.</p>
<p>Use calibrated coordinates for the hot path, but do not blindly trust them. A click is only useful if a small amount of image evidence confirms the transition.</p>
<p>For tab switching, a primary and fallback click point can replace broad candidate search. After the click, tiny swatches can confirm which tab is active. If the swatch evidence is ambiguous, a broader region comparison can still act as fallback validation.</p>
<p>For row-oriented flows, the system can capture the row before taking action, choose a click point, dispatch the click, and then wait for proof. That proof might come from a confirmation popup, a list-region change, a row-identity comparison, or a hash match against the row that should have shifted into place.</p>
<p>There is an especially practical optimization here: avoid taking separate browser screenshots for every proof crop. Compute one enclosing checkpoint region, capture it once, and crop the smaller regions in memory. That keeps the evidence model but reduces screenshot round-trips.</p>
<p>This is the core engineering lesson: do the minimum visual work that can prove the transition.</p>
<p>Not the minimum work that can probably get away with it. Not the maximum work that makes the system feel clever. The minimum work that can prove the state transition you are about to depend on.</p>
<p>In a good hybrid system:</p>
<ul>
<li>fixed coordinates handle the common path;</li>
<li>tiny swatches confirm cheap state;</li>
<li>cropped probes prove transitions;</li>
<li>dynamic scanning remains available where drift is likely;</li>
<li>artifacts and timing metadata make recalibration routine.</li>
</ul>
<p>Fast automation without artifacts is a future debugging trap. Debuggable automation can afford to be more aggressive, because when it fails you can see what it thought it saw.</p>
<h2 id="screenshot-format-is-part-of-the-algorithm">Screenshot Format Is Part of the Algorithm</h2>
<p>One of the easiest performance lessons to miss is that screenshot format matters.</p>
<p>It is natural to start with PNG files. PNG is lossless, and if cropped images eventually feed OCR or visual matching, you do not want compression artifacts in the data you are trying to read.</p>
<p>But screenshots in the hot loop are not just data. They are transport. The browser has to encode them. The automation process has to receive them. The runtime has to decode them. When you do that repeatedly, the image format becomes part of the algorithm.</p>
<p>In one recent visual automation loop, moving browser screenshots from PNG to JPEG made each screenshot roughly twice as fast, dropping from about 100 ms to about 50 ms. That is not a micro-optimization when screenshot capture sits inside every click-confirmation cycle. It changes the feel of the whole state engine.</p>
<p>That does not mean “always use JPEG”. OCR crops, audit artifacts, and failure snapshots may still deserve lossless PNG. The useful distinction is between image artifacts as records and screenshots as sensor reads. Records optimize for fidelity and inspectability. Sensor reads also have to optimize for latency.</p>
<p>If screenshots are in your inner loop, encoding and transport are in your inner loop too.</p>
<h2 id="runtime-beats-cleverness">Runtime Beats Cleverness</h2>
<p>The runtime layer is less glamorous than the vision logic, but it is what makes the system useful.</p>
<p>An autonomous visual collector needs more than a successful happy path. It needs to know when to start. It needs to retry after failure without manual intervention. It needs to stop safely in the middle of a run. It needs to leave behind enough timeline events to explain what happened. It needs to persist partial evidence before taking destructive or irreversible actions, because after a successful click the thing you meant to inspect may be gone.</p>
<p>That last point is important. In visual automation, the evidence often exists only before the action. Capture first, act second, confirm third. If the run dies after the click, the pre-action evidence should not be lost. If the next state is ambiguous, the runtime should be able to mark it for review instead of pretending the input was perfect.</p>
<p>The runtime should also measure itself:</p>
<ul>
<li>average operation cycle time;</li>
<li>average post-click confirmation time;</li>
<li>screenshot capture latency;</li>
<li>retry counts;</li>
<li>fallback path usage;</li>
<li>ambiguous-state rate;</li>
<li>artifact volume;</li>
<li>estimated throughput under current timing.</li>
</ul>
<p>Those numbers are not vanity metrics. They are how you decide whether a change actually made the system faster or merely moved the waiting around.</p>
<h2 id="where-this-pattern-applies">Where This Pattern Applies</h2>
<p>This pattern is not specific to games. It shows up anywhere the useful state is rendered but not exposed:</p>
<ul>
<li>browser apps that hide their real interface in a canvas;</li>
<li>remote desktop or VNC automation;</li>
<li>streamed enterprise software;</li>
<li>kiosk-style web views;</li>
<li>legacy applications with poor accessibility hooks;</li>
<li>visual QA systems;</li>
<li>hardware dashboards rendered through video capture;</li>
<li>any workflow where the only reliable source of truth is the screen.</li>
</ul>
<p>All of these systems have the same fork in the road. You can scan dynamically and pay for robustness. You can calibrate pixels and pay for brittleness. Or you can build a hybrid where fixed coordinates drive the hot path and small probes prove that each transition happened.</p>
<p>Reach for dynamic scanning when:</p>
<ul>
<li>the UI moves often;</li>
<li>the action is rare enough that performance does not matter much;</li>
<li>the cost of a false click is high;</li>
<li>you do not yet know the stable geometry.</li>
</ul>
<p>Reach for calibrated pixels when:</p>
<ul>
<li>the same interaction happens many times;</li>
<li>the viewport can be controlled;</li>
<li>there is a cheap visual confirmation after the click;</li>
<li>failures produce artifacts that make recalibration straightforward.</li>
</ul>
<p>The practical design usually lands in the middle. It is not a pure computer-vision system. It is not a pile of magic coordinates. It is a service built around visual state: explicit steps, visual anchors, runtime control, calibrated hot-path inputs, and enough image evidence to keep the whole thing honest.</p>
<p>When the application state only exists as pixels, do not pretend you are still automating the DOM. Build the state machine where the state actually is.</p>
Efficient NixOS Remote Deploys with Selective Closure Copying2026-04-15T11:00:00+00:002026-04-15T11:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/nixos-remote-deploys-selective-closure-copying/<p>You run <code>nixos-rebuild switch --target-host</code>, wait for the build to finish, and then watch your machine upload what feels like the entire internet to a server that could have fetched most of those paths from <code>cache.nixos.org</code> on its own.</p>
<p>This is one of those Nix deployment problems that looks inevitable until you learn the flag.</p>
<p>If you are deploying to a remote NixOS host, the first thing to try is:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nixos-rebuild switch \
--flake ".#myhost" \
--target-host deploy@example.com \
--use-remote-sudo \
--use-substitutes
</code></pre>
<p>That single <code>--use-substitutes</code> changes the copy behavior in a way that can make deploys dramatically faster. Instead of blindly pushing the full closure from your laptop or CI runner, the remote host gets a chance to fetch missing store paths from its own configured substituters first. In practice, that usually means the remote pulls public dependencies from <code>cache.nixos.org</code>, and you only upload the paths that are actually private or unavailable in cache.</p>
<p>That is the whole trick.</p>
<h2 id="the-default-behavior-is-wasteful">The default behavior is wasteful</h2>
<p>Without substitute-on-destination behavior, a remote deploy often looks like this:</p>
<ol>
<li>You build locally</li>
<li><code>nixos-rebuild</code> copies the resulting closure to the target host</li>
<li>The target activates the new system</li>
</ol>
<p>The problem is step two. Your local machine ends up pushing standard <code>nixpkgs</code> dependencies that the remote could have downloaded perfectly well on its own from a fast public cache.</p>
<p>So if your system closure includes <code>systemd</code>, <code>nginx</code>, <code>openssl</code>, <code>glibc</code>, and a hundred other ordinary paths, you are pointlessly using your own outbound bandwidth as a binary cache.</p>
<p>That hurts most on:</p>
<ul>
<li>home uplinks with weak upload bandwidth</li>
<li>remote VPS deploys over higher-latency links</li>
<li>CI runners pushing to many hosts</li>
<li>large closures where only a tiny fraction is actually custom</li>
</ul>
<h2 id="what-use-substitutes-actually-does">What <code>--use-substitutes</code> actually does</h2>
<p>The high-level flag is <code>--use-substitutes</code> on <code>nixos-rebuild</code>.</p>
<p>The lower-level behavior underneath it is <code>nix copy --substitute-on-destination</code>. The Nix reference manual describes that flag as telling the destination SSH store to try substitutes on the destination side. That is the mechanism you want.</p>
<p>So conceptually, you are switching from this:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">my machine -> push everything -> remote host
</code></pre>
<p>to this:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">my machine -> push only uncached paths -> remote host
remote host -> fetch public paths -> cache.nixos.org / other substituters
</code></pre>
<p>That is almost always the better network topology.</p>
<h2 id="the-one-flag-version">The one-flag version</h2>
<p>For many setups, the simple version is enough:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run nixpkgs#nixos-rebuild -- switch \
--flake ".#myhost" \
--target-host deploy@example.com \
--use-remote-sudo \
--use-substitutes
</code></pre>
<p>That keeps your deploy wrapper short and gets you most of the benefit immediately.</p>
<p>If your current deploy script looks like this:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run nixpkgs#nixos-rebuild -- switch \
--flake ".#myhost" \
--target-host deploy@example.com \
--use-remote-sudo
</code></pre>
<p>the diff is almost embarrassingly small:</p>
<pre data-lang="diff" class="language-diff "><code class="language-diff" data-lang="diff"> nix run nixpkgs#nixos-rebuild -- switch \
--flake ".#myhost" \
--target-host deploy@example.com \
- --use-remote-sudo
+ --use-remote-sudo \
+ --use-substitutes
</code></pre>
<p>And yet that tiny diff can be the difference between “ship the whole closure over SSH” and “let the remote fetch the boring stuff itself.”</p>
<h2 id="the-trusted-user-gotcha">The trusted-user gotcha</h2>
<p>This is the part many writeups skip.</p>
<p>If you deploy as a non-root user on the remote machine, that user needs to be trusted by the remote Nix daemon. Otherwise the substitute request can be ignored and you quietly fall back to the slow path.</p>
<p>On NixOS, that usually means:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
nix.settings.trusted-users = [ "root" "deploy" ];
}
</code></pre>
<p>If you deploy as <code>root@host</code>, root is already trusted. If you deploy as <code>deploy@host</code> and rely on <code>--use-remote-sudo</code> for activation, then <code>deploy</code> needs to be listed in <code>trusted-users</code>.</p>
<p>This matches the Nix documentation more broadly: using substituters through the daemon is a privileged capability, and the calling user needs to be trusted for it to work as intended.</p>
<p>If you forget this step, the failure mode is annoying because it often looks like nothing is wrong. The deploy still succeeds. It is just slow, and the target host behaves as though <code>--use-substitutes</code> was never there.</p>
<h2 id="how-to-tell-whether-it-is-working">How to tell whether it is working</h2>
<p>If your deploys are still dragging, check the basics:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># On the remote host
nix show-config | grep trusted-users
</code></pre>
<p>And pay attention to the copy phase. The fast path looks like the target is fetching public dependencies for itself. The slow path looks like one long stream of data being shoved from your local store to the remote store.</p>
<p>The easiest sanity check is often practical: if adding <code>--use-substitutes</code> did not reduce upload volume or deploy time, the trusted-user piece is the first thing to verify.</p>
<h2 id="the-explicit-three-step-version">The explicit three-step version</h2>
<p>Sometimes you want more control than <code>nixos-rebuild</code> gives you in one command. Maybe you want to pre-stage a system closure, inspect copy behavior, or separate build time from transfer time in your logs.</p>
<p>That is where the lower-level pattern is useful:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">TOPLEVEL=$(nix build ".#nixosConfigurations.myhost.config.system.build.toplevel" \
--print-out-paths \
--no-link)
nix copy \
--to "ssh-ng://deploy@example.com" \
--substitute-on-destination \
"$TOPLEVEL"
nixos-rebuild switch \
--flake ".#myhost" \
--target-host deploy@example.com \
--use-remote-sudo
</code></pre>
<p>This gives you three distinct phases:</p>
<ol>
<li>Build the target system locally</li>
<li>Copy the closure while allowing the destination to substitute what it can</li>
<li>Switch the remote system</li>
</ol>
<p>That is handy when you are debugging performance, staging to multiple machines, or just want a clearer operational story than “one giant command did a lot of things.”</p>
<p>You can also dry-run the copy phase:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix copy \
--to "ssh-ng://deploy@example.com" \
--substitute-on-destination \
--dry-run \
"$TOPLEVEL"
</code></pre>
<p>That is a nice way to see whether you are really pushing only the private or uncached store paths.</p>
<h2 id="when-this-helps-most">When this helps most</h2>
<p>This pattern is especially effective when your closure contains a small amount of private work layered on top of a lot of standard public dependencies.</p>
<p>Examples:</p>
<ul>
<li>a mostly normal NixOS system plus a private application package</li>
<li>a public service stack plus one custom overlay</li>
<li>a host whose secrets are handled separately but whose system packages are mostly upstream</li>
</ul>
<p>In those cases, letting the destination substitute public paths turns deploys from “upload gigabytes” into “ship the weird bits.”</p>
<h2 id="when-it-does-not-help-much">When it does not help much</h2>
<p>There are real limits.</p>
<p>If most of your closure is custom and uncached, then the remote still has to get that data from somewhere, and that somewhere is probably you. <code>--use-substitutes</code> does not magically make private builds public.</p>
<p>It also will not help if the remote host cannot reach its substituters, or if your cache configuration is incomplete.</p>
<p>So the pattern is best understood as:</p>
<ul>
<li>use destination-side substitutes for public paths</li>
<li>use your own upload bandwidth for the private leftovers</li>
</ul>
<p>That is still a huge win, just not infinite.</p>
<h2 id="pair-it-with-a-private-cache">Pair it with a private cache</h2>
<p>If you want the best version of this workflow, pair it with your own binary cache as well.</p>
<p>For example:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
nix.settings = {
trusted-users = [ "root" "deploy" ];
substituters = [
"https://cache.nixos.org"
"https://your-attic.example.com/main"
];
trusted-public-keys = [
"cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
"main:your-attic-public-key-here"
];
};
}
</code></pre>
<p>Now the remote can fetch both public dependencies and your private builds without waiting for a direct upload from the deployer. That is the point where remote deploys start feeling unfairly fast.</p>
<p>If you want to set that up yourself, I already wrote about <a href="/blog/self-hosted-nix-binary-cache-attic-garage/">self-hosting a Nix binary cache with Attic and Garage</a>.</p>
<h2 id="the-practical-rule">The practical rule</h2>
<p>If you deploy to a remote NixOS host, add <code>--use-substitutes</code> first and verify that the remote deploy user is trusted.</p>
<p>That is the highest-leverage fix.</p>
<p>Once that is in place, move to the explicit <code>nix build</code> plus <code>nix copy --substitute-on-destination</code> pattern when you need more visibility or control. And if you deploy often, add a private cache so the “custom bits” stop being uploads too.</p>
<p>You do not need to keep re-uploading the world to machines that already know how to fetch it.</p>
NUR — The Distributed Package Registry Nix Deserves2026-04-14T18:00:00+00:002026-04-14T18:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/nur-distributed-package-registry-nix/<p>If you come from Arch, the AUR feels like the obvious model for community packaging: one big shared repository, everyone submits PKGBUILDs, and the collective package count turns into a kind of ecosystem scoreboard.</p>
<p>NUR takes a different path, and I think it is the better one.</p>
<p>NUR is not a monorepo of user packages. It is a registry of user repositories. Each contributor owns their own repo, publishes whatever packages, modules, and overlays they want, and NUR indexes the result. At the time of writing, the <a rel="noopener noreferrer" target="_blank" href="https://nur.nix-community.org/">NUR package index</a> shows 5,944 packages across 369 user repositories. That is a lot of ecosystem hiding in plain sight.</p>
<p>If you have not explored it yet, you are probably underestimating what the Nix community has already packaged.</p>
<h2 id="what-nur-actually-is">What NUR actually is</h2>
<p>The important mental model is this:</p>
<ul>
<li><strong>AUR</strong> is a shared submission queue</li>
<li><strong>NUR</strong> is a federated directory</li>
</ul>
<p>That difference sounds subtle until you use it.</p>
<p>With AUR, the package lives inside the central repo. With NUR, the package lives in the maintainer’s repo, and NUR points you to it. The registry is the index, not the home.</p>
<p>That makes NUR feel much more like the rest of the Nix ecosystem. You are already used to composing inputs, pinning revisions, wiring overlays together, and deciding exactly what enters your system closure. NUR fits that model naturally.</p>
<h2 id="why-this-architecture-is-better">Why this architecture is better</h2>
<p>The simplest comparison looks like this:</p>
<table><thead><tr><th>AUR</th><th>NUR</th></tr></thead><tbody>
<tr><td>One monolithic community repo</td><td>Many independent repos collected in one index</td></tr>
<tr><td>Submit PKGBUILDs into the shared tree</td><td>Publish in your own repo</td></tr>
<tr><td>Centralized packaging workflow</td><td>Federated packaging workflow</td></tr>
<tr><td>Mostly package recipes</td><td>Packages, overlays, modules, and repo-specific conventions</td></tr>
<tr><td>Trust is broad and social</td><td>Trust can be selective and explicit</td></tr>
</tbody></table>
<p>That last point matters more than it first appears.</p>
<p>In NUR, you do not have to think in all-or-nothing terms. You can consume the whole registry via the NUR overlay if you want the giant package namespace, but you can also pull in one maintainer’s repo directly, pin it to a commit, and use only the parts you care about. That is a better fit for how serious Nix users already manage risk and reproducibility.</p>
<p>It is not that AUR is bad. The AUR is one of Arch’s best ideas. But NUR solves the same community-packaging problem in a way that feels much more native to Nix.</p>
<h2 id="discoverability-is-much-better-than-people-assume">Discoverability is much better than people assume</h2>
<p>One of the classic arguments for a monorepo is discoverability: if everything lives in one place, surely it is easier to find.</p>
<p>But NUR already gives you the searchable index people actually need. <a rel="noopener noreferrer" target="_blank" href="https://nur.nix-community.org/">nur.nix-community.org</a> lets you search across the full registry, browse repositories, and inspect what maintainers are publishing. You get the discovery benefits of aggregation without forcing everyone into one git history and one submission model.</p>
<p>That turns out to be a very good trade.</p>
<p>You can stumble across exactly the kind of wonderfully niche packaging work that thrives in Nix:</p>
<ul>
<li>Prometheus exporters that are too specific for <code>nixpkgs</code></li>
<li>Firefox extension collections packaged as reproducible derivations</li>
<li>SDDM themes that become one declarative option instead of manual file copying</li>
<li>Small infrastructure tools, IRC bots, relay daemons, and odd little utilities that absolutely belong somewhere even if they are not mainstream enough for <code>nixpkgs</code></li>
</ul>
<p>That is where NUR shines. It is the long tail of the Nix ecosystem, but with structure.</p>
<h2 id="it-is-not-just-packages">It is not just packages</h2>
<p>This is the real differentiator.</p>
<p>AUR gives you install recipes. NUR repos can give you full Nix building blocks: packages, overlays, NixOS modules, Home Manager modules, helper libraries, and flake outputs.</p>
<p>That means the thing you discover is often not just “here is a binary.” It is “here is a binary, plus a module, plus sane service wiring, plus options, plus a reusable overlay if you want to patch it.”</p>
<p>That is a much richer packaging story.</p>
<p>For example, if you consume a NUR repo directly as a flake input, you can wire its modules straight into your system:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
nur-ijohanne.url = "github:ijohanne/nur-packages";
};
outputs = { nixpkgs, nur-ijohanne, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
nur-ijohanne.nixosModules.prometheus-hue-exporter
{
services.prometheus-hue-exporter.enable = true;
}
];
};
};
}
</code></pre>
<p>And if you want the big shared namespace instead, the upstream NUR overlay works too:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs.nur.url = "github:nix-community/NUR";
outputs = { nixpkgs, nur, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
nur.modules.nixos.default
({ pkgs, ... }: {
environment.systemPackages = [
pkgs.nur.repos.mic92.goatcounter
pkgs.nur.repos.ijohanne.zot
];
})
];
};
};
}
</code></pre>
<p>That flexibility is the point. You can browse globally and consume locally.</p>
<h2 id="a-concrete-case-study-my-own-nur-repo">A concrete case study: my own NUR repo</h2>
<p>My repository at <a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/nur-packages">ijohanne/nur-packages</a> is exactly why I like this model.</p>
<p>It contains a mix of things that would be awkward to pitch as one giant upstream contribution story:</p>
<ul>
<li>Prometheus exporters for Hue, Netatmo, nftables, TeamSpeak3, Ecowitt, TP-Link P110, and PostgreSQL</li>
<li>NixOS modules for those exporters, not just package derivations</li>
<li>A hardened Firefox profile and curated Firefox addon packaging</li>
<li>Fish and Vim plugin sets</li>
<li>SDDM themes</li>
<li><code>zot</code>, <code>multicast-relay</code>, and other niche infrastructure tools</li>
<li><code>agent-skills-cli</code></li>
</ul>
<p>That is a coherent repo because it is <em>my</em> repo. It reflects what I run, maintain, and care about. And that is precisely why the federated model works: each maintainer can publish a package universe that makes sense for them without asking a central project to absorb the whole identity of it.</p>
<p>NUR turns that into something discoverable and reusable for everybody else.</p>
<h2 id="the-trust-model-is-nicer">The trust model is nicer</h2>
<p>In practice, NUR gives you a more granular trust decision than a central community repo.</p>
<p>You can decide:</p>
<ul>
<li>which repos you want to use</li>
<li>which commits you want to pin</li>
<li>whether you want the full overlay or a single upstream</li>
<li>whether you want only packages or also modules and overlays</li>
</ul>
<p>That maps directly to how Nix users already think. Reproducibility is not just about exact builds. It is also about being explicit about your inputs.</p>
<p>With NUR, the social layer and the technical layer line up more cleanly. You can say “I trust this maintainer’s repo for this host” in an actual, concrete, pin-able way.</p>
<h2 id="how-to-create-your-own-nur-repo">How to create your own NUR repo</h2>
<p>The official <a rel="noopener noreferrer" target="_blank" href="https://github.com/nix-community/NUR">NUR repository</a> expects each upstream to expose a top-level <code>default.nix</code>. In practice, many repos also expose a flake so they are easy to consume directly outside the shared overlay.</p>
<p>At the minimum, your package repo can look like this:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{ pkgs }:
{
hello-nur = pkgs.callPackage ./pkgs/hello-nur { };
}
</code></pre>
<p>If you want a flake interface too, you can layer that on top:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
description = "My NUR packages";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
outputs = { self, nixpkgs }:
let
system = "x86_64-linux";
pkgs = import nixpkgs { inherit system; };
in {
legacyPackages.${system} = import ./default.nix { inherit pkgs; };
};
}
</code></pre>
<p>Once the repo exists, you add it to <code>repos.json</code> in <a rel="noopener noreferrer" target="_blank" href="https://github.com/nix-community/NUR">nix-community/NUR</a>, open a PR, and your repo becomes part of the index. If you want updates to show up faster than the normal refresh cycle, NUR also exposes the <code>nur-update.nix-community.org</code> hook described in the upstream docs.</p>
<p>That onboarding path is refreshingly lightweight. You own your repo. NUR just helps people find it.</p>
<h2 id="why-this-matters">Why this matters</h2>
<p>The deeper story here is not “look, another package collection.”</p>
<p>The deeper story is that Nix packaging is already more composable than most ecosystems, and NUR preserves that composability instead of flattening it into one central contribution queue. It lets community packaging stay decentralized without becoming undiscoverable.</p>
<p>That is exactly the shape this ecosystem wants.</p>
<p>If you have only used <code>nixpkgs</code> plus the occasional flake input, spend half an hour browsing NUR. You will find packages you did not know existed, modules you did not realize someone had already written, and repos maintained by people solving the same weird little problems you are solving.</p>
<p>That is what a healthy package ecosystem looks like.</p>
My Codex App Workflow: Triage, Fix Threads, and Just Enough Research2026-04-13T00:14:00+00:002026-04-13T00:14:00+00:00
Ian Johannesen
https://perlpimp.net/blog/codex-app-thread-workflow/<p>There are two bad defaults when you start doing real work with a coding agent.</p>
<p>The first is one giant thread where everything happens forever. Ideas, debugging, random notes, half-formed plans, implementation, follow-up fixes. It feels convenient right up until the thread turns into soup and you cannot remember where a decision was made or which bit of context still matters.</p>
<p>The second is going full command-center mode with elaborate agent choreography for every piece of work. Parallel researchers, planners, implementers, reviewers, all passing notes to each other like you are running a tiny software consultancy inside your terminal. That can be impressive. It can also be a very efficient way to burn time and attention on process that does not actually help.</p>
<p>What I have settled into with the Codex app is a middle path: one mostly long-lived <code>Triage</code> thread, one focused <code>Fix issue-XXX</code> thread for each piece of implementation work, and occasional <code>Research XX</code> threads when I genuinely need to go learn something first.</p>
<p>It is not fancy. That is exactly why it works.</p>
<h2 id="the-problem-with-giant-threads">The problem with giant threads</h2>
<p>Long threads decay. Not immediately, but predictably.</p>
<p>At the start, a single thread feels efficient because everything is “right there”. After a while it becomes a junk drawer. A bug report sits next to a design question, which sits next to a shell transcript, which sits next to a half-baked idea you had at midnight. The agent can still technically see all of it, but that does not mean the context is useful.</p>
<p>What I care about in day-to-day use is not maximum theoretical continuity. I care about keeping the current task legible. I want the thread title to tell me what I am doing. I want the history to be easy to skim later. And I want the working context to stay small enough that the app remains fast and pleasant.</p>
<p>That means I do not want one immortal everything-thread.</p>
<h2 id="the-triage-thread-as-the-inbox">The <code>Triage</code> thread as the inbox</h2>
<p>The one long-lived exception is <code>Triage</code>.</p>
<p>That thread is my inbox. It is where I dump observations, rough ideas, “this seems off” notes, and small bits of investigation that are not yet implementation work. If I notice a bug while doing something else, it goes there. If I want to capture an idea for a post or a refactor without acting on it immediately, it goes there too.</p>
<p>This has effectively replaced the old <code>/btw</code> habit I had in Claude Code. The point is the same: capture now, sort later. The difference is that in the Codex app I like having a dedicated thread that stays open and accumulates that intake work over time.</p>
<p>The important thing is that <code>Triage</code> is not where I finish work. It is where I decide what the work is.</p>
<h2 id="github-issues-sits-in-the-middle">GitHub Issues sits in the middle</h2>
<p>From <code>Triage</code>, I file actual tasks into GitHub Issues, or any similar issue tracker an agent can handle cleanly.</p>
<p>That is the durable ledger. Threads are great for active conversation and local context. They are not where I want my task list to live. I want a real issue tracker that I can search, sort, sync, and return to later without mentally reconstructing a conversation.</p>
<p>So the flow is simple:</p>
<ol>
<li>Something comes up in <code>Triage</code>.</li>
<li>If it is real work, I file a GitHub issue.</li>
<li>The issue becomes the handoff point from vague thought to concrete task.</li>
</ol>
<p>That separation matters. It keeps the thread free to be conversational, while the issue tracker stays authoritative about what exists, what is in progress, and what still needs doing.</p>
<h2 id="one-fix-issue-xxx-thread-per-implementation-task">One <code>Fix issue-XXX</code> thread per implementation task</h2>
<p>Once I actually start working an issue, I spin up a separate thread named something like <code>Fix issue-2od</code>.</p>
<p>This is probably the single habit that improved the experience the most.</p>
<p>Instead of dragging a whole triage history behind me, I get a clean workspace focused on one job. The thread name is searchable. The history is about one thing. If I need to come back in a week, I do not have to decode whether a command or explanation was related to this issue or some other thing that happened nearby.</p>
<p>This also makes the agent feel better to use. Keeping the active context narrow tends to keep responses snappier and the conversation more relevant. I do not need the model to remember every adjacent concern I had this month. I need it to help me finish the thing in front of me.</p>
<p>Thread naming is part of the system here, not an afterthought. If I can scan the sidebar and immediately see <code>Triage</code>, <code>Fix issue-2od</code>, <code>Fix issue-31a</code>, and <code>Research Zola feeds</code>, I have already reduced a bunch of cognitive load before opening anything.</p>
<p><img src="https://perlpimp.net/blog/codex-app-thread-workflow/triage-threads-sidebar.png" alt="Codex app sidebar with triage, fix, and research threads" /></p>
<h2 id="research-xx-threads-are-for-uncertainty-not-theater"><code>Research XX</code> threads are for uncertainty, not theater</h2>
<p>I do sometimes open separate research threads, but only when the task is genuinely exploratory.</p>
<p>If I need to understand a library, compare approaches, or verify how something works before I decide what to implement, that gets a <code>Research XX</code> thread. The key is that research is its own mode. It has a different shape than implementation, and I would rather keep that separate than dilute a fix thread with a bunch of speculative branching.</p>
<p>These research threads often end the same way: by producing one or more new GitHub issues.</p>
<p>That is useful because it means the exploration is not just “context”. It turns into concrete follow-up work with clear boundaries. Once that happens, I can close the research loop and go back to the per-issue thread pattern.</p>
<h2 id="why-this-works-better-than-over-orchestration">Why this works better than over-orchestration</h2>
<p>This workflow is intentionally not an agent-swarm post.</p>
<p>I am not trying to simulate a company org chart inside the app. I do not want elaborate coordination machinery unless the problem genuinely demands it. Most day-to-day engineering work benefits more from clear boundaries than from more moving parts.</p>
<p>One inbox thread. One issue tracker. One focused implementation thread per issue. Research threads when they earn their keep.</p>
<p>That is enough structure to keep things tidy without making the workflow feel ceremonial.</p>
<h2 id="small-context-faster-app-better-history">Small context, faster app, better history</h2>
<p>The biggest benefit is that everything stays small.</p>
<p>Small context means the active thread is easier to reason about. It usually means the app feels faster. It means when I search later, I find a thread whose title actually matches the work that happened inside it. And it means I do not dread opening old conversations because each one has a clear purpose.</p>
<p>That, more than anything, is what I want from a practical coding-agent workflow. Not maximal sophistication. Not a benchmark-optimized orchestration strategy. Just a system that stays readable, searchable, and pleasant after the novelty wears off.</p>
<p>For me, this is the sweet spot: enough structure to keep momentum, not so much structure that the workflow becomes its own hobby.</p>
Monitoring nftables Firewall Rules with Prometheus and Grafana2026-04-12T00:00:00+00:002026-04-12T00:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/monitoring-nftables-prometheus-grafana/<p>nftables already counts packets and bytes for you. The frustrating part is that it keeps that data trapped inside <code>nft list ruleset</code>, tied to handle numbers that are useless unless you are staring at the exact ruleset on the exact machine at the exact moment something went wrong.</p>
<p>That is fine for one-off debugging. It is terrible for operations.</p>
<p>If you run your own router or firewall, you eventually want the same thing you want everywhere else: time-series data, alerting, dashboards, and enough context to tell the difference between “the WAN is noisy tonight” and “something is hammering the guest network and getting dropped exactly as intended.”</p>
<p>The trick that makes this workable is surprisingly small: put a <code>comment</code> on every nftables rule, then export that comment as a Prometheus label.</p>
<p>Once you do that, you stop looking at this:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">nftables_rule_packets{table="filter",chain="input",handle="15"} 48291
</code></pre>
<p>and start looking at this:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">nftables_rule_packets{table="filter",chain="input",handle="15",comment="wan drop",action="drop"} 48291
</code></pre>
<p>That one extra label turns an opaque counter into something you can reason about at a glance.</p>
<h2 id="why-handle-numbers-are-the-wrong-abstraction">Why handle numbers are the wrong abstraction</h2>
<p>nftables exposes counters per rule with the <code>counter</code> keyword. The data is there, but it is buried in the live ruleset:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">sudo nft list ruleset
</code></pre>
<p>That gives you a snapshot, not a history. You cannot ask what your drop rate looked like over the last 24 hours. You cannot correlate a spike in blocked packets with a WireGuard handshake failure, a Prometheus alert, or some guest VLAN misbehavior. And if your dashboards key off handle numbers, a rebuild or ruleset reload can make them meaningless.</p>
<p>Comments solve the identity problem.</p>
<p>Handles are implementation details. Comments are intent.</p>
<h2 id="comments-as-labels">Comments as labels</h2>
<p>The rule here is simple: every rule that matters gets both a <code>counter</code> and a <code>comment</code>.</p>
<p>I keep the comments short, lowercase, and boring on purpose. Two to four words is usually enough. The pattern is generally <code><zone/interface> <action/purpose></code>:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text"># Format: "<scope> <function>" or "<description>"
ct state invalid counter drop comment "invalid state"
ip saddr 10.0.0.0/8 tcp dport 53 counter accept comment "lan dns tcp"
ip saddr 10.0.0.0/8 udp dport 53 counter accept comment "lan dns udp"
ip saddr 10.0.0.0/8 ip protocol icmp counter accept comment "lan icmp"
iifname { "ppp0", "mobile" } icmp type echo-request limit rate 5/second burst 10 packets counter accept comment "wan ping ratelimit"
iifname { "ppp0", "mobile" } ip protocol icmp counter drop comment "wan icmp drop"
ip saddr 0.0.0.0/0 udp dport 51820 counter accept comment "wireguard"
iifname { "wifi", "wired", "mgnt", "enp1s0f1", "lo", "wg0" } counter accept comment "trusted ifaces"
iifname "guest" udp dport { 53, 67, 68 } counter accept comment "guest dns+dhcp"
iifname "guest" counter drop comment "guest drop"
iifname "camera" counter drop comment "camera drop"
iifname "ppp0" ct state { established, related } counter accept comment "wan established"
iifname "ppp0" counter drop comment "wan drop"
log prefix "nft-input-drop: " counter drop comment "default drop"
</code></pre>
<p>The useful properties of this convention are:</p>
<ul>
<li>Every rule you care about is observable.</li>
<li>The label is stable even when nftables handle numbers change.</li>
<li>The label is human-readable in Grafana legends and PromQL results.</li>
<li>The label lines up with your mental model of the firewall instead of nftables internals.</li>
</ul>
<p>The last point matters more than it sounds. “wan drop” tells you what the rule is for. “handle 15” tells you where to start digging.</p>
<h2 id="a-complete-input-chain-with-comments">A complete input chain with comments</h2>
<p>This is what the pattern looks like in practice on a router with trusted interfaces, guest isolation, a camera network, and WireGuard:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">chain input {
type filter hook input priority filter; policy drop;
ct state invalid counter drop comment "invalid state"
ip saddr 10.0.0.0/8 tcp dport 53 counter accept comment "lan dns tcp"
ip saddr 10.0.0.0/8 udp dport 53 counter accept comment "lan dns udp"
ip saddr 10.0.0.0/8 ip protocol icmp counter accept comment "lan icmp"
iifname { "ppp0", "mobile" } icmp type { destination-unreachable, time-exceeded, parameter-problem, echo-reply } counter accept comment "wan icmp replies"
iifname { "ppp0", "mobile" } icmp type echo-request limit rate 5/second burst 10 packets counter accept comment "wan ping ratelimit"
iifname { "ppp0", "mobile" } ip protocol icmp counter drop comment "wan icmp drop"
ip saddr 0.0.0.0/0 udp dport 51820 counter accept comment "wireguard"
iifname { "wifi", "wired", "mgnt", "enp1s0f1", "lo", "wg0" } counter accept comment "trusted ifaces"
iifname "guest" udp dport { 53, 67, 68 } counter accept comment "guest dns+dhcp"
iifname "guest" tcp dport 53 counter accept comment "guest dns tcp"
iifname "guest" ct state { established, related } counter accept comment "guest established"
iifname "guest" counter drop comment "guest drop"
iifname "camera" udp dport { 53, 67, 68 } counter accept comment "camera dns+dhcp"
iifname "camera" tcp dport 53 counter accept comment "camera dns tcp"
iifname "camera" ct state { established, related } counter accept comment "camera established"
iifname "camera" counter drop comment "camera drop"
ip protocol igmp counter accept comment "igmp"
ip saddr 224.0.0.0/4 counter accept comment "multicast"
iifname "ppp0" ct state { established, related } counter accept comment "wan established"
iifname "ppp0" counter drop comment "wan drop"
iifname "mobile" ct state { established, related } counter accept comment "mobile established"
iifname "mobile" counter drop comment "mobile drop"
log prefix "nft-input-drop: " counter drop comment "default drop"
}
</code></pre>
<p>There are two practical habits doing most of the work here:</p>
<ol>
<li>Every terminal rule has a <code>counter</code>.</li>
<li>Every terminal rule has a comment that explains intent, not syntax.</li>
</ol>
<p>That is what makes the exporter and the dashboard useful later.</p>
<h2 id="forward-and-nat-chains">Forward and NAT chains</h2>
<p>The same comment pattern carries over cleanly into forwarding and NAT:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">chain forward {
meta oiftype ppp tcp flags syn tcp option maxseg size set 1452 counter comment "ppp mss clamp"
type filter hook forward priority filter; policy drop;
ip protocol { tcp, udp } ct state established ct status ! dnat flow add @fastnat counter comment "flow offload non-dnat"
ct state invalid counter drop comment "invalid state"
iifname { "guest", "wifi", "wired", "mgnt", "enp1s0f1", "wg0" } oifname { "ppp0", "enp1s0f1", "mobile" } counter accept comment "lan to internet"
iifname "camera" ip saddr 10.255.200.3 oifname { "ppp0", "enp1s0f1", "mobile" } counter accept comment "unvr to internet"
iifname { "ppp0", "enp1s0f1", "mobile" } oifname { "wifi", "wired", "mgnt", "guest", "camera", "wg0" } ct state established,related counter accept comment "wan return"
iifname { "wifi", "wired", "mgnt", "enp1s0f1", "wg0" } oifname { "wifi", "wired", "mgnt", "guest", "camera", "wg0" } counter accept comment "inter-vlan"
iifname { "wifi", "wired", "mgnt", "wg0" } oifname "camera" counter accept comment "lan to camera"
iifname "camera" oifname { "wifi", "wired", "mgnt", "wg0" } ct state established,related counter accept comment "camera return"
iifname { "guest", "camera" } oifname "wired" ip daddr 10.255.101.202 udp dport 123 counter accept comment "guest+camera ntp"
log prefix "nft-forward-drop: " counter drop comment "default drop"
}
</code></pre>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">table ip nat {
chain postrouting {
type nat hook postrouting priority filter; policy accept;
oifname "ppp0" counter masquerade comment "ppp0 masquerade"
oifname "enp1s0f1" counter masquerade comment "vlan masquerade"
oifname "mobile" counter masquerade comment "mobile masquerade"
iifname "wired" oifname "wired" ct status dnat counter masquerade comment "hairpin nat"
}
}
</code></pre>
<p>Once you commit to this pattern, the ruleset reads better even before Prometheus gets involved.</p>
<h2 id="exporting-nftables-to-prometheus">Exporting nftables to Prometheus</h2>
<p>The exporter on this router is a custom NixOS module, <code>prometheus-nftables-exporter</code>, bundled in my <a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/nur-packages"><code>ijohanne/nur-packages</code></a> repository. It runs locally, executes <code>nft -j list ruleset</code>, parses the JSON, and emits Prometheus metrics for rule counters plus a small amount of table and chain structure data.</p>
<p>If you want to pull it into a NixOS configuration, the short version is to add the repo as a flake input and import the module:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
ijohanne-nur.url = "github:ijohanne/nur-packages";
};
outputs = { nixpkgs, ijohanne-nur, ... }: {
nixosConfigurations.router = nixpkgs.lib.nixosSystem {
modules = [
ijohanne-nur.nixosModules.prometheus-nftables-exporter
./configuration.nix
];
};
};
}
</code></pre>
<p>Once the module is imported, the service config stays small:</p>
<p>The NixOS configuration is intentionally small:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{ ... }:
{
services.prometheus-nftables-exporter = {
enable = true;
enableLocalScraping = true;
};
}
</code></pre>
<p><code>enableLocalScraping = true</code> is the nice part. The router’s local Prometheus instance picks the exporter up automatically, so there is no extra scrape stanza to maintain somewhere else.</p>
<p>The exporter loop is straightforward:</p>
<ol>
<li>Run <code>nft -j list ruleset</code>.</li>
<li>Walk the tables, chains, and rules.</li>
<li>Extract packet counters, byte counters, handle, action, table, chain, family, and comment.</li>
<li>Expose those fields as Prometheus metrics.</li>
</ol>
<h2 id="metrics-and-labels">Metrics and labels</h2>
<p>The rule-level metrics look like this:</p>
<table><thead><tr><th>Metric</th><th>Type</th><th>Labels</th><th>Description</th></tr></thead><tbody>
<tr><td><code>nftables_rule_packets</code></td><td>counter</td><td><code>table</code>, <code>chain</code>, <code>family</code>, <code>handle</code>, <code>action</code>, <code>comment</code></td><td>Packets matched by the rule</td></tr>
<tr><td><code>nftables_rule_bytes</code></td><td>counter</td><td><code>table</code>, <code>chain</code>, <code>family</code>, <code>handle</code>, <code>action</code>, <code>comment</code></td><td>Bytes matched by the rule</td></tr>
</tbody></table>
<p>And the structural metrics look like this:</p>
<table><thead><tr><th>Metric</th><th>Type</th><th>Labels</th><th>Description</th></tr></thead><tbody>
<tr><td><code>nftables_table_chains</code></td><td>gauge</td><td><code>table</code>, <code>family</code></td><td>Number of chains in each table</td></tr>
<tr><td><code>nftables_chain_rules</code></td><td>gauge</td><td><code>table</code>, <code>chain</code>, <code>family</code></td><td>Number of rules in each chain</td></tr>
<tr><td><code>nftables_up</code></td><td>gauge</td><td>none</td><td>Whether the exporter is healthy</td></tr>
</tbody></table>
<p>An example sample line:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">nftables_rule_packets{table="filter",chain="input",family="ip",handle="15",action="drop",comment="wan drop"} 48291
</code></pre>
<p>This is exactly why the comment label matters. Even if handle <code>15</code> becomes <code>24</code> after a rebuild, the thing you actually care about is still “wan drop”.</p>
<h2 id="the-first-promql-queries-that-matter">The first PromQL queries that matter</h2>
<p>Once the metrics exist, the first few queries are obvious and immediately useful:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text"># Rate of dropped packets per rule
rate(nftables_rule_packets{instance="$instance",action="drop"}[$__rate_interval])
# Input chain rules, only active series
rate(nftables_rule_packets{instance="$instance",chain="input",table="filter"}[$__rate_interval]) > 0
# Top 10 busiest rules by byte rate
topk(10, rate(nftables_rule_bytes{instance="$instance"}[$__rate_interval]))
# Total traffic across all counted rules
sum(rate(nftables_rule_bytes{instance="$instance"}[$__rate_interval]))
</code></pre>
<p>The key design choice is to query and display by <code>comment</code>, not by handle. Handles can still be useful for debugging against a live ruleset, but the dashboard identity should be based on names that survive rebuilds.</p>
<h2 id="building-a-grafana-dashboard-that-answers-real-questions">Building a Grafana dashboard that answers real questions</h2>
<p>The most important firewall panels are not the prettiest ones. They are the ones that answer:</p>
<ul>
<li>What is getting dropped right now?</li>
<li>Which chain is busy?</li>
<li>Is the guest or camera network doing something unusual?</li>
<li>Did a ruleset change remove an expected counter?</li>
</ul>
<p>That is why the first row I care about is drops and rejects, not total traffic.</p>
<p>The dashboard layout I ended up with looks like this:</p>
<h3 id="overview">Overview</h3>
<ul>
<li>Exporter status</li>
<li>Table count</li>
<li>Chain count</li>
<li>Rule count</li>
<li>Aggregate rule traffic</li>
</ul>
<h3 id="drops-and-rejects">Drops and rejects</h3>
<ul>
<li>Dropped packets by rule</li>
<li>Dropped bytes by rule</li>
<li>An all-drops view broken out by chain</li>
</ul>
<p>This is the panel group I watch first during scans, misconfigurations, or sudden noise on the WAN.</p>
<h3 id="chain-specific-rows">Chain-specific rows</h3>
<ul>
<li>Input packets and bytes by rule</li>
<li>Forward packets and bytes by rule</li>
<li>NAT prerouting and postrouting packets by rule</li>
<li>IPv6 input and forward packets by rule</li>
</ul>
<h3 id="top-rules-and-structure">Top rules and structure</h3>
<ul>
<li>Top 10 rules by bytes</li>
<li>Top 10 rules by packets</li>
<li>Table and chain structure tables</li>
</ul>
<p>The template variables are minimal:</p>
<ul>
<li><code>$datasource</code> for the Prometheus source</li>
<li><code>$instance</code> from <code>label_values(nftables_up, instance)</code></li>
</ul>
<h2 id="complete-dashboard-json">Complete dashboard JSON</h2>
<p>Import this via Grafana’s dashboard import UI, or provision it as a file:</p>
<pre data-lang="json" class="language-json "><code class="language-json" data-lang="json">{
"annotations": {
"list": []
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"links": [],
"panels": [
{
"collapsed": false,
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
"id": 100,
"title": "Overview",
"type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"mappings": [{ "options": { "0": { "text": "Down", "color": "red" }, "1": { "text": "Up", "color": "green" } }, "type": "value" }],
"thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] }
}
},
"gridPos": { "h": 4, "w": 4, "x": 0, "y": 1 },
"id": 1,
"options": { "colorMode": "background", "graphMode": "none", "reduceOptions": { "calcs": ["lastNotNull"] }, "textMode": "value" },
"title": "Exporter Status",
"type": "stat",
"targets": [{ "expr": "nftables_up{instance=\"$instance\"}", "refId": "A" }]
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "thresholds": { "mode": "absolute", "steps": [{ "color": "blue", "value": null }] } } },
"gridPos": { "h": 4, "w": 5, "x": 4, "y": 1 },
"id": 2,
"options": { "colorMode": "background", "graphMode": "none", "reduceOptions": { "calcs": ["lastNotNull"] }, "textMode": "value" },
"title": "Tables",
"type": "stat",
"targets": [{ "expr": "count(nftables_table_chains{instance=\"$instance\"})", "refId": "A" }]
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "thresholds": { "mode": "absolute", "steps": [{ "color": "blue", "value": null }] } } },
"gridPos": { "h": 4, "w": 5, "x": 9, "y": 1 },
"id": 3,
"options": { "colorMode": "background", "graphMode": "none", "reduceOptions": { "calcs": ["lastNotNull"] }, "textMode": "value" },
"title": "Chains",
"type": "stat",
"targets": [{ "expr": "sum(nftables_table_chains{instance=\"$instance\"})", "refId": "A" }]
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "thresholds": { "mode": "absolute", "steps": [{ "color": "blue", "value": null }] } } },
"gridPos": { "h": 4, "w": 5, "x": 14, "y": 1 },
"id": 4,
"options": { "colorMode": "background", "graphMode": "none", "reduceOptions": { "calcs": ["lastNotNull"] }, "textMode": "value" },
"title": "Rules",
"type": "stat",
"targets": [{ "expr": "sum(nftables_chain_rules{instance=\"$instance\"})", "refId": "A" }]
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "unit": "Bps", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] } } },
"gridPos": { "h": 4, "w": 5, "x": 19, "y": 1 },
"id": 5,
"options": { "colorMode": "background", "graphMode": "area", "reduceOptions": { "calcs": ["lastNotNull"] }, "textMode": "value" },
"title": "Total Rule Traffic",
"type": "stat",
"targets": [{ "expr": "sum(rate(nftables_rule_bytes{instance=\"$instance\"}[$__rate_interval]))", "refId": "A" }]
},
{
"collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 }, "id": 101, "title": "Drops & Rejects", "type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 1, "fillOpacity": 10, "spanNulls": false }, "unit": "pps" } },
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 6 },
"id": 10,
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] }, "tooltip": { "mode": "multi" } },
"title": "Dropped Packets (rate)",
"description": "All rules with drop action — key indicator of blocked traffic",
"type": "timeseries",
"targets": [{ "expr": "rate(nftables_rule_packets{instance=\"$instance\",action=\"drop\"}[$__rate_interval])", "legendFormat": "{{chain}} — {{comment}}", "refId": "A" }]
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 1, "fillOpacity": 10, "spanNulls": false }, "unit": "Bps" } },
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 6 },
"id": 11,
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] }, "tooltip": { "mode": "multi" } },
"title": "Dropped Bytes (rate)",
"type": "timeseries",
"targets": [{ "expr": "rate(nftables_rule_bytes{instance=\"$instance\",action=\"drop\"}[$__rate_interval])", "legendFormat": "{{chain}} — {{comment}}", "refId": "A" }]
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 1, "fillOpacity": 10, "spanNulls": false }, "unit": "pps" } },
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 14 },
"id": 12,
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] }, "tooltip": { "mode": "multi" } },
"title": "All Drops by Chain",
"description": "All rules with drop action — catch-all and explicit drops",
"type": "timeseries",
"targets": [{ "expr": "rate(nftables_rule_packets{instance=\"$instance\",action=\"drop\"}[$__rate_interval])", "legendFormat": "{{family}}/{{chain}} handle={{handle}}", "refId": "A" }]
},
{
"collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 22 }, "id": 102, "title": "Input Chain", "type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 1, "fillOpacity": 10, "spanNulls": false, "stacking": { "mode": "normal" } }, "unit": "pps" } },
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 23 },
"id": 20,
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "lastNotNull"] }, "tooltip": { "mode": "multi" } },
"title": "Input — Packets by Rule",
"type": "timeseries",
"targets": [{ "expr": "rate(nftables_rule_packets{instance=\"$instance\",chain=\"input\",table=\"filter\"}[$__rate_interval]) > 0", "legendFormat": "{{action}} {{comment}}", "refId": "A" }]
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 1, "fillOpacity": 10, "spanNulls": false, "stacking": { "mode": "normal" } }, "unit": "Bps" } },
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 23 },
"id": 21,
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "lastNotNull"] }, "tooltip": { "mode": "multi" } },
"title": "Input — Bytes by Rule",
"type": "timeseries",
"targets": [{ "expr": "rate(nftables_rule_bytes{instance=\"$instance\",chain=\"input\",table=\"filter\"}[$__rate_interval]) > 0", "legendFormat": "{{action}} {{comment}}", "refId": "A" }]
},
{
"collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 31 }, "id": 103, "title": "Forward Chain", "type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 1, "fillOpacity": 10, "spanNulls": false, "stacking": { "mode": "normal" } }, "unit": "pps" } },
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 32 },
"id": 30,
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "lastNotNull"] }, "tooltip": { "mode": "multi" } },
"title": "Forward — Packets by Rule",
"type": "timeseries",
"targets": [{ "expr": "rate(nftables_rule_packets{instance=\"$instance\",chain=\"forward\",table=\"filter\"}[$__rate_interval]) > 0", "legendFormat": "{{action}} {{comment}}", "refId": "A" }]
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 1, "fillOpacity": 10, "spanNulls": false, "stacking": { "mode": "normal" } }, "unit": "Bps" } },
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 32 },
"id": 31,
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "lastNotNull"] }, "tooltip": { "mode": "multi" } },
"title": "Forward — Bytes by Rule",
"type": "timeseries",
"targets": [{ "expr": "rate(nftables_rule_bytes{instance=\"$instance\",chain=\"forward\",table=\"filter\"}[$__rate_interval]) > 0", "legendFormat": "{{action}} {{comment}}", "refId": "A" }]
},
{
"collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 40 }, "id": 104, "title": "NAT", "type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 1, "fillOpacity": 10, "spanNulls": false, "stacking": { "mode": "normal" } }, "unit": "pps" } },
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 41 },
"id": 40,
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "lastNotNull"] }, "tooltip": { "mode": "multi" } },
"title": "NAT Prerouting — Packets by Rule",
"type": "timeseries",
"targets": [{ "expr": "rate(nftables_rule_packets{instance=\"$instance\",chain=\"prerouting\",table=\"nat\"}[$__rate_interval]) > 0", "legendFormat": "{{action}} {{comment}}", "refId": "A" }]
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 1, "fillOpacity": 10, "spanNulls": false, "stacking": { "mode": "normal" } }, "unit": "pps" } },
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 41 },
"id": 41,
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "lastNotNull"] }, "tooltip": { "mode": "multi" } },
"title": "NAT Postrouting — Packets by Rule",
"type": "timeseries",
"targets": [{ "expr": "rate(nftables_rule_packets{instance=\"$instance\",chain=\"postrouting\",table=\"nat\"}[$__rate_interval]) > 0", "legendFormat": "{{action}} {{comment}}", "refId": "A" }]
},
{
"collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 49 }, "id": 105, "title": "IPv6 Filter", "type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 1, "fillOpacity": 10, "spanNulls": false, "stacking": { "mode": "normal" } }, "unit": "pps" } },
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 50 },
"id": 50,
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "lastNotNull"] }, "tooltip": { "mode": "multi" } },
"title": "IPv6 Input — Packets by Rule",
"type": "timeseries",
"targets": [{ "expr": "rate(nftables_rule_packets{instance=\"$instance\",chain=\"input\",family=\"ip6\"}[$__rate_interval]) > 0", "legendFormat": "{{table}}/{{chain}} {{action}} {{comment}}", "refId": "A" }]
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 1, "fillOpacity": 10, "spanNulls": false, "stacking": { "mode": "normal" } }, "unit": "pps" } },
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 50 },
"id": 51,
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "lastNotNull"] }, "tooltip": { "mode": "multi" } },
"title": "IPv6 Forward — Packets by Rule",
"type": "timeseries",
"targets": [{ "expr": "rate(nftables_rule_packets{instance=\"$instance\",chain=\"forward\",family=\"ip6\"}[$__rate_interval]) > 0", "legendFormat": "{{table}}/{{chain}} {{action}} {{comment}}", "refId": "A" }]
},
{
"collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 58 }, "id": 106, "title": "Top Rules", "type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 1, "fillOpacity": 10, "spanNulls": false }, "unit": "Bps" } },
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 59 },
"id": 60,
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi" } },
"title": "Top 10 Rules by Bytes",
"type": "timeseries",
"targets": [{ "expr": "topk(10, rate(nftables_rule_bytes{instance=\"$instance\"}[$__rate_interval]))", "legendFormat": "{{table}}/{{chain}} {{action}} {{comment}}", "refId": "A" }]
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "lineWidth": 1, "fillOpacity": 10, "spanNulls": false }, "unit": "pps" } },
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 59 },
"id": 61,
"options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }, "tooltip": { "mode": "multi" } },
"title": "Top 10 Rules by Packets",
"type": "timeseries",
"targets": [{ "expr": "topk(10, rate(nftables_rule_packets{instance=\"$instance\"}[$__rate_interval]))", "legendFormat": "{{table}}/{{chain}} {{action}} {{comment}}", "refId": "A" }]
},
{
"collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 67 }, "id": 107, "title": "Structure", "type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": { "custom": { "align": "auto" }, "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [{ "color": "blue", "value": null }] } },
"overrides": [{ "matcher": { "id": "byName", "options": "Chains" }, "properties": [{ "id": "custom.cellOptions", "value": { "type": "color-background" } }] }, { "matcher": { "id": "byName", "options": "Rules" }, "properties": [{ "id": "custom.cellOptions", "value": { "type": "color-background" } }] }]
},
"gridPos": { "h": 6, "w": 12, "x": 0, "y": 68 },
"id": 70,
"options": { "showHeader": true },
"title": "Tables & Chains",
"type": "table",
"targets": [{ "expr": "nftables_table_chains{instance=\"$instance\"}", "refId": "A", "instant": true, "format": "table" }],
"transformations": [{ "id": "organize", "options": { "excludeByName": { "Time": true, "__name__": true, "instance": true, "job": true }, "renameByName": { "table": "Table", "family": "Family", "Value": "Chains" } } }]
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": { "custom": { "align": "auto" }, "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [{ "color": "blue", "value": null }] } },
"overrides": [{ "matcher": { "id": "byName", "options": "Rules" }, "properties": [{ "id": "custom.cellOptions", "value": { "type": "color-background" } }] }]
},
"gridPos": { "h": 6, "w": 12, "x": 12, "y": 68 },
"id": 71,
"options": { "showHeader": true },
"title": "Chains & Rules",
"type": "table",
"targets": [{ "expr": "nftables_chain_rules{instance=\"$instance\"}", "refId": "A", "instant": true, "format": "table" }],
"transformations": [{ "id": "organize", "options": { "excludeByName": { "Time": true, "__name__": true, "instance": true, "job": true }, "renameByName": { "table": "Table", "family": "Family", "chain": "Chain", "Value": "Rules" } } }]
}
],
"schemaVersion": 39,
"tags": ["nftables", "firewall"],
"templating": {
"list": [
{ "current": {}, "includeAll": false, "multi": false, "name": "datasource", "query": "prometheus", "refresh": 1, "type": "datasource" },
{ "current": {}, "datasource": { "type": "prometheus", "uid": "${datasource}" }, "definition": "label_values(nftables_up, instance)", "includeAll": false, "multi": false, "name": "instance", "query": { "query": "label_values(nftables_up, instance)", "refId": "StandardVariableQuery" }, "refresh": 2, "sort": 1, "type": "query" }
]
},
"time": { "from": "now-6h", "to": "now" },
"timepicker": {},
"timezone": "",
"title": "nftables Firewall",
"uid": "nftables-firewall",
"refresh": "30s"
}
</code></pre>
<h2 id="quick-health-checks">Quick health checks</h2>
<p>These are the first commands I use to verify that the whole stack is telling the truth:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># Verify the ruleset actually contains counters
sudo nft list ruleset | grep -c "counter"
# Check that every counted rule has a comment
sudo nft list ruleset | grep "counter" | grep -v "comment"
# This should print nothing.
# Spot-check live rules with comments
sudo nft list ruleset | grep "comment" | head -20
# Confirm the exporter is running
systemctl status prometheus-nftables-exporter
# Check the metrics endpoint
curl -s http://localhost:9630/metrics | grep nftables_up
# Find the busiest drop rules
curl -s http://localhost:9630/metrics | grep 'action="drop"' | sort -t' ' -k2 -rn | head -5
</code></pre>
<p>And if the exporter looks wrong, inspect the raw nftables JSON directly:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">sudo nft -j list ruleset | jq .
</code></pre>
<h2 id="common-failure-modes">Common failure modes</h2>
<p>There are a few ways to make this look more broken than it is:</p>
<ul>
<li>Rules without <code>counter</code> will never show up as time series.</li>
<li>Rules without <code>comment</code> will show up, but they will be hard to distinguish in dashboards.</li>
<li>Handle numbers will change after ruleset reloads or rebuilds, which is exactly why the dashboard should key off comments instead.</li>
<li>A very large ruleset means more time series, because every counted rule becomes its own metric series.</li>
<li>The exporter needs enough privilege to run <code>nft list ruleset</code>; the NixOS module handles that, but it is worth remembering when debugging.</li>
</ul>
<p>The first two are the important ones. If a rule matters, give it a counter and a comment.</p>
<h2 id="the-transferable-idea">The transferable idea</h2>
<p>The exporter is useful, but the real pattern here is bigger than one exporter or one firewall.</p>
<p>When a system already has counters but poor names, add names close to the source and export those names as labels.</p>
<p>That is the whole trick.</p>
<p>In nftables, <code>comment</code> is the naming layer. Once every rule carries intent alongside behavior, Prometheus and Grafana can do the rest. Your firewall stops being a wall of anonymous handles and starts being an observable system you can actually operate.</p>
Hrafnsyn: Unified Air and Sea Tracking with Phoenix LiveView, PostGIS, and Nix2026-04-11T15:30:00+00:002026-04-11T15:30:00+00:00
Ian Johannesen
https://perlpimp.net/blog/hrafnsyn-unified-air-and-sea-tracking/<p>I like projects that are easy to explain in one sentence and then keep getting more interesting the closer you look.</p>
<p>Hrafnsyn starts with a very simple pitch: it puts aircraft and vessels on one living map. ADS-B on the plane side, AIS on the boat side, Phoenix LiveView in front, PostgreSQL/PostGIS underneath, and realtime updates the whole way through.</p>
<p>That is already a fun project. But the part that makes it satisfying to me is that it is not just a map widget or a repo full of half-connected experiments. It is a composed operational system. Collectors, a stable ingest boundary, normalized persistence, a useful UI, auth modes, monitoring, and Nix-native deployment all line up in one codebase.</p>
<p>And now it is not just local anymore. Hrafnsyn is deployed from a real NixOS configuration, consumed through my NUR package/module path, monitored as a real service, and using a packaged aircraft-enrichment dataset that materially improves the aircraft side. That changes the story from “interesting Phoenix side project” to “this is actually coherent end to end.”</p>
<h2 id="one-map-two-feed-types-one-ingest-boundary">One map, two feed types, one ingest boundary</h2>
<p>The UI headline is obvious: aircraft and vessels share one dashboard. But the architectural point is the seam behind it.</p>
<p>Hrafnsyn is intentionally split into three layers:</p>
<ol>
<li>source collection</li>
<li>normalized ingest and persistence</li>
<li>realtime presentation</li>
</ol>
<p>Each upstream feed gets its own long-lived collector process. Right now the contracts are deliberately boring:</p>
<ul>
<li>aircraft via <code>GET /data/aircraft.json</code></li>
<li>vessels via <code>GET /api/ships_array.json</code></li>
</ul>
<p>That is exactly what I want here. One process per source, isolated restart behavior, clear source identity for logs and metrics, and no special cases once the payload has been normalized.</p>
<p>The interesting part is that both collectors converge on a very small shared ingest path:</p>
<pre data-lang="elixir" class="language-elixir "><code class="language-elixir" data-lang="elixir">def ingest_batch(source, observations) do
touched_ids =
observations
|> Enum.reduce([], fn observation, acc ->
with {:ok, normalized} <- Observation.new(observation),
{:ok, track} <- upsert_track(source, normalized),
{:ok, _point} <- insert_track_point(track, source, normalized) do
[track.id | acc]
else
_ -> acc
end
end)
|> Enum.uniq()
if touched_ids != [] do
Phoenix.PubSub.broadcast(Hrafnsyn.PubSub, @topic, {:tracks_updated, touched_ids})
end
{:ok, touched_ids}
end
</code></pre>
<p>That is the core of the whole app. HTTP collectors use it. Future publishers can use it. The gRPC ingest surface can use it too. Once observations cross that boundary, the rest of the system does not need to care where they came from.</p>
<h2 id="the-merge-model-is-what-makes-it-useful">The merge model is what makes it useful</h2>
<p>The storage layout is where the “unified tracker” idea turns into something more than a single map layer.</p>
<p>Hrafnsyn keeps two related tables:</p>
<ul>
<li><code>tracks</code> for the latest merged state of a vehicle</li>
<li><code>track_points</code> for the append-only history</li>
</ul>
<p>The schema makes the merge rules explicit:</p>
<pre data-lang="elixir" class="language-elixir "><code class="language-elixir" data-lang="elixir">create unique_index(:tracks, [:vehicle_type, :identity])
create unique_index(:track_points, [:track_id, :source_id, :observed_at])
</code></pre>
<p>That means a plane is uniquely identified by its hex code and a vessel by its MMSI, scoped by vehicle type. If multiple sources report the same vehicle, they merge into one current track. But every source can still leave behind its own historical points.</p>
<p>That combination is the right one for this kind of system:</p>
<ul>
<li>the dashboard shows a clean current state</li>
<li>search works against a normalized identity surface</li>
<li>route history stays durable instead of evaporating with the last websocket message</li>
</ul>
<p>The LiveView side stays pleasantly direct because persistence and fan-out are already solved. When tracks update, Phoenix PubSub pushes the change and the dashboard refreshes the map and side panels without inventing a custom frontend state machine for everything.</p>
<h2 id="the-aircraft-side-got-meaningfully-better">The aircraft side got meaningfully better</h2>
<p>One of the newer pieces, and one of the most worthwhile ones, is the packaged static aircraft database.</p>
<p>Live ADS-B feeds are useful, but they are often incomplete. Registration might be missing. Aircraft type may be absent or inconsistent. Some of the details you actually want for a good operational dashboard are available, just not reliably from the live stream itself.</p>
<p>Hrafnsyn now supports a dump1090-derived NDJSON metadata artifact keyed by ICAO hex. It adds:</p>
<ul>
<li>registration overrides</li>
<li>aircraft type</li>
<li>type description</li>
<li>wake turbulence category</li>
</ul>
<p>The loader is intentionally simple:</p>
<pre data-lang="elixir" class="language-elixir "><code class="language-elixir" data-lang="elixir">@default_record %{
registration: nil,
aircraft_type: nil,
type_description: nil,
wake_turbulence_category: nil
}
</code></pre>
<p>And the enrichment precedence is exactly what you would hope:</p>
<pre data-lang="elixir" class="language-elixir "><code class="language-elixir" data-lang="elixir">%__MODULE__{
observation
| registration: observation.registration || static.registration || derived.registration,
aircraft_type: observation.aircraft_type || static.aircraft_type,
type_description:
observation.type_description || normalize_type_description(static.type_description),
wake_turbulence_category:
observation.wake_turbulence_category || static.wake_turbulence_category,
country: observation.country || derived.country
}
</code></pre>
<p>In other words:</p>
<ul>
<li>live feed values win first</li>
<li>packaged static metadata fills the gaps second</li>
<li>ICAO-derived fallbacks come last</li>
</ul>
<p>That is a strong model because it improves the plane side without pretending the static dataset is the source of truth for everything.</p>
<p>More importantly, this database is not a random side file I copy into place by hand. It is a first-class flake output:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">aircraftDb = pkgs.runCommand "hrafnsyn-aircraft-db" {
nativeBuildInputs = [ pkgs.python3 ];
} ''
mkdir -p "$out/share/hrafnsyn"
python ${./scripts/build_aircraft_db.py} \
${dump1090Src} \
"$out/share/hrafnsyn/aircraft-db.ndjson" \
--metadata-output "$out/share/hrafnsyn/aircraft-db-metadata.json" \
--source-revision ${dump1090Rev}
'';
packages."aircraft-db" = aircraftDb;
packages.aircraftDb = aircraftDb;
</code></pre>
<p>That makes a real difference. The app package and the enrichment package can move through the same Nix-native pipeline instead of one being declarative and the other being “well, remember to put this file on disk somewhere.”</p>
<h2 id="the-local-developer-loop-is-unusually-friendly">The local developer loop is unusually friendly</h2>
<p>I care a lot about whether a project feels easy to pick back up after not touching it for a week. Hrafnsyn does.</p>
<p>The shortest local path is:</p>
<pre data-lang="sh" class="language-sh "><code class="language-sh" data-lang="sh">nix develop
app
</code></pre>
<p>And <code>app</code> is not magic. It is just a small helper that does the right boring things in the right order:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">app = pkgs.writeShellScriptBin "app" ''
set -euo pipefail
pg-reset
pg-start
eval "$(pg-env)"
mix ecto.setup
exec mix phx.server
'';
</code></pre>
<p>There is still a manual path if I want it:</p>
<pre data-lang="sh" class="language-sh "><code class="language-sh" data-lang="sh">nix develop
pg-start
eval "$(pg-env)"
mix setup
mix phx.server
</code></pre>
<p>But the helper matters. The flake does not just build the release; it makes the local development loop pleasant too.</p>
<h2 id="two-clean-consumption-paths">Two clean consumption paths</h2>
<p>This is one of the reasons I think Hrafnsyn is more interesting than a normal “here is my Phoenix app” project. It is packaged to be consumed the same way I want to consume other software myself.</p>
<p>The first route is direct: add the project as a flake input and use its package or NixOS module directly.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs.hrafnsyn.url = "github:ijohanne/hrafnsyn";
outputs = { nixpkgs, hrafnsyn, ... }: {
nixosConfigurations.tracker = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
hrafnsyn.nixosModules.default
({ pkgs, ... }: {
services.hrafnsyn = {
enable = true;
package = hrafnsyn.packages.${pkgs.system}.default;
aircraftDbPackage = hrafnsyn.packages.${pkgs.system}.aircraftDb;
};
})
];
};
};
}
</code></pre>
<p>The second route is through <a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/nur-packages"><code>ijohanne/nur-packages</code></a>, which exposes both the application and the aircraft metadata package:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">legacyPackages = forPackSystems (system:
(import ./default.nix {
pkgs = import nixpkgs { inherit system; };
inherit sources;
}) // {
hrafnsyn = inputs.hrafnsyn.packages.${system}.default;
hrafnsyn-aircraft-db = inputs.hrafnsyn.packages.${system}.aircraftDb;
});
nixosModules = (import ./modules) // {
hrafnsyn = inputs.hrafnsyn.nixosModules.default;
};
</code></pre>
<p>That gives consumers both paths:</p>
<ul>
<li>use <code>inputs.hrafnsyn</code> directly</li>
<li>use the curated NUR path for packages and module imports</li>
</ul>
<p>I like that because it mirrors how I actually reuse personal infrastructure projects. Some are easier to consume directly. Some fit better through a curated package collection. Hrafnsyn does not force the choice.</p>
<h2 id="the-nixos-module-grew-into-a-real-deployment-story">The NixOS module grew into a real deployment story</h2>
<p>This is the biggest reason the project feels complete now.</p>
<p>The module is no longer just enough to say “yes, you could probably run this on NixOS.” It supports the actual pieces that make a deployment feel native:</p>
<ul>
<li>structured <code>database</code> settings with <code>host</code>, <code>name</code>, <code>user</code>, and optional <code>passwordFile</code></li>
<li><code>aircraftDbPackage</code> as the preferred way to inject the packaged metadata artifact</li>
<li><code>aircraftDbPath</code> as a raw-path escape hatch when needed</li>
<li>structured <code>sources</code> rendered into <code>HRAFNSYN_SOURCES_JSON</code></li>
<li>optional nginx helper</li>
<li>optional dedicated metrics port</li>
<li>readonly-public versus authenticated-private mode</li>
</ul>
<p>The runtime contract in <code>config/runtime.exs</code> matches that shape cleanly. For PostgreSQL, it accepts either a traditional <code>DATABASE_URL</code> or structured <code>DATABASE_HOST</code>, <code>DATABASE_NAME</code>, <code>DATABASE_USER</code>, and <code>DATABASE_PASSWORD</code>. For aircraft enrichment, it consumes <code>HRAFNSYN_AIRCRAFT_DB_PATH</code>. Nothing here feels bolted on.</p>
<p>On NixOS, the structured local PostgreSQL path is especially nice because the module can create the database and user automatically, then enable <code>citext</code>, <code>pg_trgm</code>, and <code>postgis</code> before <code>hrafnsyn.service</code> starts. The nginx helper also does the right thing when the app binds wildcard addresses like <code>0.0.0.0</code>: nginx still proxies back over loopback instead of trying to hairpin through the public interface.</p>
<p>Here is the sort of representative configuration I mean:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
services.postgresql.enable = true;
services.hrafnsyn = {
enable = true;
package = nurPackages.hrafnsyn;
aircraftDbPackage = nurPackages.hrafnsyn-aircraft-db;
host = "tracks.example.com";
port = 4020;
metricsPort = 4022;
listenAddress = "0.0.0.0";
autoMigrate = true;
publicReadonly = false;
database = {
host = "/run/postgresql";
name = "hrafnsyn";
user = "hrafnsyn";
};
secretKeyBaseFile = /run/secrets/hrafnsyn-secret-key-base;
sources = [
{
id = "planes-main";
name = "Airplane SDR";
vehicleType = "plane";
adapter = "dump1090";
baseUrl = "http://collector-a.internal";
pollIntervalMs = 1000;
}
{
id = "boats-main";
name = "Boat SDR";
vehicleType = "vessel";
adapter = "ais_catcher";
baseUrl = "http://collector-b.internal:8100";
pollIntervalMs = 2500;
}
];
};
}
</code></pre>
<p>That is not hypothetical anymore. I am running a deployment in this style now, via the NUR module path, with managed secrets, local PostgreSQL socket auth, the packaged aircraft DB wired in declaratively, and the dashboard in authenticated/private mode rather than public-readonly mode.</p>
<p>I am being intentionally vague about hostnames, addresses, and secret wiring, because there is no value in dumping private infrastructure details into a blog post. But the important part is that the deployment is real enough to prove the package and module interfaces are not decorative.</p>
<h2 id="public-display-mode-and-private-ops-mode-both-make-sense">Public display mode and private ops mode both make sense</h2>
<p>I also like the auth split here because it matches the actual use cases.</p>
<p><code>publicReadonly = true</code> is good for a publicly visible tracking screen or a household dashboard where anonymous viewers can look but not touch anything.</p>
<p><code>publicReadonly = false</code> flips the posture. Anonymous web users are redirected to login, tracking gRPC calls require JWTs, and ingestion can require authenticated admin tokens. That is the mode I am using for the live deployment right now.</p>
<p>This is a nice example of a small configuration flag doing real design work. It is not “auth later.” The public and private operating modes are part of the application model.</p>
<h2 id="monitoring-is-part-of-the-project-not-an-afterthought">Monitoring is part of the project, not an afterthought</h2>
<p>Hrafnsyn exposes <code>/metrics</code> on the main endpoint and can split metrics onto a dedicated port with <code>metricsPort</code>. The module can also append a Prometheus scrape job and provision the bundled Grafana dashboard.</p>
<p>That matters because it pushes the project over the line from neat UI into real service. I can deploy it, scrape it, and drop the shipped dashboard into Grafana instead of promising myself I will “add observability later.”</p>
<p>Again, that is not theoretical for me anymore. The live deployment is scraped by Prometheus on its dedicated metrics port, and the bundled Hrafnsyn Grafana dashboard is provisioned from the repo.</p>
<h2 id="why-i-think-this-project-is-satisfying">Why I think this project is satisfying</h2>
<p>The fun part of Hrafnsyn is not just that it shows planes and boats together. It is that several small systems decisions reinforce each other cleanly:</p>
<ul>
<li>collectors stay simple</li>
<li>ingest stays centralized</li>
<li>storage keeps both current state and history</li>
<li>LiveView gets realtime updates without frontend contortions</li>
<li>packaging covers both the app and the aircraft metadata artifact</li>
<li>deployment works directly or through NUR</li>
<li>auth can be public-readonly or intentionally private</li>
<li>monitoring is already part of the story</li>
</ul>
<p>That is the kind of software project I enjoy most. Not a giant platform. Not a throwaway demo. Just one coherent codebase where UI, ingest, storage, enrichment, auth, monitoring, and deployment all belong to the same idea.</p>
<p>And now that it is deployed for real, I find it much easier to argue that Hrafnsyn is interesting for the right reasons. It is not simply “look, I put a map on the web.” It is a practical Phoenix and Nix system with a clean boundary design, a nice operational shape, and just enough ambition to stay fun.</p>
Building a Stratum 1 NTP Server with GPS/PPS on Raspberry Pi2026-04-10T12:00:00+00:002026-04-10T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/stratum-1-ntp-server-gps-pps-raspberry-pi/<p>If you already run a homelab, eventually you start wanting your time to come from inside the house instead of “some pool servers on the internet, probably.” Not because pool.ntp.org is bad. It is excellent. But because once you have Prometheus, Grafana, UPS-backed hosts, and a mild infrastructure problem, the next logical step is apparently “build a stratum 1 clock.”</p>
<p>The good news is that this is entirely doable with a Raspberry Pi 4, a decent GNSS receiver, and chrony. The less good news is that the obvious path through <code>ntpd</code> is a swamp of half-working refclock drivers, shared memory weirdness, and one especially annoying circular dependency. I tried the obvious thing so you do not have to.</p>
<p>This setup ended up here:</p>
<ul>
<li>Raspberry Pi 4 running Arch Linux ARM</li>
<li>u-blox ZED-F9P GNSS module for serial time + PPS</li>
<li><code>gpsd</code> for GPS data and coarse time</li>
<li><code>chrony</code> for actual clock discipline</li>
<li><code>chrony_exporter</code>, Prometheus, and Grafana for visibility</li>
</ul>
<p>The end result is a LAN-served stratum 1 NTP server with roughly 150-400ns offset from UTC once it has settled. Which is a mildly absurd amount of precision for a thing sitting next to a router, but that is part of the charm.</p>
<h2 id="hardware">Hardware</h2>
<p>The hardware is simple:</p>
<ul>
<li>Raspberry Pi 4 running 64-bit Arch Linux ARM</li>
<li>u-blox ZED-F9P multi-constellation GNSS receiver</li>
<li>USB serial link, with the receiver showing up as <code>/dev/ttyACM0</code></li>
<li>PPS wiring exposed as <code>/dev/pps0</code></li>
</ul>
<p>I symlinked the serial device to <code>/dev/gps0</code> with udev because device names that change under you are funny exactly once.</p>
<p>The software stack for the working setup:</p>
<ul>
<li><code>gpsd 3.26.1</code></li>
<li><code>chrony 4.8</code></li>
<li><code>chrony_exporter 0.11.0</code></li>
<li>Prometheus on a separate monitoring host</li>
<li>Grafana on that same monitoring host</li>
</ul>
<h2 id="why-gps-time-is-two-signals-not-one">Why GPS Time Is Two Signals, Not One</h2>
<p>The important conceptual piece is that GPS timekeeping is really two separate signals:</p>
<ol>
<li>NMEA or UBX messages tell you which second it is.</li>
<li>PPS tells you exactly when that second starts.</li>
</ol>
<p>Those are not the same job.</p>
<p>The serial GPS stream gives you timestamps, but they arrive through USB, the kernel, buffering, and userspace parsing. That makes them accurate in the “yes, it is definitely this second” sense, but jittery by tens of milliseconds.</p>
<p>PPS, short for pulse per second, is a hardware edge emitted exactly on the second boundary. The kernel timestamps that edge with far better precision than userspace can manage. PPS is what gets you from “roughly correct” to “this box is now taking nanoseconds personally.”</p>
<p>The data path looks like this:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">GNSS receiver --USB serial--> gpsd --SHM NTP0--> chrony
GNSS receiver --PPS GPIO-----> kernel PPS API (/dev/pps0) --> chrony
chrony --> LAN clients
chrony_exporter --> Prometheus --> Grafana
</code></pre>
<p><code>gpsd</code> also writes PPS into SHM, but that turns out to be the wrong place to consume it from. More on that in the <code>ntpd</code> war story section.</p>
<h3 id="shm-segments-from-gpsd">SHM segments from gpsd</h3>
<p>If you inspect shared memory, gpsd creates segments for NTP consumers:</p>
<table><thead><tr><th>Key</th><th>Segment</th><th>Permissions</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>0x4e545030</code></td><td><code>NTP0</code></td><td><code>600</code></td><td>Coarse GPS time</td></tr>
<tr><td><code>0x4e545031</code></td><td><code>NTP1</code></td><td><code>600</code></td><td>PPS for privileged consumers</td></tr>
<tr><td><code>0x4e545032</code></td><td><code>NTP2</code></td><td><code>666</code></td><td>PPS for unprivileged consumers</td></tr>
</tbody></table>
<p>That permission split matters. <code>NTP0</code> is root-only. If your time daemon runs as an unprivileged user, life gets interesting in the bad way.</p>
<h2 id="getting-gpsd-to-behave">Getting gpsd To Behave</h2>
<p>On a distro that uses <code>/etc/default/gpsd</code>, the config is straightforward:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">START_DAEMON="true"
GPSD_OPTIONS="-n"
DEVICES="/dev/gps0 /dev/pps0"
USBAUTO="true"
</code></pre>
<p>Two bits matter most:</p>
<ul>
<li><code>-n</code> makes <code>gpsd</code> start polling immediately instead of waiting for a client.</li>
<li>You must list both the serial device and the PPS device.</li>
</ul>
<p>Once that is in place, the debugging commands you actually care about are:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># Is gpsd alive?
systemctl status gpsd
ps aux | grep gpsd
# Do you have a fix?
cgps -s
# Raw interactive view
gpsmon
# Stream raw JSON
gpspipe -w | head -20
# Specifically look for PPS events
gpspipe -w | grep PPS
# See SHM segments
ipcs -m
</code></pre>
<h3 id="what-healthy-gpsd-output-looks-like">What healthy gpsd output looks like</h3>
<p>A normal TPV message looks something like this:</p>
<pre data-lang="json" class="language-json "><code class="language-json" data-lang="json">{"class":"TPV","device":"/dev/gps0","status":2,"mode":3,
"time":"2026-04-02T11:23:02.000Z","lat":36.42399,"lon":-5.15240,...}
</code></pre>
<p>The fields worth caring about:</p>
<ul>
<li><code>mode=3</code> means you have a 3D fix.</li>
<li><code>status=2</code> means DGPS-quality fix.</li>
<li><code>time</code> is the coarse second count chrony will use as an anchor.</li>
</ul>
<p>A PPS message looks like this:</p>
<pre data-lang="json" class="language-json "><code class="language-json" data-lang="json">{"class":"PPS","device":"/dev/pps0","real_sec":1775128983,"real_nsec":0,
"clock_sec":1775128982,"clock_nsec":999955940,"precision":-20,"shm":"NTP2"}
</code></pre>
<p>The important part here is the split between the true GPS second and the system clock’s view of when that edge happened. If the machine clock is off, those numbers will disagree. That becomes important later.</p>
<h3 id="common-gpsd-potholes">Common gpsd potholes</h3>
<ul>
<li>No serial data at all: on Raspberry Pi, make sure the login shell is not squatting on the UART and verify you are reading the right device.</li>
<li><code>gpsmon</code> looks empty: often a display quirk, not a gpsd failure. If <code>cgps -s</code> works, you are probably fine.</li>
<li>First fix takes forever: a cold receiver with no almanac can take a while. Outdoors with clear sky, the ZED-F9P is usually much faster.</li>
<li>SHM permissions look wrong: they are wrong, just usually by design.</li>
</ul>
<h2 id="pps-the-part-that-actually-makes-this-precise">PPS: The Part That Actually Makes This Precise</h2>
<p>Before touching NTP at all, verify that PPS is really arriving:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">ls -la /dev/pps0
sudo ppstest /dev/pps0
</code></pre>
<p>Healthy <code>ppstest</code> output looks like this:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">source 0 - assert 1775128192.913287329, sequence: 12896874 - clear 0.000000000, sequence: 0
source 0 - assert 1775128193.913286383, sequence: 12896875 - clear 0.000000000, sequence: 0
</code></pre>
<p>What you want:</p>
<ul>
<li>timestamps exactly one second apart</li>
<li>steadily increasing sequence numbers</li>
<li>very little jitter between pulses</li>
</ul>
<p>What you do not want is no output at all, because that means your wiring, GPIO config, or receiver PPS output is lying to you.</p>
<p>If <code>/dev/pps0</code> does not exist, stop there and fix the kernel/device-tree side first. No NTP daemon can save you from a missing pulse input.</p>
<h2 id="the-ntpd-dead-end">The ntpd Dead End</h2>
<p>Naturally, I started with <code>ntpd</code>, because that is what a lot of old GPS/NTP docs still point at. This was a mistake.</p>
<p>The first attempt was the classic shared-memory plus PPS refclock setup:</p>
<pre data-lang="conf" class="language-conf "><code class="language-conf" data-lang="conf">server 127.127.28.0 minpoll 4 maxpoll 4 prefer
fudge 127.127.28.0 refid GPS
server 127.127.22.0 minpoll 4 maxpoll 4
fudge 127.127.22.0 refid PPS
</code></pre>
<p>That lasted right up until the logs said:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">refclock_newpeer: clock type 22 invalid
</code></pre>
<p>So the ATOM driver was not even compiled into the Arch Linux ARM package. Excellent start. The driver that is supposed to read PPS directly from the kernel was simply unavailable.</p>
<p>I then tried the GPSD JSON driver. Also bad. It connected, then mostly sulked, and eventually produced the usual <code>clk_no_reply</code> style misery.</p>
<p>That left SHM.</p>
<p>SHM kind of works, in the same way a chair with three legs kind of works if you are very still. The problem is this:</p>
<ol>
<li><code>gpsd</code> timestamps PPS events for SHM using the system clock.</li>
<li>Before the time daemon has disciplined the system clock, that clock is wrong.</li>
<li>Therefore PPS-via-SHM is wrong by exactly the amount you need PPS to fix.</li>
</ol>
<p>In my case the PPS SHM source sat around 57ms off and would not converge. That is the circular dependency:</p>
<ul>
<li>you need accurate PPS to fix the clock</li>
<li>but PPS through SHM depends on the clock already being accurate</li>
</ul>
<p>Then there was the permissions issue layered on top:</p>
<ul>
<li><code>NTP0</code> is root-only</li>
<li><code>ntpd</code> runs as <code>ntp</code></li>
<li>the daemon can attach to the segment, but that does not mean it can read it reliably enough to build a clean solution</li>
</ul>
<p>At that point <code>ntpd</code> had consumed enough of my evening and earned nothing further.</p>
<h2 id="chrony-saves-the-day">chrony Saves The Day</h2>
<p>The reason chrony works is very simple: it does not insist on walking through the swamp.</p>
<p>This line is the whole difference:</p>
<pre data-lang="conf" class="language-conf "><code class="language-conf" data-lang="conf">refclock PPS /dev/pps0 refid PPS lock GPS
</code></pre>
<p>chrony reads PPS directly from the kernel PPS API through <code>/dev/pps0</code>. No gpsd SHM timestamping, no circular dependency, no drama. The kernel timestamps the interrupt edge. That is the thing you wanted all along.</p>
<p>The full working <code>chrony.conf</code> looked like this:</p>
<pre data-lang="conf" class="language-conf "><code class="language-conf" data-lang="conf">refclock SHM 0 refid GPS precision 1e-1 offset 0.0 delay 0.2
refclock PPS /dev/pps0 refid PPS lock GPS
server 0.arch.pool.ntp.org iburst
server 1.arch.pool.ntp.org iburst
server 2.arch.pool.ntp.org iburst
server 3.arch.pool.ntp.org iburst
driftfile /var/lib/chrony/drift
allow 10.255.0.0/16
makestep 1 3
rtcsync
</code></pre>
<p>What each piece is doing:</p>
<ul>
<li><code>refclock SHM 0</code> reads coarse GPS time from gpsd. That is the “which second is this?” source.</li>
<li><code>refclock PPS /dev/pps0 lock GPS</code> reads the precise edge from the kernel and ties it to the GPS second numbering.</li>
<li>pool servers are there as sanity and fallback, not as the primary source.</li>
<li><code>allow</code> lets LAN clients use the Pi as their NTP server.</li>
<li><code>makestep 1 3</code> lets chrony fix a badly wrong clock quickly at startup instead of spending half a lifetime slewing it back into reality.</li>
</ul>
<h3 id="the-two-chronyc-commands-that-matter">The two chronyc commands that matter</h3>
<p>First:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">chronyc sources -v
</code></pre>
<p>Healthy output:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
#- GPS 0 4 377 24 +54ms[ +54ms] +/- 200ms
#* PPS 0 4 377 22 -153ns[-1337ns] +/- 334ns
^- tock.espanix.net 1 6 77 26 -554us[ -555us] +/- 6323us
</code></pre>
<p>This is exactly what you want to see:</p>
<ul>
<li><code>GPS</code> is noisy by tens of milliseconds, because serial time is supposed to be noisy.</li>
<li><code>PPS</code> is sitting down in the nanoseconds, because that is the actual precision source.</li>
<li>internet peers are reachable but clearly not what the clock is following.</li>
</ul>
<p>Then:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">chronyc tracking
</code></pre>
<p>Example output:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">Reference ID : 50505300 (PPS)
Stratum : 1
System time : 0.000000313 seconds slow of NTP time
Last offset : -0.000000397 seconds
RMS offset : 0.000004104 seconds
Frequency : 5.823 ppm fast
</code></pre>
<p>The headline numbers:</p>
<ul>
<li><code>Reference ID : PPS</code> means the Pi is locked to the pulse source, not a remote server.</li>
<li><code>Stratum : 1</code> means it is directly attached to a reference clock.</li>
<li><code>System time</code> in the sub-microsecond range means the whole thing is doing what you built it to do.</li>
</ul>
<p>The 5-6ppm oscillator drift on a Raspberry Pi is also completely normal. It is not a fancy oven-controlled oscillator. It is a tiny board that mostly wants to run Linux and warm up slightly.</p>
<h3 id="if-chrony-looks-wrong">If chrony looks wrong</h3>
<p>The usual failure modes are mercifully sane:</p>
<ul>
<li>PPS shows <code>?</code>: chrony has no coarse time source to lock PPS to, so go fix gpsd/SHM first.</li>
<li>GPS offset is 50-100ms: that is normal, not a bug.</li>
<li>Large startup step: normal if the box booted with stale RTC state or no RTC at all. That is what <code>makestep</code> is for.</li>
</ul>
<h2 id="exporting-chrony-to-prometheus">Exporting chrony To Prometheus</h2>
<p>Once the clock itself works, you want to stop SSHing in every time you wonder whether it still works.</p>
<p>I used <code>chrony_exporter 0.11.0</code>:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">curl -LO https://github.com/SuperQ/chrony_exporter/releases/download/v0.11.0/chrony_exporter-0.11.0.linux-arm64.tar.gz
tar xzf chrony_exporter-0.11.0.linux-arm64.tar.gz
sudo cp chrony_exporter-0.11.0.linux-arm64/chrony_exporter /usr/local/bin/
</code></pre>
<p>The systemd unit:</p>
<pre data-lang="ini" class="language-ini "><code class="language-ini" data-lang="ini">[Unit]
Description=Chrony Exporter
After=chronyd.service
[Service]
User=root
ExecStart=/usr/local/bin/chrony_exporter \
--collector.sources \
--collector.serverstats \
--collector.tracking \
--chrony.address=unix:///run/chrony/chronyd.sock \
--collector.chmod-socket
Restart=always
[Install]
WantedBy=multi-user.target
</code></pre>
<h3 id="why-the-unix-socket-matters">Why the Unix socket matters</h3>
<p>The exporter can talk to chronyd in two ways:</p>
<ul>
<li>UDP command port (<code>127.0.0.1:323</code>)</li>
<li>Unix socket (<code>/run/chrony/chronyd.sock</code>)</li>
</ul>
<p>UDP sounds simpler. It is not simpler.</p>
<p>With UDP, the tracking collector may work, but <code>serverstats</code> tends to run into authentication and <code>UNAUTH</code> problems. <code>cmdallow</code> is not enough to make the more privileged commands happy. You end up staring at <code>chrony_up 0</code> and wondering why a daemon on localhost is acting like you are an intruder.</p>
<p>The Unix socket is the correct answer:</p>
<ul>
<li>no UDP command ACL weirdness</li>
<li>all collectors work</li>
<li>permissions are handled explicitly</li>
</ul>
<p>If the exporter misbehaves, these are the checks worth running:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">systemctl status chrony-exporter
journalctl -u chrony-exporter -n 30 --no-pager
curl -s http://localhost:9123/metrics | grep chrony_up
curl -s http://localhost:9123/metrics | grep ^chrony_
</code></pre>
<p>If you need more detail, add <code>--log.level=debug</code> and look for <code>UNAUTH</code> or socket-permission complaints.</p>
<h3 id="prometheus-scrape-config">Prometheus scrape config</h3>
<p>The scrape side is unremarkable, which is how monitoring should be:</p>
<pre data-lang="yaml" class="language-yaml "><code class="language-yaml" data-lang="yaml">scrape_configs:
- job_name: chrony
static_configs:
- targets: ['chronos:9123']
labels:
instance: chronos
</code></pre>
<h3 id="metrics-that-actually-matter">Metrics that actually matter</h3>
<p>Tracking metrics tell you whether the machine itself is healthy:</p>
<table><thead><tr><th>Metric</th><th>Meaning</th><th>Good value</th></tr></thead><tbody>
<tr><td><code>chrony_up</code></td><td>Exporter can talk to chronyd</td><td><code>1</code></td></tr>
<tr><td><code>chrony_tracking_stratum</code></td><td>NTP stratum</td><td><code>1</code></td></tr>
<tr><td><code>chrony_tracking_system_time_seconds</code></td><td>Current system offset</td><td>under <code>1us</code></td></tr>
<tr><td><code>chrony_tracking_last_offset_seconds</code></td><td>Latest measurement</td><td>under <code>1us</code></td></tr>
<tr><td><code>chrony_tracking_rms_offset_seconds</code></td><td>Long-term average offset</td><td>under <code>10us</code></td></tr>
<tr><td><code>chrony_tracking_frequency_ppms</code></td><td>Local oscillator drift</td><td>stable and boring</td></tr>
<tr><td><code>chrony_tracking_skew_ppms</code></td><td>Error bound on frequency</td><td>low</td></tr>
</tbody></table>
<p>Source metrics tell you whether individual peers and refclocks still look sane:</p>
<table><thead><tr><th>Metric</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>chrony_sources_last_sample_offset_seconds</code></td><td>Offset per source</td></tr>
<tr><td><code>chrony_sources_last_sample_error_margin_seconds</code></td><td>Error margin per source</td></tr>
<tr><td><code>chrony_sources_reachability_ratio</code></td><td>Poll success ratio</td></tr>
<tr><td><code>chrony_sources_stratum</code></td><td>Source stratum</td></tr>
<tr><td><code>chrony_sources_state_info</code></td><td><code>sync</code>, <code>candidate</code>, <code>outlier</code>, <code>unreachable</code></td></tr>
<tr><td><code>chrony_sources_polling_interval_seconds</code></td><td>Poll interval</td></tr>
</tbody></table>
<p>The nice thing about this combination is that it tells you both whether the local clock is good and why it is good.</p>
<h2 id="grafana-dashboard">Grafana Dashboard</h2>
<p>The dashboard I ended up with has three sections:</p>
<ul>
<li>Status row for “is this box healthy right now?”</li>
<li>Tracking row for offset, drift, and dispersion over time</li>
<li>Sources row for per-source behavior, including a table that shows which source is actually syncing</li>
</ul>
<p>The useful mental model is:</p>
<ul>
<li><code>GPS</code> should be noisy and coarse</li>
<li><code>PPS</code> should be boring and near zero</li>
<li>remote internet peers should be the backup singers, not the lead vocalist</li>
</ul>
<p>Here is the full dashboard JSON. Paste it into Grafana’s import dialog or provision it from a file.</p>
<pre data-lang="json" class="language-json "><code class="language-json" data-lang="json">{
"annotations": {
"list": []
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"links": [],
"panels": [
{
"collapsed": false,
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
"id": 1,
"title": "Status",
"type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"mappings": [
{ "options": { "0": { "color": "red", "text": "DOWN" }, "1": { "color": "green", "text": "UP" } }, "type": "value" }
],
"thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] }
},
"overrides": []
},
"gridPos": { "h": 4, "w": 3, "x": 0, "y": 1 },
"id": 2,
"options": { "colorMode": "background", "graphMode": "none", "justifyMode": "center", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "value" },
"title": "Chrony Status",
"targets": [{ "expr": "chrony_up{instance=\"chronos\"}", "refId": "A" }],
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 2 }, { "color": "red", "value": 3 }] }
},
"overrides": []
},
"gridPos": { "h": 4, "w": 3, "x": 3, "y": 1 },
"id": 3,
"options": { "colorMode": "value", "graphMode": "none", "justifyMode": "center", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "value" },
"title": "Stratum",
"targets": [{ "expr": "chrony_tracking_stratum{instance=\"chronos\"}", "refId": "A" }],
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"unit": "s",
"decimals": 3,
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.001 }, { "color": "red", "value": 0.01 }] }
},
"overrides": []
},
"gridPos": { "h": 4, "w": 4, "x": 6, "y": 1 },
"id": 4,
"options": { "colorMode": "value", "graphMode": "area", "justifyMode": "center", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "value" },
"title": "System Time Offset",
"targets": [{ "expr": "chrony_tracking_system_time_seconds{instance=\"chronos\"}", "refId": "A" }],
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"unit": "s",
"decimals": 6,
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.001 }, { "color": "red", "value": 0.01 }] }
},
"overrides": []
},
"gridPos": { "h": 4, "w": 4, "x": 10, "y": 1 },
"id": 5,
"options": { "colorMode": "value", "graphMode": "area", "justifyMode": "center", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "value" },
"title": "Last Offset",
"targets": [{ "expr": "chrony_tracking_last_offset_seconds{instance=\"chronos\"}", "refId": "A" }],
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"unit": "s",
"decimals": 6,
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.001 }, { "color": "red", "value": 0.01 }] }
},
"overrides": []
},
"gridPos": { "h": 4, "w": 4, "x": 14, "y": 1 },
"id": 6,
"options": { "colorMode": "value", "graphMode": "area", "justifyMode": "center", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "value" },
"title": "RMS Offset",
"targets": [{ "expr": "chrony_tracking_rms_offset_seconds{instance=\"chronos\"}", "refId": "A" }],
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] }
},
"overrides": []
},
"gridPos": { "h": 4, "w": 3, "x": 18, "y": 1 },
"id": 7,
"options": { "colorMode": "value", "graphMode": "none", "justifyMode": "center", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "value" },
"title": "Reference",
"targets": [{ "expr": "chrony_tracking_info{instance=\"chronos\"}", "legendFormat": "{{tracking_name}}", "refId": "A" }],
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"unit": "ppm",
"decimals": 3,
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 10 }, { "color": "red", "value": 50 }] }
},
"overrides": []
},
"gridPos": { "h": 4, "w": 3, "x": 21, "y": 1 },
"id": 8,
"options": { "colorMode": "value", "graphMode": "area", "justifyMode": "center", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "value" },
"title": "Frequency Error",
"targets": [{ "expr": "chrony_tracking_frequency_ppms{instance=\"chronos\"}", "refId": "A" }],
"type": "stat"
},
{
"collapsed": false,
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 },
"id": 10,
"title": "Tracking",
"type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": { "axisBorderShow": false, "axisCenteredZero": true, "axisLabel": "", "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "pointSize": 5, "showPoints": "never", "spanNulls": false },
"unit": "s",
"decimals": 6
},
"overrides": []
},
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 6 },
"id": 11,
"options": { "legend": { "calcs": ["mean", "lastNotNull", "min", "max"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "none" } },
"title": "System Clock Offset",
"targets": [
{ "expr": "chrony_tracking_system_time_seconds{instance=\"chronos\"}", "legendFormat": "system time", "refId": "A" },
{ "expr": "chrony_tracking_last_offset_seconds{instance=\"chronos\"}", "legendFormat": "last offset", "refId": "B" },
{ "expr": "chrony_tracking_rms_offset_seconds{instance=\"chronos\"}", "legendFormat": "RMS offset", "refId": "C" }
],
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisLabel": "ppm", "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "pointSize": 5, "showPoints": "never", "spanNulls": false },
"unit": "ppm",
"decimals": 3
},
"overrides": []
},
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 6 },
"id": 12,
"options": { "legend": { "calcs": ["mean", "lastNotNull", "min", "max"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "none" } },
"title": "Frequency",
"targets": [
{ "expr": "chrony_tracking_frequency_ppms{instance=\"chronos\"}", "legendFormat": "frequency error", "refId": "A" },
{ "expr": "chrony_tracking_residual_frequency_ppms{instance=\"chronos\"}", "legendFormat": "residual frequency", "refId": "B" },
{ "expr": "chrony_tracking_skew_ppms{instance=\"chronos\"}", "legendFormat": "skew (error bound)", "refId": "C" }
],
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisLabel": "", "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "pointSize": 5, "showPoints": "never", "spanNulls": false },
"unit": "s",
"decimals": 6
},
"overrides": []
},
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 14 },
"id": 13,
"options": { "legend": { "calcs": ["mean", "lastNotNull", "min", "max"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "none" } },
"title": "Root Delay & Dispersion",
"targets": [
{ "expr": "chrony_tracking_root_delay_seconds{instance=\"chronos\"}", "legendFormat": "root delay", "refId": "A" },
{ "expr": "chrony_tracking_root_dispersion_seconds{instance=\"chronos\"}", "legendFormat": "root dispersion", "refId": "B" }
],
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisLabel": "", "drawStyle": "line", "fillOpacity": 0, "lineWidth": 1, "pointSize": 5, "showPoints": "never", "spanNulls": false },
"unit": "s"
},
"overrides": []
},
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 14 },
"id": 14,
"options": { "legend": { "calcs": ["lastNotNull"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "none" } },
"title": "Update Interval",
"targets": [
{ "expr": "chrony_tracking_update_interval_seconds{instance=\"chronos\"}", "legendFormat": "update interval", "refId": "A" }
],
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 22 },
"id": 20,
"title": "Sources",
"type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": { "axisBorderShow": false, "axisCenteredZero": true, "axisLabel": "", "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "pointSize": 5, "showPoints": "never", "spanNulls": false },
"unit": "s",
"decimals": 6
},
"overrides": []
},
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 23 },
"id": 21,
"options": { "legend": { "calcs": ["mean", "lastNotNull"], "displayMode": "table", "placement": "bottom", "sortBy": "Last *", "sortDesc": false }, "tooltip": { "mode": "multi", "sort": "none" } },
"title": "Source Offsets",
"targets": [
{ "expr": "chrony_sources_last_sample_offset_seconds{instance=\"chronos\"}", "legendFormat": "{{source_name}}", "refId": "A" }
],
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisLabel": "", "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "pointSize": 5, "showPoints": "never", "spanNulls": false },
"unit": "s",
"decimals": 6
},
"overrides": []
},
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 23 },
"id": 22,
"options": { "legend": { "calcs": ["mean", "lastNotNull"], "displayMode": "table", "placement": "bottom", "sortBy": "Last *", "sortDesc": false }, "tooltip": { "mode": "multi", "sort": "none" } },
"title": "Source Error Margin",
"targets": [
{ "expr": "chrony_sources_last_sample_error_margin_seconds{instance=\"chronos\"}", "legendFormat": "{{source_name}}", "refId": "A" }
],
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisLabel": "", "drawStyle": "line", "fillOpacity": 30, "lineWidth": 1, "pointSize": 5, "showPoints": "never", "spanNulls": false },
"min": 0,
"max": 1,
"unit": "percentunit",
"decimals": 0
},
"overrides": []
},
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 32 },
"id": 23,
"options": { "legend": { "calcs": ["lastNotNull"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "none" } },
"title": "Source Reachability",
"targets": [
{ "expr": "chrony_sources_reachability_ratio{instance=\"chronos\"}", "legendFormat": "{{source_name}}", "refId": "A" }
],
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisLabel": "", "drawStyle": "line", "fillOpacity": 0, "lineWidth": 1, "pointSize": 5, "showPoints": "never", "spanNulls": false },
"unit": "s"
},
"overrides": []
},
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 32 },
"id": 24,
"options": { "legend": { "calcs": ["lastNotNull"], "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "none" } },
"title": "Source Polling Interval",
"targets": [
{ "expr": "chrony_sources_polling_interval_seconds{instance=\"chronos\"}", "legendFormat": "{{source_name}}", "refId": "A" }
],
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"cellOptions": { "type": "auto" },
"inspect": false
},
"mappings": [],
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] }
},
"overrides": [
{
"matcher": { "id": "byName", "options": "State" },
"properties": [
{
"id": "mappings",
"value": [
{ "options": { "outlier": { "color": "orange", "text": "outlier" }, "sync": { "color": "green", "text": "sync" }, "candidate": { "color": "blue", "text": "candidate" }, "unreachable": { "color": "red", "text": "unreachable" } }, "type": "value" }
]
},
{ "id": "custom.cellOptions", "value": { "mode": "basic", "type": "color-background" } }
]
},
{
"matcher": { "id": "byName", "options": "Stratum" },
"properties": [{ "id": "custom.width", "value": 80 }]
},
{
"matcher": { "id": "byName", "options": "Mode" },
"properties": [{ "id": "custom.width", "value": 130 }]
},
{
"matcher": { "id": "byName", "options": "Reachability" },
"properties": [
{ "id": "unit", "value": "percentunit" },
{ "id": "decimals", "value": 0 },
{ "id": "custom.cellOptions", "value": { "mode": "basic", "type": "color-background" } },
{ "id": "thresholds", "value": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "yellow", "value": 0.5 }, { "color": "green", "value": 0.875 }] } }
]
},
{
"matcher": { "id": "byName", "options": "Offset" },
"properties": [{ "id": "unit", "value": "s" }, { "id": "decimals", "value": 6 }]
},
{
"matcher": { "id": "byName", "options": "Error Margin" },
"properties": [{ "id": "unit", "value": "s" }, { "id": "decimals", "value": 6 }]
}
]
},
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 40 },
"id": 25,
"options": { "showHeader": true, "footer": { "show": false } },
"title": "Source Details",
"targets": [
{ "expr": "chrony_sources_state_info{instance=\"chronos\"}", "format": "table", "instant": true, "refId": "A" },
{ "expr": "chrony_sources_stratum{instance=\"chronos\"}", "format": "table", "instant": true, "refId": "B" },
{ "expr": "chrony_sources_reachability_ratio{instance=\"chronos\"}", "format": "table", "instant": true, "refId": "C" },
{ "expr": "chrony_sources_last_sample_offset_seconds{instance=\"chronos\"}", "format": "table", "instant": true, "refId": "D" },
{ "expr": "chrony_sources_last_sample_error_margin_seconds{instance=\"chronos\"}", "format": "table", "instant": true, "refId": "E" }
],
"transformations": [
{
"id": "joinByField",
"options": { "byField": "source_name", "mode": "outer" }
},
{
"id": "organize",
"options": {
"excludeByName": {
"Time": true, "Time 1": true, "Time 2": true, "Time 3": true, "Time 4": true, "Time 5": true,
"__name__": true, "__name__ 1": true, "__name__ 2": true, "__name__ 3": true, "__name__ 4": true,
"instance": true, "instance 1": true, "instance 2": true, "instance 3": true, "instance 4": true,
"job": true, "job 1": true, "job 2": true, "job 3": true, "job 4": true,
"source_address": true, "source_address 1": true, "source_address 2": true, "source_address 3": true, "source_address 4": true,
"source_name 1": true, "source_name 2": true, "source_name 3": true, "source_name 4": true,
"tracking_address": true, "tracking_name": true, "tracking_refid": true
},
"renameByName": {
"source_name": "Source",
"source_mode": "Mode",
"source_state": "State",
"Value #A": "",
"Value #B": "Stratum",
"Value #C": "Reachability",
"Value #D": "Offset",
"Value #E": "Error Margin"
}
}
},
{
"id": "filterByValue",
"options": {
"filters": [{ "fieldName": "Source", "config": { "id": "isNotNull" } }],
"match": "all",
"type": "include"
}
}
],
"type": "table"
}
],
"schemaVersion": 39,
"templating": {
"list": [
{
"current": { "selected": false, "text": "Prometheus", "value": "PBFA97CFB590B2093" },
"hide": 0,
"includeAll": false,
"name": "datasource",
"options": [],
"query": "prometheus",
"refresh": 1,
"type": "datasource"
}
]
},
"time": { "from": "now-6h", "to": "now" },
"timepicker": {},
"timezone": "",
"title": "Chrony / GPS Time Server",
"uid": "chrony-gps",
"version": 1
}
</code></pre>
<h2 id="quick-health-check">Quick Health Check</h2>
<p>When you want to verify the full stack without thinking too hard, these are enough:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># GPS serial data arriving?
gpspipe -w -n 3 | grep TPV
# PPS pulses arriving?
sudo ppstest /dev/pps0
# gpsd seeing PPS too?
gpspipe -w -n 3 | grep PPS
# chrony locked to PPS with tiny offset?
chronyc tracking
# all time sources sane?
chronyc sources -v
# exporter up?
curl -s http://localhost:9123/metrics | grep chrony_up
# scrape works from the monitoring host?
curl -s http://chronos:9123/metrics | grep chrony_tracking_stratum
</code></pre>
<p>If those checks pass, the machine is doing its job.</p>
<h2 id="wrapping-up">Wrapping Up</h2>
<p>The working recipe is surprisingly clean once you stop trying to force <code>ntpd</code> through it:</p>
<ul>
<li>use <code>gpsd</code> for coarse time and visibility</li>
<li>read PPS directly from <code>/dev/pps0</code> with chrony</li>
<li>let chrony serve the LAN</li>
<li>export the state so you can see when something drifts, drops, or silently stops being clever</li>
</ul>
<p>The core lesson is that GPS plus PPS is a two-signal system, and your time daemon needs to respect that split. Let GPS tell you which second it is. Let PPS tell you exactly when it started. Let chrony do the fusion. And let <code>ntpd</code> enjoy retirement.</p>
Reliable Headless Raspberry Pi Provisioning on NixOS2026-04-09T02:40:00+00:002026-04-09T02:40:00+00:00
Ian Johannesen
https://perlpimp.net/blog/reliable-headless-raspberry-pi-provisioning-nixos/<p>Headless Raspberry Pi provisioning sounds like it should be the easy case. Build an image, flash it, power it on, and SSH in. In practice, the first boot is where all the brittleness hides. If Wi-Fi is not up, if the host key is not the one you expected, or if secret decryption depends on services that themselves depend on decrypted secrets, you do not get a graceful fallback. You get a Pi that never appears on the network.</p>
<p>In this repo, the workflow we ended up trusting is:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run .#raspberry-pi-provision-image -- <host> <output-path-or-directory>
</code></pre>
<p>That wrapper builds the host’s NixOS <code>sdImage</code>, copies it to the destination you asked for, and injects the bootstrap files the machine needs before its first boot. For <code>bastet</code>, those files are:</p>
<ul>
<li><code>/var/lib/bootstrap/ssh_host_ed25519_key</code></li>
<li><code>/var/lib/bootstrap/ssh_host_ed25519_key.pub</code></li>
<li><code>/var/lib/networkmanager/system-connections.env</code></li>
</ul>
<p>Those three files solve the hard part: stable SSH identity and working Wi-Fi from the first power-on, before the device is reachable for any remote deployment step.</p>
<p>The important lesson was not “use this exact script.” It was more general: for headless bootstrap on NixOS, image-time secret injection is often more reliable than trying to decrypt first-boot secrets on the device itself. And if you do inject mutable bootstrap state, it belongs under <code>/var/lib/...</code>, not <code>/etc</code>.</p>
<h2 id="the-provisioning-goal">The provisioning goal</h2>
<p>The target was simple enough:</p>
<ol>
<li>Build a NixOS image for a Pi host.</li>
<li>Put that image on removable media.</li>
<li>Power on the Pi somewhere inconvenient.</li>
<li>Have it come up on Wi-Fi immediately with a predictable SSH host key.</li>
</ol>
<p>That last point matters more than it sounds. If the host key is precomputed, you can pin trust ahead of time instead of accepting a surprise key on first contact. If Wi-Fi credentials are already in place, the Pi can join the network without any keyboard, monitor, serial console, or “just plug it into Ethernet for the first boot” workaround.</p>
<p>For a small headless device, first boot is not the time to discover that your secret-management chain is technically elegant but operationally fragile.</p>
<h2 id="the-first-idea-decrypt-on-the-pi-at-boot">The first idea: decrypt on the Pi at boot</h2>
<p>The original design was more declarative on paper.</p>
<p>Keep the image generic. Store Wi-Fi credentials in SOPS. Let the Pi decrypt them on first boot. Use the host’s precomputed SSH key as the age identity. Then let the networking stack consume the decrypted output and bring the machine online.</p>
<p>I still think this is an understandable instinct. It matches the way we want the rest of NixOS to work:</p>
<ul>
<li>secrets live encrypted in the repo</li>
<li>the machine decrypts what it needs locally</li>
<li>runtime services consume those secrets from known paths</li>
</ul>
<p>The problem is that on a fresh, headless Pi, this creates a bootstrap chain with too many timing-sensitive links:</p>
<ol>
<li>The host key has to be present.</li>
<li>SOPS has to use that key successfully.</li>
<li>The decrypted Wi-Fi secret has to land where networking expects it.</li>
<li>Networking has to come up correctly on the first try.</li>
<li>Remote access depends on all of the above already working.</li>
</ol>
<p>That is not impossible. It is just brittle enough that when it fails, recovery is annoying and usually physical.</p>
<p>This is the same family of problem I wrote about in <a href="/blog/sops-bootstrap-problem/">Solving the NixOS SOPS Bootstrap Problem</a>, but harsher. A server that fails a bootstrap deploy can often still be reached over SSH or through a provider console. A Raspberry Pi on somebody else’s shelf does not usually give you that luxury.</p>
<h2 id="what-actually-failed">What actually failed</h2>
<p>Several separate issues pushed us away from first-boot decryption and toward provisioning-time injection.</p>
<h2 id="injecting-into-etc-was-the-wrong-model">Injecting into <code>/etc</code> was the wrong model</h2>
<p>The first mistake was treating <code>/etc</code> as a safe place to drop out-of-band bootstrap files into the image.</p>
<p>That works on plenty of conventional Linux setups. On NixOS, it is the wrong mental model. <code>/etc</code> is declarative and system-managed. If you manually inject files into it before boot, you are competing with the activation logic that will happily regenerate or replace parts of it from the system configuration.</p>
<p>That gave us two flavors of breakage:</p>
<ul>
<li>Wi-Fi-related files could disappear before the service that needed them had actually consumed them.</li>
<li>The injected SSH host key could exist on the first boot and then fail to persist the way we intended across later boots.</li>
</ul>
<p>This was the turning point in the design. The problem was not only “how do we get the secret there?” It was “what kind of path is appropriate for mutable bootstrap state on NixOS?”</p>
<p>The answer is not <code>/etc</code>. It is <code>/var/lib/...</code>.</p>
<h2 id="sops-at-boot-was-too-timing-sensitive">SOPS-at-boot was too timing-sensitive</h2>
<p>The next problem was the decryption chain itself.</p>
<p>On a running NixOS machine, SOPS-managed secrets are great. On a headless first boot where networking depends on the secret being available immediately, they become part of the boot-critical path. That means every dependency matters:</p>
<ul>
<li>the key material has to be present already</li>
<li>the decryption step has to run at the right time</li>
<li>the consumer has to wait for the output</li>
<li>the output path has to survive activation</li>
</ul>
<p>If any part of that chain is wrong, the Pi does not partially succeed. It just never joins the network. That is exactly the sort of failure mode you want to design out of a headless device.</p>
<p>In other words, the issue was not that SOPS is bad. The issue was asking first-boot decryption to solve a bootstrap problem that is easier to solve one step earlier, on the provisioning machine.</p>
<h2 id="wpa-supplicant-made-host-local-bootstrap-harder-than-it-needed-to-be"><code>wpa_supplicant</code> made host-local bootstrap harder than it needed to be</h2>
<p>The earliest iterations were more <code>wpa_supplicant</code>-centric, using host-local wireless configuration paths. That worked in the sense that it was possible to make it go, but it pushed more host-specific logic into the image build than I wanted, and it was not especially pleasant to reason about.</p>
<p>We eventually moved the shared Raspberry Pi path to <code>NetworkManager</code>, which made the image behavior more uniform and the bootstrap layout simpler. That let us inject one runtime file:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">/var/lib/networkmanager/system-connections.env
</code></pre>
<p>and have the image consume it from a persistent runtime location instead of trying to smuggle mutable first-boot state through a declarative <code>/etc</code> path.</p>
<p>This is not an argument that <code>wpa_supplicant</code> is wrong in general. It is an argument that, in this setup, <code>NetworkManager</code> was easier to make boring. And boring is exactly what you want from bootstrap networking.</p>
<h2 id="the-provisioner-initially-lied-to-us">The provisioner initially lied to us</h2>
<p>This one was especially sneaky.</p>
<p>An early version of the image wrapper reported success even when the <code>debugfs</code> writes had failed, because the directories inside the ext4 image did not exist yet. So the script looked happy, the image looked provisioned, and the Pi still came up missing the files we thought we had injected.</p>
<p>That is the worst kind of automation bug: a false positive.</p>
<p>We fixed it by making the provisioner much less trusting:</p>
<ul>
<li>create the destination directories inside the image explicitly</li>
<li>write the files only after those directories exist</li>
<li>assert that the injected files are present afterward</li>
<li>verify the finished image contents directly</li>
</ul>
<p>Once the script started failing hard instead of cheerfully pretending, the whole flow became much easier to trust.</p>
<h2 id="the-design-we-kept">The design we kept</h2>
<p>The reliable version of the workflow is much simpler:</p>
<ol>
<li>Keep the source secrets in SOPS in the repo.</li>
<li>Decrypt them on the provisioning machine, where the age identity already exists.</li>
<li>Build the host’s <code>sdImage</code>.</li>
<li>Inject the runtime files directly into the image before first boot.</li>
<li>Store those files under persistent runtime paths in <code>/var/lib/...</code>.</li>
<li>Point services at those paths directly.</li>
</ol>
<p>For this host, that means:</p>
<ul>
<li>OpenSSH uses <code>/var/lib/bootstrap/ssh_host_ed25519_key</code></li>
<li>NetworkManager reads <code>/var/lib/networkmanager/system-connections.env</code></li>
</ul>
<p>That design is slightly less symmetrical than “the machine decrypts everything itself at boot,” but it is much more reliable in the only moment that really matters: the first boot when the device is not yet reachable.</p>
<h2 id="why-var-lib-is-the-real-lesson">Why <code>/var/lib</code> is the real lesson</h2>
<p>The larger design lesson here is not specific to Raspberry Pis.</p>
<p>If a file is mutable runtime state, or bootstrap state that should survive independently of declarative <code>/etc</code> generation, it probably belongs somewhere under <code>/var/lib</code>. On NixOS, this is not just a filesystem preference. It is a boundary between system-managed declarative configuration and persistent machine-local state.</p>
<p>Once I started looking at the problem through that lens, the right shape became obvious:</p>
<ul>
<li><code>/etc</code> is for declarative configuration rendered by the system</li>
<li><code>/var/lib</code> is for persistent mutable state owned by services or bootstrap tooling</li>
</ul>
<p>Trying to force bootstrap secrets into <code>/etc</code> was really a category error.</p>
<h2 id="the-tradeoff">The tradeoff</h2>
<p>The compromise is that the Wi-Fi secret is currently treated as day-0 bootstrap state.</p>
<p>A plain remote <code>nixos-rebuild</code> later does not automatically rotate the injected Wi-Fi credential unless we add a second runtime-managed secret update path for post-bootstrap changes. So yes, the current setup gives up some declarative neatness.</p>
<p>I think that is the correct trade.</p>
<p>Wi-Fi PSKs do not change often. First-boot connectivity absolutely has to work. Once the Pi is online and reachable, every other kind of refinement becomes easy again. The hard problem is not long-term secret rotation. The hard problem is getting a tiny headless box to show up reliably the first time.</p>
<p>If the price of that reliability is that bootstrap networking is provisioned one step earlier, on the machine building the image, I will happily pay it.</p>
<h2 id="the-practical-workflow">The practical workflow</h2>
<p>For a new managed Raspberry Pi host, the workflow now looks like this:</p>
<ol>
<li>Add the host and image target.</li>
<li>Put <code>wifi_ssid</code>, <code>wifi_psk</code>, and the precomputed SSH host key in the host’s SOPS file.</li>
<li>Run:</li>
</ol>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run .#raspberry-pi-provision-image -- <host> <destination>
</code></pre>
<ol start="4">
<li>Flash the resulting image.</li>
<li>Boot the Pi.</li>
<li>Verify that it is reachable and exporting what you expect:</li>
</ol>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">ssh-keyscan <ip>
upsc -l
curl http://<ip>:9199/ups_metrics
</code></pre>
<p>That last verification block is intentionally concrete. When the machine is a small appliance-style host, “can I SSH in?” is necessary but not sufficient. If it is supposed to be handling UPS monitoring or some other edge function, check the real service path immediately while the provisioning details are still fresh in your head.</p>
<h2 id="closing-thought">Closing thought</h2>
<p>I started out wanting a more elegant first-boot story. What I ended up wanting more was one that worked every time.</p>
<p>For headless Raspberry Pi provisioning on NixOS, image-time secret injection turned out to be the pragmatic answer. Decrypt on the provisioning machine. Inject only the runtime files needed for first boot. Put them under <code>/var/lib/...</code>, not <code>/etc</code>. Let the first boot be boring.</p>
<p>That is not the most purely declarative design. It is, however, the one I would trust before mailing a Pi to another building and hoping it comes back on Wi-Fi.</p>
Monitoring UPS Systems with NUT, Prometheus, and Grafana2026-04-08T00:13:34+00:002026-04-08T00:13:34+00:00
Ian Johannesen
https://perlpimp.net/blog/monitoring-ups-nut-prometheus-grafana/<p>Power failures are one of those problems that feel theoretical right up until a UPS starts beeping and something important goes dark. At that point you do not want to SSH into a host and manually run <code>upsc</code> just to figure out whether the battery is healthy, how much load the unit is carrying, or whether the mains voltage has been wobbling all afternoon.</p>
<p>I wanted the same thing I want from the rest of my infrastructure: one place to look, one scrape path, and graphs that tell me whether the UPS is quietly doing its job or about to ruin my evening.</p>
<p>The setup here monitors two Eaton units:</p>
<table><thead><tr><th>Host</th><th>OS</th><th>UPS Model</th><th>Rated Power</th><th>Role</th></tr></thead><tbody>
<tr><td><code>fatty</code></td><td>FreeBSD 14.7</td><td>Eaton Ellipse PRO 1600</td><td>1600 VA</td><td>Main server running bhyve VMs</td></tr>
<tr><td><code>chronos</code></td><td>Arch Linux ARM</td><td>Eaton 3S 550</td><td>550 VA</td><td>Raspberry Pi GPS/NTP time server</td></tr>
</tbody></table>
<p>Both UPS units are USB-attached to their local host. NUT talks to the hardware, <code>nut_exporter</code> translates that into Prometheus metrics, Prometheus scrapes it, and Grafana turns it into something you can glance at in five seconds.</p>
<h2 id="how-the-stack-fits-together">How the stack fits together</h2>
<p>The data path is simple:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">UPS --USB--> NUT driver + upsd + upsmon --TCP:3493--> nut_exporter --HTTP:9199/ups_metrics--> Prometheus --> Grafana
</code></pre>
<p>NUT is really three pieces:</p>
<ol>
<li><code>usbhid-ups</code> talks to the UPS over USB HID.</li>
<li><code>upsd</code> serves UPS state to clients on TCP port 3493.</li>
<li><code>upsmon</code> watches battery state and handles shutdown behavior.</li>
</ol>
<p><code>nut_exporter</code> connects to <code>upsd</code>, asks for a defined set of variables, and exposes them as Prometheus metrics. Prometheus does not need to know anything about USB, Eaton, or NUT internals. It just scrapes HTTP.</p>
<h2 id="nut-configuration">NUT configuration</h2>
<p>The configuration is almost the same on both hosts. The big FreeBSD wrinkle is pathing: on Linux the config usually lives under <code>/etc/nut/</code>, while on FreeBSD it typically lives under <code>/usr/local/etc/nut/</code>. The service names differ too: Linux usually splits the units out, while FreeBSD gives you <code>nut</code> and <code>nut_upsmon</code>.</p>
<p>The core device definition is minimal:</p>
<pre data-lang="ini" class="language-ini "><code class="language-ini" data-lang="ini"># /etc/nut/ups.conf on Linux
# /usr/local/etc/nut/ups.conf on FreeBSD
[ups]
driver = usbhid-ups
port = auto
desc = "Eaton UPS"
</code></pre>
<p><code>port = auto</code> is usually enough for a single directly attached unit. The section name matters because that becomes the UPS identifier later, for example <code>upsc ups@localhost</code>.</p>
<p>Restrict <code>upsd</code> to localhost if the exporter runs on the same machine:</p>
<pre data-lang="ini" class="language-ini "><code class="language-ini" data-lang="ini"># upsd.conf
LISTEN 127.0.0.1 3493
</code></pre>
<p>And the basic <code>upsmon</code> user and monitor configuration looks like this:</p>
<pre data-lang="ini" class="language-ini "><code class="language-ini" data-lang="ini"># upsd.users
[monitor]
password = secret
upsmon master
</code></pre>
<pre data-lang="ini" class="language-ini "><code class="language-ini" data-lang="ini"># upsmon.conf
MONITOR ups@localhost 1 monitor secret master
SHUTDOWNCMD "/sbin/shutdown -h +0"
</code></pre>
<p>For a directly attached UPS on a single host, standalone mode is the right fit:</p>
<pre data-lang="ini" class="language-ini "><code class="language-ini" data-lang="ini"># nut.conf
MODE=standalone
</code></pre>
<p>That tells NUT to run the driver, <code>upsd</code>, and <code>upsmon</code> together.</p>
<h2 id="verifying-nut-before-you-add-prometheus">Verifying NUT before you add Prometheus</h2>
<p>Do not start with Grafana. Start with <code>upsc</code>.</p>
<p>These are the checks that matter most:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># Linux
systemctl status nut-server
systemctl status nut-monitor
systemctl status nut-driver
# FreeBSD
service nut status
service nut_upsmon status
# List UPS names known to upsd
upsc -l
# Dump all variables
upsc ups@localhost
# Spot check the useful ones
upsc ups@localhost battery.charge
upsc ups@localhost ups.status
upsc ups@localhost input.voltage
# Check direct driver state
upsdrvctl status
</code></pre>
<p>If this layer is broken, Prometheus is not the problem.</p>
<p>The common failure modes are the usual ones:</p>
<ul>
<li>USB permissions are wrong, so the driver cannot open the device.</li>
<li>The wrong driver is configured in <code>ups.conf</code>.</li>
<li>The UPS has gone stale and <code>upsc</code> reports <code>Data stale</code>.</li>
<li><code>upsd</code> is not listening where you think it is.</li>
<li>FreeBSD-specific config paths or service names got copied from Linux docs without adjustment.</li>
</ul>
<p>For Eaton gear, <code>usbhid-ups</code> is usually the right driver. If the box has multiple USB UPS devices, you may need to add <code>serial</code>, <code>vendorid</code>, or <code>productid</code> instead of relying on autodetect.</p>
<h2 id="exporting-nut-metrics">Exporting NUT metrics</h2>
<p>The exporter here is <a rel="noopener noreferrer" target="_blank" href="https://github.com/DRuggeri/nut_exporter">DRuggeri’s nut_exporter</a>. It is a small Go binary that queries <code>upsd</code> and exposes UPS metrics over HTTP.</p>
<p>The key behavior to remember is that it serves UPS metrics on <code>/ups_metrics</code>, not the Prometheus default <code>/metrics</code>. Miss that detail and you will spend too long staring at empty scrape targets.</p>
<h3 id="installing-the-exporter">Installing the exporter</h3>
<p>Example release install commands:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># Linux arm64 (Raspberry Pi)
curl -LO https://github.com/DRuggeri/nut_exporter/releases/download/2.5.2/nut_exporter-2.5.2.linux-arm64.tar.gz
tar xzf nut_exporter-2.5.2.linux-arm64.tar.gz
sudo cp nut_exporter-2.5.2.linux-arm64/nut_exporter /usr/local/bin/
# FreeBSD amd64
curl -LO https://github.com/DRuggeri/nut_exporter/releases/download/2.5.2/nut_exporter-2.5.2.freebsd-amd64.tar.gz
tar xzf nut_exporter-2.5.2.freebsd-amd64.tar.gz
sudo cp nut_exporter-2.5.2.freebsd-amd64/nut_exporter /usr/local/bin/
</code></pre>
<h3 id="running-it">Running it</h3>
<p>This is the flag set I care about:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">/usr/local/bin/nut_exporter \
--web.listen-address=:9199 \
--metrics.namespace=network_ups_tools \
--nut.vars_enable=battery.charge,battery.voltage,battery.runtime,input.voltage,input.voltage.nominal,output.voltage,output.frequency,ups.load,ups.status,ups.power,ups.power.nominal,ups.realpower,ups.beeper.status
</code></pre>
<p>Those flags do three useful things:</p>
<ul>
<li><code>--web.listen-address=:9199</code> chooses the HTTP port.</li>
<li><code>--metrics.namespace=network_ups_tools</code> keeps the metric names conventional.</li>
<li><code>--nut.vars_enable=...</code> whitelists the exact NUT variables worth scraping.</li>
</ul>
<p>Without the variable whitelist, you end up exporting a lot of noise. UPS metrics are already niche enough; there is no need to make the series set bigger than necessary.</p>
<h3 id="systemd-service-on-linux">systemd service on Linux</h3>
<pre data-lang="ini" class="language-ini "><code class="language-ini" data-lang="ini">[Unit]
Description=NUT Exporter
After=nut-server.service
[Service]
User=nobody
ExecStart=/usr/local/bin/nut_exporter \
--web.listen-address=:9199 \
--metrics.namespace=network_ups_tools \
--nut.vars_enable=battery.charge,battery.voltage,battery.runtime,input.voltage,input.voltage.nominal,output.voltage,output.frequency,ups.load,ups.status,ups.power,ups.power.nominal,ups.realpower,ups.beeper.status
Restart=always
[Install]
WantedBy=multi-user.target
</code></pre>
<h3 id="freebsd-startup">FreeBSD startup</h3>
<p>On FreeBSD I keep it simple in <code>rc.conf</code>:</p>
<pre data-lang="sh" class="language-sh "><code class="language-sh" data-lang="sh">nut_exporter_enable="YES"
nut_exporter_flags="--web.listen-address=:9199 --metrics.namespace=network_ups_tools --nut.vars_enable=battery.charge,battery.voltage,battery.runtime,input.voltage,input.voltage.nominal,output.voltage,output.frequency,ups.load,ups.status,ups.power,ups.power.nominal,ups.realpower,ups.beeper.status"
</code></pre>
<p>If you prefer a dedicated rc.d script, that works too. The important part is that the exporter comes up after NUT and points at the same UPS name that <code>upsc</code> uses.</p>
<h3 id="debugging-the-exporter">Debugging the exporter</h3>
<p>These checks tell you quickly whether the exporter layer is alive:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">ps aux | grep nut_exporter
curl -s http://localhost:9199/ups_metrics | head -30
curl -s http://localhost:9199/ups_metrics | grep network_ups_tools_ups_status
curl -s http://localhost:9199/ups_metrics | grep network_ups_tools_battery_charge
curl -s http://localhost:9199/ups_metrics | grep network_ups_tools_device_info
</code></pre>
<p>And from the monitoring host:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">curl -s http://fatty:9199/ups_metrics | grep network_ups_tools_ups_load
curl -s http://chronos:9199/ups_metrics | grep network_ups_tools_ups_load
</code></pre>
<p>If the endpoint is empty, either <code>upsd</code> is unreachable or the UPS name is wrong. If the metrics exist but values are stale or zero, NUT itself is usually the real problem.</p>
<h2 id="the-metrics-worth-caring-about">The metrics worth caring about</h2>
<p>The exporter produces a useful mix of identity, status, load, battery, and electrical metrics.</p>
<p>Device identity comes through as labels:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">network_ups_tools_device_info{mfr="EATON",model="Ellipse PRO 1600",serial="0",type="ups"} 1
network_ups_tools_device_info{mfr="EATON",model="Eaton 3S 550",serial="Blank",type="ups"} 1
</code></pre>
<p>The most important metric family is the status flags:</p>
<table><thead><tr><th>Metric</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>network_ups_tools_ups_status{flag="OL"}</code></td><td>Online, mains power present</td></tr>
<tr><td><code>network_ups_tools_ups_status{flag="OB"}</code></td><td>On battery</td></tr>
<tr><td><code>network_ups_tools_ups_status{flag="LB"}</code></td><td>Low battery</td></tr>
<tr><td><code>network_ups_tools_ups_status{flag="CHRG"}</code></td><td>Charging</td></tr>
<tr><td><code>network_ups_tools_ups_status{flag="DISCHRG"}</code></td><td>Discharging</td></tr>
<tr><td><code>network_ups_tools_ups_status{flag="BOOST"}</code></td><td>Boosting low input voltage</td></tr>
<tr><td><code>network_ups_tools_ups_status{flag="TRIM"}</code></td><td>Trimming high input voltage</td></tr>
<tr><td><code>network_ups_tools_ups_status{flag="RB"}</code></td><td>Replace battery</td></tr>
</tbody></table>
<p>Under normal conditions, <code>OL=1</code> and everything else is <code>0</code>. During an outage you should see <code>OB=1</code>, <code>OL=0</code>, and usually <code>DISCHRG=1</code>.</p>
<p>The rest are the practical numbers:</p>
<table><thead><tr><th>Metric</th><th>Unit</th><th>Why it matters</th></tr></thead><tbody>
<tr><td><code>network_ups_tools_ups_realpower</code></td><td>watts</td><td>Actual power draw</td></tr>
<tr><td><code>network_ups_tools_ups_power</code></td><td>VA</td><td>Apparent power</td></tr>
<tr><td><code>network_ups_tools_ups_power_nominal</code></td><td>VA</td><td>Rated UPS capacity</td></tr>
<tr><td><code>network_ups_tools_ups_load</code></td><td>percent</td><td>Capacity percentage in use</td></tr>
<tr><td><code>network_ups_tools_battery_charge</code></td><td>percent</td><td>Battery state of charge</td></tr>
<tr><td><code>network_ups_tools_battery_runtime</code></td><td>seconds</td><td>Estimated remaining runtime</td></tr>
<tr><td><code>network_ups_tools_battery_voltage</code></td><td>volts</td><td>Battery voltage</td></tr>
<tr><td><code>network_ups_tools_input_voltage</code></td><td>volts</td><td>Incoming mains voltage</td></tr>
<tr><td><code>network_ups_tools_output_voltage</code></td><td>volts</td><td>UPS output voltage</td></tr>
<tr><td><code>network_ups_tools_output_frequency</code></td><td>hertz</td><td>Output frequency</td></tr>
</tbody></table>
<p>A handy derived value for dashboards is capacity utilization:</p>
<pre data-lang="promql" class="language-promql "><code class="language-promql" data-lang="promql">(network_ups_tools_ups_realpower * 100) / network_ups_tools_ups_power_nominal
</code></pre>
<p>That is often more honest than the vendor’s own load percentage, especially when you want a quick visual threshold against the unit’s rated capacity.</p>
<p>One cross-model wrinkle: not every UPS exposes the same variables. My Eaton Ellipse PRO 1600 reports things like <code>input.voltage</code>, <code>output.frequency</code>, and <code>ups.power</code>. The smaller Eaton 3S 550 does not expose all of those. That is normal. The exporter simply omits missing variables instead of inventing zeros.</p>
<h2 id="scraping-it-with-prometheus">Scraping it with Prometheus</h2>
<p>This is the scrape configuration on the monitoring host:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{ network, ... }:
{
services.prometheus.scrapeConfigs = [
{
job_name = "nut";
honor_labels = true;
metrics_path = "/ups_metrics";
static_configs = [
{
targets = [ "${network.hosts.fatty.ip}:9199" ];
labels = { instance = "fatty"; };
}
{
targets = [ "${network.hosts.chronos-wired.ip}:9199" ];
labels = { instance = "chronos"; };
}
];
}
];
}
</code></pre>
<p>Again, the important line is <code>metrics_path = "/ups_metrics"</code>. Everything else is ordinary Prometheus.</p>
<h2 id="building-the-grafana-dashboard">Building the Grafana dashboard</h2>
<p>The dashboard I wanted was straightforward:</p>
<ul>
<li>status at the top so I can see online vs on-battery immediately</li>
<li>current voltage, load, runtime, and beeper state as stat panels</li>
<li>smoothed time series for power, battery charge, load, runtime, and voltages</li>
<li>a variable for selecting one or more UPS instances</li>
<li>a smoothing interval variable so noisy readings do not turn every graph into static</li>
</ul>
<p>The useful template variables are:</p>
<ul>
<li><code>$instance</code>, populated from <code>label_values(network_ups_tools_ups_load, instance)</code></li>
<li><code>$smoothing_interval</code>, with values like <code>1m</code>, <code>5m</code>, <code>15m</code>, <code>1h</code>, <code>1d</code>, and <code>1w</code></li>
</ul>
<p>For the time series panels I use <code>avg_over_time(...[$smoothing_interval])</code>. UPS readings are often twitchy enough that raw charts are less helpful than a short moving average.</p>
<p>Here is the full dashboard JSON:</p>
<pre data-lang="json" class="language-json "><code class="language-json" data-lang="json">{
"annotations": { "list": [] },
"description": "UPS monitoring via NUT exporter (DRuggeri)",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"links": [],
"panels": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"mappings": [
{
"options": {
"0": { "text": "Offline", "color": "red" },
"1": { "text": "Online", "color": "green" }
},
"type": "value"
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": null },
{ "color": "green", "value": 1 }
]
}
},
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
"id": 1,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "network_ups_tools_ups_status{instance=~\"$instance\", flag=\"OL\"}",
"instant": true,
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"title": "Status",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"thresholds": {
"mode": "absolute",
"steps": [{ "color": "green", "value": null }]
},
"unit": "volt"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 4, "x": 6, "y": 0 },
"id": 2,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "network_ups_tools_input_voltage{instance=~\"$instance\"}",
"instant": true,
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"title": "Input Voltage",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"decimals": 0,
"max": 100,
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "#EAB839", "value": 50 },
{ "color": "red", "value": 75 }
]
},
"unit": "percent"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 4, "x": 10, "y": 0 },
"id": 3,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "network_ups_tools_ups_load{instance=~\"$instance\"}",
"instant": true,
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"title": "Load",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": null },
{ "color": "orange", "value": 300 },
{ "color": "green", "value": 600 }
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 4, "x": 14, "y": 0 },
"id": 4,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "network_ups_tools_battery_runtime{instance=~\"$instance\"}",
"instant": true,
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"title": "Runtime",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"mappings": [
{
"options": {
"0": { "text": "Disabled", "color": "red" },
"1": { "text": "Enabled", "color": "green" }
},
"type": "value"
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": null },
{ "color": "green", "value": 1 }
]
}
},
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 18, "y": 0 },
"id": 5,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "network_ups_tools_ups_beeper_status{instance=~\"$instance\"}",
"instant": true,
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"title": "Beeper Status",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "opacity",
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true,
"stacking": { "group": "A", "mode": "none" },
"thresholdsStyle": { "mode": "off" }
},
"decimals": 1,
"unit": "watt"
},
"overrides": []
},
"gridPos": { "h": 10, "w": 24, "x": 0, "y": 4 },
"id": 6,
"options": {
"legend": {
"calcs": ["lastNotNull", "mean"],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": { "mode": "multi", "sort": "none" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "avg_over_time(network_ups_tools_ups_realpower{instance=~\"$instance\"}[$smoothing_interval])",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"title": "Power Consumption [$smoothing_interval]",
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "opacity",
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true,
"stacking": { "group": "A", "mode": "none" },
"thresholdsStyle": { "mode": "off" }
},
"decimals": 0,
"max": 100,
"min": 0,
"unit": "percent"
},
"overrides": []
},
"gridPos": { "h": 6, "w": 12, "x": 0, "y": 14 },
"id": 7,
"options": {
"legend": {
"calcs": ["lastNotNull"],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": { "mode": "multi", "sort": "none" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "avg_over_time(network_ups_tools_battery_charge{instance=~\"$instance\"}[$smoothing_interval])",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"title": "Battery Charge [$smoothing_interval]",
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "thresholds" },
"max": 100,
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "red", "value": 80 }
]
},
"unit": "percent"
},
"overrides": []
},
"gridPos": { "h": 6, "w": 12, "x": 12, "y": 14 },
"id": 8,
"options": {
"minVizHeight": 75,
"minVizWidth": 75,
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"showThresholdLabels": false,
"showThresholdMarkers": true,
"sizing": "auto"
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "(network_ups_tools_ups_realpower{instance=~\"$instance\"} * 100) / network_ups_tools_ups_power_nominal{instance=~\"$instance\"}",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"title": "Capacity Utilization",
"type": "gauge"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "opacity",
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true,
"stacking": { "group": "A", "mode": "none" },
"thresholdsStyle": { "mode": "off" }
},
"max": 100,
"min": 0,
"unit": "percent"
},
"overrides": []
},
"gridPos": { "h": 6, "w": 12, "x": 0, "y": 20 },
"id": 9,
"options": {
"legend": {
"calcs": ["lastNotNull"],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": { "mode": "multi", "sort": "none" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "avg_over_time(network_ups_tools_ups_load{instance=~\"$instance\"}[$smoothing_interval])",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"title": "Load [$smoothing_interval]",
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "opacity",
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true,
"stacking": { "group": "A", "mode": "none" },
"thresholdsStyle": { "mode": "off" }
},
"unit": "s"
},
"overrides": []
},
"gridPos": { "h": 6, "w": 12, "x": 12, "y": 20 },
"id": 10,
"options": {
"legend": {
"calcs": ["lastNotNull", "min"],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": { "mode": "multi", "sort": "none" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "avg_over_time(network_ups_tools_battery_runtime{instance=~\"$instance\"}[$smoothing_interval])",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"title": "Runtime [$smoothing_interval]",
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "opacity",
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true,
"stacking": { "group": "A", "mode": "none" },
"thresholdsStyle": { "mode": "off" }
},
"decimals": 1,
"unit": "volt"
},
"overrides": []
},
"gridPos": { "h": 6, "w": 12, "x": 0, "y": 26 },
"id": 11,
"options": {
"legend": {
"calcs": ["lastNotNull"],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": { "mode": "multi", "sort": "none" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "avg_over_time(network_ups_tools_input_voltage{instance=~\"$instance\"}[$smoothing_interval])",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"title": "Input Voltage [$smoothing_interval]",
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "opacity",
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true,
"stacking": { "group": "A", "mode": "none" },
"thresholdsStyle": { "mode": "off" }
},
"decimals": 1,
"unit": "volt"
},
"overrides": []
},
"gridPos": { "h": 6, "w": 12, "x": 12, "y": 26 },
"id": 12,
"options": {
"legend": {
"calcs": ["lastNotNull"],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": { "mode": "multi", "sort": "none" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "avg_over_time(network_ups_tools_output_voltage{instance=~\"$instance\"}[$smoothing_interval])",
"legendFormat": "{{instance}}",
"refId": "A"
}
],
"title": "Output Voltage [$smoothing_interval]",
"type": "timeseries"
}
],
"preload": false,
"refresh": "10s",
"schemaVersion": 42,
"tags": [],
"templating": {
"list": [
{
"current": {},
"includeAll": false,
"multi": false,
"name": "datasource",
"query": "prometheus",
"refresh": 1,
"type": "datasource"
},
{
"current": { "text": ["All"], "value": ["$__all"] },
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"definition": "label_values(network_ups_tools_ups_load, instance)",
"includeAll": true,
"label": "Instance",
"multi": true,
"name": "instance",
"query": { "query": "label_values(network_ups_tools_ups_load, instance)", "refId": "StandardVariableQuery" },
"refresh": 1,
"sort": 2,
"type": "query"
},
{
"current": { "text": "15m", "value": "15m" },
"includeAll": false,
"label": "Smoothing",
"name": "smoothing_interval",
"options": [
{ "selected": false, "text": "1m", "value": "1m" },
{ "selected": false, "text": "5m", "value": "5m" },
{ "selected": true, "text": "15m", "value": "15m" },
{ "selected": false, "text": "1h", "value": "1h" },
{ "selected": false, "text": "1d", "value": "1d" },
{ "selected": false, "text": "1w", "value": "1w" }
],
"query": "1m,5m,15m,1h,1d,1w",
"type": "custom"
}
]
},
"time": { "from": "now-7d", "to": "now" },
"timepicker": {},
"timezone": "",
"title": "NUT UPS",
"uid": "nut-ups"
}
</code></pre>
<h2 id="quick-health-checks">Quick health checks</h2>
<p>These are the commands I actually want handy when something looks wrong:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># On the UPS host
upsc ups@localhost
upsc ups@localhost ups.status
upsc ups@localhost battery.charge
upsc ups@localhost ups.load
curl -s http://localhost:9199/ups_metrics | grep network_ups_tools_ups_status
# From the monitoring host
curl -s http://fatty:9199/ups_metrics | grep ups_load
curl -s http://chronos:9199/ups_metrics | grep ups_load
# Check for on-battery state
curl -s http://fatty:9199/ups_metrics | grep 'flag="OB"'
</code></pre>
<p>If <code>OB</code> flips to <code>1</code>, you are on battery. If <code>LB</code> joins it, stop admiring the dashboard and start caring about shutdown sequencing.</p>
<h2 id="wrapping-up">Wrapping up</h2>
<p>This stack is not complicated, but it is full of little edges that are easy to forget six months later: FreeBSD path differences, NUT service naming, the exporter’s <code>/ups_metrics</code> path, and the fact that different UPS models expose different variables.</p>
<p>Once those are handled, though, UPS monitoring becomes just another Prometheus job. You get trend data for load, battery, and voltage, quick confirmation that a host is still online, and a much better answer than “the UPS is making noises again.”</p>
Monitoring Gardena on NixOS with prometheus-gardena-exporter2026-04-07T00:10:00+00:002026-04-07T00:10:00+00:00
Ian Johannesen
https://perlpimp.net/blog/prometheus-gardena-exporter-nixos/<p>Your Gardena setup already knows useful things. Soil humidity. Soil temperature. Whether a valve is open. Whether a sensor battery is about to ruin your weekend. The problem is that all of it lives inside a phone app, which is a terrible place for operational data.</p>
<p>If you already run Prometheus and Grafana, that is where this belongs.</p>
<p><a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/prometheus-gardena-exporter">prometheus-gardena-exporter</a> is a Rust exporter for the Gardena smart system API. It handles the OAuth client credentials flow, discovers your Gardena location, pulls an initial snapshot, keeps a live WebSocket connection open for realtime updates, periodically reconciles state, and exposes the whole thing as Prometheus metrics. It also ships with a NixOS module, optional local scrape wiring, and a Grafana dashboard, so you do not have to spend your evening building plumbing before you can graph a watering zone.</p>
<p>The useful twist is water usage. Gardena does not give you a real flow meter reading here. The exporter models liters from valve runtime with a default liters-per-minute value, and lets you override that globally or per valve. So no, it is not a meter. But yes, it can still become a pretty useful guesstimate once you tune it to your actual zones.</p>
<h2 id="how-the-data-gets-there">How the data gets there</h2>
<p>The path from Gardena to Grafana looks like this:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">Gardena API
-> OAuth token
-> location snapshot
-> WebSocket updates
prometheus-gardena-exporter
-> /metrics
Prometheus
-> queries
Grafana
</code></pre>
<p>At startup, the exporter authenticates with the Husqvarna auth API using your application key and secret, discovers available locations, picks the configured location or auto-selects the only available one, and fetches a full location snapshot. After that it subscribes to the WebSocket stream and keeps an in-memory view of device and service state current. A periodic snapshot refresh reconciles anything the live stream might have missed.</p>
<p>That design matters because the Gardena API is not something you want to poll on every Prometheus scrape. The exporter absorbs the API weirdness once and presents a normal <code>/metrics</code> endpoint to the rest of your monitoring stack.</p>
<h2 id="getting-it-into-your-flake">Getting it into your flake</h2>
<p>You have two straightforward ways to consume this on NixOS.</p>
<h3 id="via-my-ijohanne-nur-packages-flake">Via my <code>ijohanne</code> NUR packages flake</h3>
<p>If you already pull in my package set, import the module from there:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.11";
ijohanne-nur.url = "github:ijohanne/nur-packages";
};
outputs = { nixpkgs, ijohanne-nur, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
modules = [
ijohanne-nur.nixosModules.prometheus-gardena-exporter
./configuration.nix
];
};
};
}
</code></pre>
<h3 id="directly-from-the-exporter-flake">Directly from the exporter flake</h3>
<p>If you want to pin just this exporter and nothing else, pull it straight from the repo:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.11";
prometheus-gardena-exporter.url = "github:ijohanne/prometheus-gardena-exporter";
};
outputs = { nixpkgs, prometheus-gardena-exporter, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
modules = [
prometheus-gardena-exporter.nixosModules.default
./configuration.nix
];
};
};
}
</code></pre>
<p>Same module either way. Pick whichever matches how you already manage third-party flake inputs.</p>
<h2 id="gardena-application-setup">Gardena application setup</h2>
<p>Before NixOS gets involved, you need a Gardena application in the Husqvarna developer portal:</p>
<ol>
<li>Create or open an app at <a rel="noopener noreferrer" target="_blank" href="https://developer.husqvarnagroup.cloud/apps">developer.husqvarnagroup.cloud/apps</a></li>
<li>Add a redirect URL such as <code>http://localhost</code></li>
<li>Copy the application key and application secret</li>
</ol>
<p>One detail worth calling out because it is mildly annoying the first time you hit it: the application key is used as both the <code>X-Api-Key</code> header and the OAuth <code>client_id</code>. The application secret is the OAuth <code>client_secret</code>.</p>
<p>The exporter includes a helper command that prints the raw token request:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run . -- print-token-curl
</code></pre>
<p>That expands to:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">curl -fsSL -X POST -d "grant_type=client_credentials&client_id=$GARDENA_APPLICATION_KEY&client_secret=$GARDENA_APPLICATION_SECRET" \
https://api.authentication.husqvarnagroup.dev/v1/oauth2/token
</code></pre>
<p>If you want to test the auth flow directly:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">export GARDENA_APPLICATION_KEY="..."
export GARDENA_APPLICATION_SECRET="..."
nix run . -- fetch-token --raw
</code></pre>
<p>To discover available locations:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run . -- list-locations \
--application-key "$GARDENA_APPLICATION_KEY" \
--application-secret "$GARDENA_APPLICATION_SECRET"
</code></pre>
<p>If the application only has access to one location, the exporter can auto-select it. If it has access to more than one, set <code>locationId</code>.</p>
<h2 id="nixos-configuration">NixOS configuration</h2>
<p>Here is the practical NixOS setup:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
services.prometheus-gardena-exporter = {
enable = true;
enableLocalScraping = true;
enableGrafanaDashboard = true;
applicationKeyFile = /run/secrets/gardena-application-key;
applicationSecretFile = /run/secrets/gardena-application-secret;
locationId = "db789fe8-2af2-4eaf-a0ef-8ad795617971";
estimatedFlowLitersPerMinute = 3.5;
estimatedFlowLitersPerMinuteByValve = {
"5f7a3e6e-1111-2222-3333-444444444444" = 1.2;
"8c9d0a1b-5555-6666-7777-888888888888" = 6.0;
};
validateAuthOnStartup = true;
};
}
</code></pre>
<p>Three options do most of the quality-of-life work here:</p>
<ul>
<li><code>enableLocalScraping = true;</code> adds a Prometheus scrape target for you</li>
<li><code>enableGrafanaDashboard = true;</code> provisions the included Gardena dashboard automatically</li>
<li><code>validateAuthOnStartup = true;</code> fails fast if the app key or secret is wrong instead of looking “healthy” while doing nothing useful</li>
</ul>
<p>There is also a nice Nix-specific detail in the module implementation: the service uses <code>LoadCredential</code>, so the application key and secret are read from files at runtime and never written into the Nix store.</p>
<p>By default the exporter listens on <code>127.0.0.1:9134</code>, which is exactly what I want for a local Prometheus scrape.</p>
<h2 id="finding-the-valve-ids-you-actually-need">Finding the valve IDs you actually need</h2>
<p>Per-valve flow overrides are keyed by Gardena <code>service_id</code>, not by the friendly zone names from the app. That sounds annoying until you realize the exporter already has a helper for it:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run . -- list-valves \
--application-key "$GARDENA_APPLICATION_KEY" \
--application-secret "$GARDENA_APPLICATION_SECRET"
</code></pre>
<p>Without Nix:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">cargo run -- list-valves \
--application-key "$GARDENA_APPLICATION_KEY" \
--application-secret "$GARDENA_APPLICATION_SECRET"
</code></pre>
<p>The output is tab-separated and includes:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">location_id location device_id controller_name service_id valve_name
</code></pre>
<p>That gives you the stable <code>service_id</code> values you need for <code>estimatedFlowLitersPerMinuteByValve</code>.</p>
<h2 id="water-usage-is-a-model-not-a-meter">Water usage is a model, not a meter</h2>
<p>This is the part worth being explicit about.</p>
<p>Gardena cannot tell this exporter how many liters actually flowed through a zone. There is no physical flow sensor data here. What you get instead is a modeled estimate based on valve-open time.</p>
<p>The built-in default is <code>3.5 L/min</code>. That number is derived from a rough monthly estimate:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">5 m3/month = 5000 L/month
45 watering minutes/day x 30 days = 1350 watering minutes/month
5000 / 1350 = 3.7 L/min
</code></pre>
<p>The exporter rounds that down slightly and uses <code>3.5 L/min</code> as a conservative default.</p>
<p>That default is fine as a starting point. It is also where people stop too early.</p>
<p>If one zone is a drip line and another is a spray-heavy bed, using one shared liters-per-minute value for both is going to produce nonsense-shaped precision. The fix is per-valve overrides:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.prometheus-gardena-exporter = {
estimatedFlowLitersPerMinute = 3.5;
estimatedFlowLitersPerMinuteByValve = {
"5f7a3e6e-1111-2222-3333-444444444444" = 1.2;
"8c9d0a1b-5555-6666-7777-888888888888" = 6.0;
};
};
</code></pre>
<p>That is where the model gets much more believable. You are still estimating, but now you are estimating with zone-specific assumptions instead of pretending every part of the garden consumes water the same way.</p>
<p>This is most useful when:</p>
<ul>
<li>only one valve is active at a time</li>
<li>each zone has a reasonably consistent emitter profile</li>
<li>your water pressure is fairly stable</li>
</ul>
<p>Treat the result as a guesstimate with sharp edges, not as a utility-grade reading. It is still good enough to answer real questions like “which zone is responsible for most of this month’s watering” or “did that new schedule double runtime for the thirsty bed again?”</p>
<h2 id="what-you-get-in-prometheus-and-grafana">What you get in Prometheus and Grafana</h2>
<p>The exporter covers the useful things first.</p>
<p>On the exporter-health side, you get metrics like:</p>
<ul>
<li><code>gardena_exporter_connected</code></li>
<li><code>gardena_exporter_last_successful_sync_timestamp_seconds</code></li>
<li><code>gardena_exporter_websocket_reconnects_total</code></li>
<li><code>gardena_exporter_snapshot_refreshes_total</code></li>
</ul>
<p>On the device side, you get:</p>
<ul>
<li><code>gardena_sensor_soil_humidity_percent</code></li>
<li><code>gardena_sensor_soil_temperature_celsius</code></li>
<li><code>gardena_sensor_ambient_temperature_celsius</code></li>
<li><code>gardena_sensor_light_intensity_lux</code></li>
<li><code>gardena_device_battery_level_percent</code></li>
<li><code>gardena_device_rf_link_level_percent</code></li>
</ul>
<p>And on the watering side, the interesting metrics are:</p>
<ul>
<li><code>gardena_valve_open</code></li>
<li><code>gardena_valve_duration_seconds</code></li>
<li><code>gardena_valve_estimated_water_liters_total</code></li>
<li><code>gardena_valve_estimated_current_water_flow_liters_per_minute</code></li>
<li><code>gardena_estimated_water_liters_total</code></li>
<li><code>gardena_estimated_current_water_flow_liters_per_minute</code></li>
</ul>
<p>The included Grafana dashboard leans into the practical views:</p>
<ul>
<li>exporter connection and active valve count</li>
<li>average soil humidity and low-battery quick checks</li>
<li>soil humidity and soil temperature by zone</li>
<li>ambient temperature and light where the hardware exposes them</li>
<li>valve status tables</li>
<li>selected-range estimated water usage</li>
<li>cumulative estimated water by zone</li>
</ul>
<p>In other words, it starts where you would probably end up after an afternoon of dashboard-building anyway.</p>
<h2 id="sharp-edges-and-limitations">Sharp edges and limitations</h2>
<p>The exporter is honest about the shape of the underlying API:</p>
<ul>
<li>snapshot endpoints are rate-limited, so the exporter serves cached state and does not poll on every scrape</li>
<li>the WebSocket URL is short-lived, so the exporter requests it and connects immediately</li>
<li>some Gardena sensors only report <code>soilHumidity</code> and <code>soilTemperature</code>; ambient temperature and light are optional</li>
<li>estimated water totals live in memory and reset when the exporter restarts</li>
</ul>
<p>None of those are deal-breakers. They are just things you want to know before you build a monthly watering report and then restart the service halfway through the month.</p>
<h2 id="garden-telemetry-that-behaves-like-infrastructure">Garden telemetry that behaves like infrastructure</h2>
<p>That is really the point of this exporter.</p>
<p>Your irrigation system should not be a glossy black box that only exists in a mobile app. It should be queryable, graphable, alertable, and boring in the same way the rest of your stack is boring. Gardena provides the devices. Prometheus and Grafana provide the observability story. This exporter is the bridge between the two.</p>
<p>And the water estimation model is a good example of the overall approach: no fake certainty, no pretending the API provides data it does not. Just a useful, tunable approximation that gets better when you tell it what your valves actually do.</p>
<p>That is the kind of tradeoff I will take every time.</p>
BorgBackup and rsync.net on NixOS: Declarative Offsite Backups2026-04-06T12:00:00+00:002026-04-06T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/borgbackup-rsync-net-nixos/<p>If you want offsite backups on NixOS without building a whole backup platform around yourself, Borg plus rsync.net is a very good stopping point.</p>
<p>Borg gives you deduplication, compression, authenticated encryption, and a backup format that is pleasant to inspect and restore from. rsync.net gives you a plain SSH-accessible storage account on top of ZFS, which means you are not learning a proprietary API or shoving tarballs into an object store and hoping future-you still remembers the quirks.</p>
<p>The part that makes this especially nice on NixOS is <code>services.borgbackup.jobs</code>. You can start with the raw CLI until the workflow makes sense, then move the whole thing into declarative config with timers, secrets, retention, and systemd ordering.</p>
<p>This post goes in that order: first the five Borg commands you actually need to know, then the NixOS module, then the operational bits that matter in real life.</p>
<h2 id="install-borg-on-nixos">Install Borg on NixOS</h2>
<p>If you want to run manual restore and verification commands on the machine itself, install Borg explicitly:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">environment.systemPackages = [ pkgs.borgbackup ];
</code></pre>
<p>If you are only using the NixOS module, the backup job will bring Borg along for the service anyway. I still like having the CLI available locally because a backup you cannot inspect manually is not a backup you really trust yet.</p>
<h2 id="rsync-net-setup-ssh-key-and-remote-borg-path">rsync.net setup: SSH key and remote Borg path</h2>
<p>Before touching NixOS config, make sure the boring pieces work:</p>
<ol>
<li>Generate an SSH key dedicated to this backup target.</li>
<li>Add the public key to your rsync.net account.</li>
<li>Verify that you can log in and run the Borg binary rsync.net exposes.</li>
</ol>
<p>For current rsync.net setups, the important quirk is usually not a long hardcoded path like <code>/usr/local/bin/borg1/borg1</code>. Their current docs expose versioned commands such as <code>borg14</code>, so test that directly:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">ssh -i /run/secrets/backup_ssh_key 12345@usw-s001.rsync.net borg14 --version
</code></pre>
<p>If that works, you can tell Borg to use that remote binary via <code>BORG_REMOTE_PATH=borg14</code>. If rsync.net changes their preferred version later, follow their current docs instead of copying an old path from a blog post.</p>
<h2 id="the-borg-crash-course">The Borg crash course</h2>
<p>You do not need to memorize all of Borg. For a normal backup workflow, you mostly care about <code>init</code>, <code>create</code>, <code>list</code>, <code>extract</code>, and <code>prune</code>.</p>
<p>I like setting the repository and SSH options up front:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">export REPO="12345@usw-s001.rsync.net:backups/my-stuff"
export BORG_RSH='ssh -i /run/secrets/backup_ssh_key'
export BORG_REMOTE_PATH='borg14'
</code></pre>
<h3 id="initialize-the-repository">Initialize the repository</h3>
<p>This creates the remote repository metadata and chooses the encryption mode:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">borg init --encryption=repokey-blake2 "$REPO"
</code></pre>
<p>For rsync.net, <code>repokey-blake2</code> is the sensible default. Your data is encrypted before it leaves the machine, and the repository key lives with the repo but is protected by your passphrase. If you pick <code>encryption=none</code>, you are explicitly choosing simplicity over confidentiality.</p>
<h3 id="create-an-archive">Create an archive</h3>
<p>This is the actual backup run:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">borg create --stats --progress \
"$REPO::'{hostname}-{now}'" \
/etc \
/var/lib/myapp
</code></pre>
<p>Every <code>create</code> adds a new archive to the same repository. Because Borg deduplicates chunks across archives, daily backups are not just “tar the whole machine again” with a nicer brand name.</p>
<h3 id="list-archives">List archives</h3>
<p>This shows what is actually in the repository:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">borg list "$REPO"
</code></pre>
<p>Use this early and often. If you are not checking that archives exist with the names you expect, you are doing backup astrology.</p>
<h3 id="extract-data">Extract data</h3>
<p>This is the command that matters when you are tired and something is already on fire:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">mkdir -p /tmp/borg-restore-test
cd /tmp/borg-restore-test
borg extract "$REPO::myhost-2026-04-06T02:00:00" etc/ssh/sshd_config
</code></pre>
<p>One important Borg gotcha: <code>extract</code> writes into your current working directory. Do not casually run it from <code>/</code>.</p>
<h3 id="prune-old-archives">Prune old archives</h3>
<p>This is retention:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">borg prune --list --dry-run \
--glob-archives '{hostname}-*' \
--keep-daily 7 \
--keep-weekly 4 \
--keep-monthly 6 \
"$REPO"
</code></pre>
<p>Run the dry run first. Always. <code>prune</code> is how you turn “this looks tidy” into “I deleted the only restore point from two Tuesdays ago.”</p>
<p>Also note that manual pruning and space reclamation are separate steps in Borg. <code>prune</code> decides what to delete; <code>compact</code> reclaims the space afterward.</p>
<h2 id="moving-to-the-nixos-module">Moving to the NixOS module</h2>
<p>Once the CLI flow makes sense, <code>services.borgbackup.jobs</code> is the nicer way to live with it.</p>
<p>This is a minimal declarative rsync.net job:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{ config, ... }:
{
sops.secrets.backup_ssh_key = {
mode = "0400";
owner = "root";
group = "root";
};
sops.secrets.borg_passphrase = {
mode = "0400";
owner = "root";
group = "root";
};
services.borgbackup.jobs.documents = {
paths = [ "/srv/documents" ];
repo = "12345@usw-s001.rsync.net:backups/documents";
doInit = true;
encryption = {
mode = "repokey-blake2";
passCommand = "cat ${config.sops.secrets.borg_passphrase.path}";
};
compression = "auto,zstd";
startAt = "daily";
environment = {
BORG_RSH = "ssh -i ${config.sops.secrets.backup_ssh_key.path}";
BORG_REMOTE_PATH = "borg14";
};
prune.keep = {
daily = 7;
weekly = 4;
monthly = 6;
};
};
}
</code></pre>
<p>That already gets you most of the value:</p>
<ul>
<li>a scheduled backup job</li>
<li>automatic repository initialization on first run</li>
<li>encrypted backups without hardcoding secrets into the Nix store</li>
<li>retention policy in the same place as the backup definition</li>
</ul>
<p>If you prefer to initialize the repository manually first, you can do that and leave <code>doInit = false;</code>. I still like <code>doInit = true;</code> for fresh hosts because it removes one more piece of “remember to do this by hand later.”</p>
<h2 id="encryption-modes-what-to-pick">Encryption modes: what to pick</h2>
<p>For rsync.net, I would usually choose one of these:</p>
<ul>
<li><code>mode = "repokey-blake2"</code> if you want the normal encrypted-client-side setup</li>
<li><code>mode = "none"</code> only if you intentionally do not want Borg encryption</li>
</ul>
<p>The “none” variant is valid, just not my default:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.borgbackup.jobs.media = {
paths = [ "/srv/media" ];
repo = "12345@usw-s001.rsync.net:backups/media";
encryption.mode = "none";
compression = "auto,zstd";
startAt = "daily";
environment = {
BORG_RSH = "ssh -i ${config.sops.secrets.backup_ssh_key.path}";
BORG_REMOTE_PATH = "borg14";
};
};
</code></pre>
<p>If the remote side is outside your trust boundary, use encryption. That is the entire point of Borg being pleasant for offsite backups.</p>
<h2 id="dump-first-back-up-second">Dump first, back up second</h2>
<p>Backing up a live database directory directly is how you get a repository full of consistency-shaped lies. The safer pattern is:</p>
<ol>
<li>dump the database into a staging directory</li>
<li>make Borg wait for that dump</li>
<li>back up the dump directory</li>
</ol>
<p>For anything non-trivial, I prefer a dedicated oneshot service over jamming everything into <code>preHook</code>. You get clearer logs, a real unit name, and a cleaner failure boundary.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{ config, pkgs, ... }:
{
systemd.services.pg-dump-for-borg = {
description = "Create PostgreSQL dump before Borg backup";
serviceConfig = {
Type = "oneshot";
User = "postgres";
};
script = ''
${pkgs.coreutils}/bin/install -d -m 0700 /var/backup/postgresql
${config.services.postgresql.package}/bin/pg_dumpall \
--clean \
--if-exists \
> /var/backup/postgresql/all.sql
'';
};
systemd.services."borgbackup-job-postgresql" = {
requires = [ "pg-dump-for-borg.service" ];
after = [ "pg-dump-for-borg.service" ];
};
services.borgbackup.jobs.postgresql = {
paths = [ "/var/backup/postgresql" ];
repo = "12345@usw-s001.rsync.net:backups/postgresql";
doInit = true;
encryption = {
mode = "repokey-blake2";
passCommand = "cat ${config.sops.secrets.borg_passphrase.path}";
};
compression = "auto,zstd";
startAt = "daily";
environment = {
BORG_RSH = "ssh -i ${config.sops.secrets.backup_ssh_key.path}";
BORG_REMOTE_PATH = "borg14";
};
prune.keep = {
daily = 7;
weekly = 4;
monthly = 6;
};
};
}
</code></pre>
<p>The key detail is <code>requires</code> plus <code>after</code> on <code>borgbackup-job-postgresql</code>. Starting the Borg job pulls in the dump unit first, and a dump failure stops the backup instead of quietly archiving stale data from yesterday.</p>
<p>For small prep work, <code>preHook</code> is fine. For database dumps, VM snapshots, or anything else you may need to debug later, a dedicated unit usually ages better.</p>
<h2 id="secrets-with-sops-nix">Secrets with sops-nix</h2>
<p>This is where <code>sops-nix</code> earns its keep. The two secrets you usually care about are:</p>
<ul>
<li>the SSH private key used to reach rsync.net</li>
<li>the Borg passphrase</li>
</ul>
<p>Keep both decrypted at activation time, not embedded in Nix expressions:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">sops.secrets.backup_ssh_key = {
mode = "0400";
owner = "root";
group = "root";
};
sops.secrets.borg_passphrase = {
mode = "0400";
owner = "root";
group = "root";
};
</code></pre>
<p>Then reference the generated paths:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">environment = {
BORG_RSH = "ssh -i ${config.sops.secrets.backup_ssh_key.path}";
BORG_REMOTE_PATH = "borg14";
};
encryption = {
mode = "repokey-blake2";
passCommand = "cat ${config.sops.secrets.borg_passphrase.path}";
};
</code></pre>
<p>That keeps the secret material out of the world-readable Nix store, which is the line you do not want to cross just because “it was only a backup key.”</p>
<h2 id="monitoring-and-verification">Monitoring and verification</h2>
<p>Do not stop at “the timer exists.”</p>
<p>At minimum, check the service and read the logs:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">systemctl status borgbackup-job-documents.service
journalctl -u borgbackup-job-documents.service -e
</code></pre>
<p>Then verify the repository itself:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">export REPO="12345@usw-s001.rsync.net:backups/documents"
export BORG_RSH='ssh -i /run/secrets/backup_ssh_key'
export BORG_REMOTE_PATH='borg14'
borg list "$REPO"
borg info "$REPO"
borg check "$REPO"
</code></pre>
<p>And every so often, do the part people skip: restore a file into a throwaway directory and confirm it is actually readable. “The backup job stayed green for six months” is not the same thing as “restore works.”</p>
<h2 id="the-shape-of-a-good-setup">The shape of a good setup</h2>
<p>The nice thing about this stack is that it scales from “I just need offsite copies of a few directories” to “I need a clean, repeatable backup story for stateful services” without changing tools halfway through.</p>
<p>The progression is straightforward:</p>
<ol>
<li>Learn <code>borg init</code>, <code>create</code>, <code>list</code>, <code>extract</code>, and <code>prune</code>.</li>
<li>Confirm rsync.net access with your SSH key and remote Borg path.</li>
<li>Move the workflow into <code>services.borgbackup.jobs</code>.</li>
<li>Put the SSH key and passphrase behind <code>sops-nix</code>.</li>
<li>Make pre-backup state capture explicit with systemd dependencies.</li>
<li>Test restores like you expect your future self to need them.</li>
</ol>
<p>Borg does not make backups magical. It just makes them boring in the best possible way. On NixOS, that is exactly what you want.</p>
Refactoring Dotfiles into a Dendritic Layout2026-04-05T12:00:00+00:002026-04-05T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/refactoring-dotfiles-dendritic-layout/<p>I recently reorganized my dotfiles repo into a Dendritic-style layout. The interesting part was not the rename spree. It was the change in what the repository <em>means</em> when you open it.</p>
<p>Before, the repo was mostly shaped around hosts plus a big <code>configs/</code> tree. That worked for a long time. But as the flake grew across NixOS, nix-darwin, Home Manager, custom packages, patches, and a pile of shared service logic, the old structure started making more and more questions annoying to answer.</p>
<p>Where does this behavior belong?</p>
<p>Is this reusable or host-local?</p>
<p>Should a new machine copy that chunk from another host, or import something more general?</p>
<p>Why does this thing live under <code>configs/</code> if it is clearly not just “configuration” in the narrow sense?</p>
<p>The Dendritic refactor improved a lot of that. It also introduced some real costs. This post is not “everyone should organize dotfiles this way.” It is a higher-level look at what got clearer, what setting up a new host looks like now, and why the clearer architecture also buys you a certain amount of boilerplate and indirection.</p>
<h2 id="what-dendritic-means-here">What Dendritic means here</h2>
<p>Dendritic is not a framework. It is not a new library. It is a repo organization pattern built around flake-parts-style modules and feature-oriented composition.</p>
<p>The basic idea is that you stop treating your repository primarily as:</p>
<ul>
<li>a set of hosts</li>
<li>plus a bag of helper files</li>
</ul>
<p>and start treating it as:</p>
<ul>
<li>a set of reusable aspects and services</li>
<li>grouped by whether they are public, private, or inventory data</li>
<li>with hosts acting more like manifests that opt into those pieces</li>
</ul>
<p>That distinction matters because it changes the default question from:</p>
<p>“Which host owns this logic?”</p>
<p>to:</p>
<p>“Which feature or aspect should define this logic?”</p>
<p>For a small repo, that can be overkill. For a repo that has crossed into multi-host, multi-platform, partly-public, partly-private territory, that question is often the more useful one.</p>
<h2 id="old-shape-versus-new-shape">Old shape versus new shape</h2>
<p>The old repo had a familiar feel:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">.
├── flake.nix
├── hosts/
│ ├── goose/
│ ├── ij-desktop/
│ ├── khosu/
│ └── ...
└── configs/
├── programs/
├── profiles/
├── users/
├── server.nix
├── network.nix
├── secrets.nix
└── ...
</code></pre>
<p>That is a perfectly reasonable place to start. Hosts are visible. Shared things exist. Nothing feels especially abstract.</p>
<p>The problem is that <code>configs/</code> becomes a junk drawer. Languages, user defaults, deployment helpers, shared nix settings, patches, service building blocks, and private inventory all end up adjacent even though they are not really the same kind of thing.</p>
<p>The new layout is more opinionated:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">.
├── flake.nix
├── hosts/
│ ├── goose/
│ ├── ij-desktop/
│ ├── khosu/
│ └── ...
└── modules/
├── community/
│ ├── home/
│ ├── nixos/
│ ├── darwin/
│ ├── lib/
│ ├── packages/
│ └── patches/
└── private/
├── inventory/
├── nixos/
├── darwin/
├── home/
└── shared/
</code></pre>
<p>This split does two important things immediately.</p>
<p>First, it separates <em>publicly reusable</em> modules from <em>repo-private</em> composition. Second, it gives inventory data its own home instead of letting it blur into general-purpose modules.</p>
<p>That alone made the repo easier to read.</p>
<h2 id="what-got-clearer">What got clearer</h2>
<p>The biggest gain was not “more reuse,” although there is more reuse. The biggest gain was clarity of intent.</p>
<p>In the new tree, directories and file names communicate purpose more directly:</p>
<ul>
<li><code>modules/community</code> means reusable surface area I would feel good exporting</li>
<li><code>modules/private</code> means repo-specific composition I want to reuse internally</li>
<li><code>modules/private/inventory</code> means facts about this fleet, not abstract modules</li>
</ul>
<p>That is much better than a flat <code>configs/</code> namespace where everything looks like just another config file.</p>
<p>It also made several kinds of questions easier to answer.</p>
<h2 id="easier-question-is-this-public-private-or-inventory">Easier question: is this public, private, or inventory?</h2>
<p>This used to be fuzzy. A helper might be generic in spirit but buried next to host-specific assumptions. A user registry might live near modules that actually describe behavior. Network facts might be imported like they were just another feature.</p>
<p>Now the repo answers that more directly:</p>
<ul>
<li>public reusable composition goes in <code>modules/community</code></li>
<li>private reusable composition goes in <code>modules/private</code></li>
<li>shared facts about hosts and users go in <code>modules/private/inventory</code></li>
</ul>
<p>That makes maintenance easier because the placement itself carries architectural meaning.</p>
<p>If a file lands in the wrong layer, it feels wrong immediately.</p>
<h2 id="easier-question-where-should-a-new-behavior-go">Easier question: where should a new behavior go?</h2>
<p>Under the old layout, the easiest move was often to open the closest host file and add more logic there.</p>
<p>That works until it doesn’t. Host files slowly become half manifest, half implementation, and then you start copying chunks between them because some other host wants “basically the same thing.”</p>
<p>With the new structure, the default move is more often:</p>
<ol>
<li>decide whether the behavior is public, private, or inventory</li>
<li>decide whether it is a service, aspect, profile, shared helper, package, or patch</li>
<li>make the host opt into it</li>
</ol>
<p>That is a more structured workflow. It creates a little more friction up front, but it produces cleaner boundaries over time.</p>
<h2 id="easier-question-what-does-this-host-actually-do">Easier question: what does this host actually do?</h2>
<p>One of the most noticeable improvements is that host files got thinner.</p>
<p>A host like <code>goose</code> now reads more like a statement of composition:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">imports = [
modules.public.nixos.aspects.serverBase
(import modules.private.nixos.aspects.managedRemoteHost {
host = "goose";
sopsFile = ../../secrets/goose.yaml;
})
modules.private.nixos.aspects.gooseServices
modules.public.nixos.services.smsGatewayClient
(import modules.public.nixos.services.wanFailover { ... })
./hardware-configuration.nix
./networking.nix
];
</code></pre>
<p>That is a nicer top-level read than a long host file that mixes identity, hardware, service definitions, package choices, firewall details, and helper logic all in one place.</p>
<p>You can look at the host and quickly understand its shape:</p>
<ul>
<li>base server behavior</li>
<li>remote-host management behavior</li>
<li>host-specific service bundle</li>
<li>explicit shared services</li>
<li>hardware and networking</li>
</ul>
<p>That is a good trade. The host becomes easier to scan because implementation has moved somewhere named.</p>
<h2 id="easier-question-how-do-i-set-up-a-new-host-now">Easier question: how do I set up a new host now?</h2>
<p>This is where the new layout feels most obviously different.</p>
<p>Before, adding a new host often meant copying an existing host, then editing it until it fit. You could do that fast, but it encouraged silent inheritance by copy-paste. The host would work, but the structure would not necessarily tell you what was reusable and what was accidental.</p>
<p>Now the setup path is more deliberate.</p>
<p>At a high level, adding a host means:</p>
<ol>
<li>create the host directory</li>
<li>decide which reusable aspects apply</li>
<li>decide what inventory entries it needs</li>
<li>add only the genuinely host-local configuration in the host file</li>
</ol>
<p>That feels a bit more like architecture and a bit less like cloning a previous machine.</p>
<p>For example, a new server host does not need to rediscover how to be “a server” from scratch. It can import a named base aspect. A workstation does not need to inline shared Nix CLI defaults, shared shell behavior, or local flake deployment setup if those already exist as named pieces.</p>
<p>This is the part I like most: new hosts are less about copying a specimen and more about selecting building blocks.</p>
<h2 id="over-time-some-things-become-much-easier-to-understand">Over time, some things become much easier to understand</h2>
<p>This is the main payoff.</p>
<p>When the repository grows, understanding is less about reading every host and more about understanding the vocabulary of the repo:</p>
<ul>
<li>what an aspect means</li>
<li>what belongs in a service module versus a profile</li>
<li>where inventory lives</li>
<li>which pieces are intended to be shared across NixOS, Darwin, Home Manager, or packages</li>
</ul>
<p>Once that vocabulary stabilizes, the codebase gets easier to navigate because concepts have homes.</p>
<p>You stop finding the same kind of logic in three or four different places under slightly different names. You stop wondering whether a host-local tweak is actually the canonical implementation of a broader idea. You stop treating the <code>flake.nix</code> file as a manual import spreadsheet.</p>
<p>That last point matters more than it sounds. A smaller, more manifest-like <code>flake.nix</code> is not just aesthetically nicer. It reduces import churn. It reduces the feeling that every new module requires touching central wiring by hand. And it helps the flake describe the repo instead of micromanaging it.</p>
<h2 id="cross-platform-sharing-got-cleaner-too">Cross-platform sharing got cleaner too</h2>
<p>One reason the old structure was getting strained is that the repo is not just NixOS hosts.</p>
<p>It spans:</p>
<ul>
<li>NixOS</li>
<li>nix-darwin</li>
<li>Home Manager</li>
<li>helper libraries</li>
<li>custom packages</li>
<li>patches</li>
</ul>
<p>Those are different configuration classes, but they often want to express the same <em>feature-level</em> concern. A workstation baseline is not only a NixOS thing. A deploy helper is not only a host concern. Shared development ergonomics may affect Home Manager and system modules together.</p>
<p>The Dendritic layout gives those things more coherent homes. You can keep related ideas near each other even when they cut across configuration classes.</p>
<p>That does not eliminate complexity, but it makes the complexity easier to name.</p>
<h2 id="the-tradeoff-you-buy-clarity-with-more-structure">The tradeoff: you buy clarity with more structure</h2>
<p>This is where the cost shows up.</p>
<p>The new layout is clearer in the large, but it is not automatically simpler in the small.</p>
<p>Sometimes the old answer to “where do I put this?” was just “put it in the host.” That is crude, but it is low-friction. The new answer may involve:</p>
<ul>
<li>creating a new aspect</li>
<li>giving it a good name</li>
<li>deciding whether it is public or private</li>
<li>importing it in the right place</li>
<li>possibly adding inventory support</li>
</ul>
<p>That is more boilerplate.</p>
<p>Not absurd boilerplate, but real boilerplate. You are making more small files. You are adding more named composition points. You are spending more effort on repo shape.</p>
<p>For a repo at the right scale, that cost is worth paying. For a smaller repo, it can feel like architecture cosplay.</p>
<h2 id="indirection-is-the-other-real-cost">Indirection is the other real cost</h2>
<p>Thin hosts are nice to read, but they move behavior elsewhere. That means tracing behavior can become harder.</p>
<p>If you are new to the repo and you open a host file, you might see clean imports and named aspects, but not the full behavior. To understand what the machine actually does, you have to follow the graph:</p>
<ul>
<li>from host</li>
<li>to aspect</li>
<li>to service tree</li>
<li>to shared helpers</li>
<li>maybe to inventory data</li>
</ul>
<p>That is a better model once you know the repo. It is not always a better first impression.</p>
<p>So the trade is basically this:</p>
<ul>
<li>host-centric layout gives you more immediate locality</li>
<li>dendritic layout gives you better long-term semantic organization</li>
</ul>
<p>There is no free lunch there.</p>
<h2 id="naming-becomes-architecture">Naming becomes architecture</h2>
<p>This is one of the more interesting side effects.</p>
<p>When the repo is built around aspects and module trees, names matter more. A weak name creates confusion far beyond one file. A good name becomes a durable concept that helps the rest of the repo make sense.</p>
<p>That means the refactor shifts some of the burden from file placement to naming quality.</p>
<p>If an aspect is called something vague like <code>common</code> or <code>defaults</code>, it does not help much. If it is named for a clear responsibility like <code>serverBase</code>, <code>managedRemoteHost</code>, or <code>workstationSecrets</code>, then the host composition starts reading like intent instead of implementation trivia.</p>
<p>This is good architecture pressure, but it is still pressure. You have to keep the vocabulary disciplined.</p>
<h2 id="newcomers-may-feel-more-friction">Newcomers may feel more friction</h2>
<p>This structure is not as immediately obvious as “there are some hosts and a configs directory.”</p>
<p>A newcomer now has to learn:</p>
<ul>
<li>the community/private/inventory split</li>
<li>what counts as an aspect</li>
<li>where shared behavior is expected to live</li>
<li>which modules are meant to be reused and which are just internal glue</li>
</ul>
<p>That is a steeper conceptual ramp.</p>
<p>The payoff is that once they learn it, the repo is more coherent. But the up-front learning cost is real, and I would not pretend otherwise.</p>
<p>Auto-imported module trees make this even more true. They keep the flake smaller and reduce manual wiring, which I like, but they also make the repo feel a little more magical. Explicit imports are noisier, but they are easier to explain in one sentence.</p>
<p>Again: tradeoff, not free win.</p>
<h2 id="verification-matters-more-during-refactors-like-this">Verification matters more during refactors like this</h2>
<p>Even though I am not focusing this post on the migration itself, one lesson from the refactor is worth calling out because it connects directly to the structure question.</p>
<p>This style of reorganization makes semantic equivalence easy to <em>intend</em> and easy to get slightly wrong.</p>
<p>Most of the repo kept the same normalized configuration shape after the rewrite, which is exactly what I wanted. But careful comparison still caught a couple of genuine drifts:</p>
<ul>
<li><code>ij-desktop</code> briefly lost <code>nix-command flakes</code></li>
<li>shared node-exporter factoring briefly implied port <code>9100</code> in a way that needed tightening back up</li>
</ul>
<p>Those are not catastrophic bugs. They are exactly the kind of subtle drift you get when you move behavior into reusable layers.</p>
<p>That does not argue against the layout. It argues for stronger verification when you reorganize into this style. The more you rely on reusable composition, the more you need confidence that the composition still evaluates to what you think it does.</p>
<h2 id="so-was-it-worth-it">So was it worth it?</h2>
<p>For this repo, yes.</p>
<p>Not because Dendritic is universally superior. Not because host-centric repos are wrong. Not because more module trees automatically mean better architecture.</p>
<p>It was worth it because this flake had already crossed the point where host-local logic, private inventory, reusable modules, public exports, and cross-platform sharing were all competing inside one old layout. The repo wanted stronger boundaries, and the Dendritic split gave it those.</p>
<p>What I gained was mostly clarity:</p>
<ul>
<li>clearer separation between public, private, and inventory concerns</li>
<li>thinner hosts that read more like manifests</li>
<li>easier reuse across NixOS, Darwin, Home Manager, packages, patches, and helpers</li>
<li>better path semantics and less junk-drawer ambiguity</li>
<li>a smaller, more declarative <code>flake.nix</code></li>
</ul>
<p>What I paid was also real:</p>
<ul>
<li>more conceptual overhead</li>
<li>more indirection</li>
<li>more naming burden</li>
<li>more boilerplate around introducing new concepts cleanly</li>
</ul>
<p>That feels like the honest summary.</p>
<p>If your repo is still basically one or two machines with a few shared modules, I would not rush into this. A straightforward host-centric tree may still be the right tool. But if your flake is starting to feel like a mixed pile of hosts, helpers, secrets, service trees, and cross-platform behavior, then reorganizing around clearer module layers can buy back a lot of understanding.</p>
<p>Just do not mistake that for simplicity.</p>
<p>It is a trade from one kind of complexity to another. In this case, I think it was the right trade.</p>
Legacy Deployment with Nix Flake Apps and systemd User Services2026-04-04T00:07:26+00:002026-04-04T00:07:26+00:00
Ian Johannesen
https://perlpimp.net/blog/legacy-deployment-nix-flake-apps-systemd-user-services/<p>You have a nice NixOS module for production. It builds the service, wires up secrets, configures systemd, maybe nginx too. Then reality intrudes: one box is still Debian, another is some inherited Ubuntu VM, and you still need to deploy the same service there without inventing a second operational universe.</p>
<p>The usual answer is “use Ansible,” or Docker, or a pile of shell scripts in <code>~/bin</code> that quietly turn into a homegrown deployment system. But if the service already lives in a flake, you can keep the operational logic there too.</p>
<p>The pattern is simple: use flake <code>apps</code> outputs as your legacy deployment interface. <code>nix run .#setup</code>, <code>nix run .#install</code>, <code>nix run .#build-legacy</code>, <code>nix run .#dump-db</code> and so on. Same repo, same package graph, same binary. NixOS hosts use the module. Non-NixOS hosts use the apps.</p>
<p>This gives you one source of truth for build inputs and a second, lighter orchestration path for machines that are not ready for full NixOS.</p>
<h2 id="the-goal">The goal</h2>
<p>You want the same flake to support two deployment modes:</p>
<ul>
<li><strong>NixOS</strong>: declarative module, system service, hardened options, proper secret management</li>
<li><strong>Legacy Linux</strong>: unprivileged user account, <code>systemd --user</code>, environment files, wrapper scripts, and one-command deploys</li>
</ul>
<p>Not identical orchestration. Identical artifact.</p>
<p>That distinction matters. You do <strong>not</strong> want two build systems. You want one build and two ways to run it.</p>
<h2 id="first-make-the-deploy-user-a-trusted-nix-user">First: make the deploy user a trusted Nix user</h2>
<p>Before anything else, the user running <code>nix build</code> on the deploy target needs permission to pass Nix settings to the daemon. This is especially important if your flake has private GitHub inputs.</p>
<p>On the target host:</p>
<pre data-lang="ini" class="language-ini "><code class="language-ini" data-lang="ini"># /etc/nix/nix.conf
trusted-users = root deploy-user
</code></pre>
<p>Then restart the daemon:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">sudo systemctl restart nix-daemon
</code></pre>
<p>Without this, the deploy user can invoke Nix, but the daemon may ignore user-supplied settings such as access tokens or custom substituters. That turns into mysterious failures when a private flake input suddenly looks “missing.”</p>
<p>If your flake pulls from private GitHub repos, add an access token in the deploy user’s config:</p>
<pre data-lang="ini" class="language-ini "><code class="language-ini" data-lang="ini"># ~/.config/nix/nix.conf
access-tokens = github.com=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
</code></pre>
<p>Now <code>nix build</code> on that host can fetch private <code>github:</code> inputs directly. On NixOS you’d normally do this declaratively and inject the token from sops-nix or agenix. On a legacy host, this is usually a one-time machine bootstrap step.</p>
<h2 id="shared-paths-in-the-flake">Shared paths in the flake</h2>
<p>The apps all share the same path conventions. Put those once near the top of your <code>perSystem</code>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">let
projectDir = "$HOME/git/my-service";
binDir = "$HOME/bin/my-service";
aliasPrefix = "my_service";
loadEnv = ''
cd "${projectDir}"
if [ -f .env.production ]; then
set -a; source .env.production; set +a
elif [ -f .env.local ]; then
set -a; source .env.local; set +a
fi
'';
in
</code></pre>
<p>This is the first useful trick: treat environment loading as a reusable shell fragment, not something each script reimplements badly.</p>
<p><code>.env.production</code> wins over <code>.env.local</code>. That gives you one convention for deploy targets and another for local testing. <code>set -a</code> means everything sourced gets exported automatically, so your wrapper scripts and systemd unit see the same variables.</p>
<h2 id="app-1-setup-for-first-time-validation">App 1: <code>setup</code> for first-time validation</h2>
<p>Your first deploy should not start with “run a giant script and hope.” Give yourself a small setup app that creates directories and validates the build output:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">apps.setup = flake-utils.lib.mkApp {
drv = pkgs.writeShellApplication {
name = "setup";
text = ''
${loadEnv}
echo "Creating data directory..."
mkdir -p "${projectDir}/.data"
if [ ! -L "${projectDir}/result" ]; then
echo "Run 'nix build' first, then re-run setup."
exit 1
fi
echo "Validating configuration..."
"${projectDir}/result/bin/my-service" --check-config
echo "Setup complete."
'';
};
};
</code></pre>
<p>That <code>--check-config</code> flag is optional. The point is to let the service validate itself before you wire it into systemd. If your app doesn’t have a config check command, replace it with whatever proves the environment is sane: a database ping, a migration status command, a dry-run boot.</p>
<h2 id="app-2-build-legacy-as-the-deploy-button">App 2: <code>build-legacy</code> as the deploy button</h2>
<p>This is the core of the pattern. You want a single command that stops the old service, builds the current flake, writes the systemd user unit, reloads it, and starts the service again.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">apps.build-legacy = flake-utils.lib.mkApp {
drv = pkgs.writeShellApplication {
name = "build-legacy";
runtimeInputs = with pkgs; [ systemd ];
text = ''
set -euo pipefail
${loadEnv}
echo "Stopping service..."
systemctl --user stop my-service || true
echo "Building..."
nix build
echo "Writing systemd unit..."
mkdir -p "$HOME/.config/systemd/user"
cat > "$HOME/.config/systemd/user/my-service.service" <<UNIT
[Unit]
Description=My Service Daemon
After=network.target
[Service]
Type=exec
ExecStart=${projectDir}/result/bin/my-service
Environment=SERVICE_PORT=''${SERVICE_PORT:-8080}
Environment=SERVICE_BIND_ADDRESS=''${SERVICE_BIND_ADDRESS:-127.0.0.1}
Environment=SERVICE_DB_PATH=''${SERVICE_DB_PATH:-${projectDir}/.data/my-service.db}
EnvironmentFile=-${projectDir}/.env.production
Restart=on-failure
RestartSec=5
[Install]
WantedBy=default.target
UNIT
systemctl --user daemon-reload
systemctl --user enable --now my-service
echo "Service started."
'';
};
};
</code></pre>
<p>A few design choices here are doing real work:</p>
<ul>
<li><code>systemctl --user</code> keeps the whole thing unprivileged. No root-owned unit, no <code>sudo systemctl restart</code>, no system-level service management for a simple single-user deploy.</li>
<li><code>systemctl --user stop ... || true</code> makes first deploy idempotent. There might not be anything running yet.</li>
<li><code>ExecStart=${projectDir}/result/bin/my-service</code> points at the current <code>nix build</code> symlink, so each deploy naturally flips the unit to the newest build result.</li>
<li><code>Environment=</code> lines define safe defaults, while <code>EnvironmentFile=-...</code> lets <code>.env.production</code> override them at runtime.</li>
<li>The <code>-</code> on <code>EnvironmentFile</code> is important. It tells systemd “missing file is fine.”</li>
</ul>
<p>That last point keeps first boot and bootstrap paths simple. Your unit should not explode just because the env file is absent.</p>
<h2 id="why-generate-the-unit-from-nix-instead-of-committing-it">Why generate the unit from Nix instead of committing it</h2>
<p>Because the unit is part of the deployment interface, and the deployment interface belongs next to the binary definition.</p>
<p>If you commit <code>deploy/my-service.service</code> separately, it will drift. Someone tweaks the binary path, or adds a new env var, or changes restart behavior, and now the Nix package and the checked-in unit file disagree. Generating the unit inside <code>writeShellApplication</code> keeps the operational wrapper versioned with the build logic.</p>
<p>It also makes the unit templated by construction. Paths, ports, binary names, env conventions: they all come from the same Nix values used elsewhere in the flake.</p>
<h2 id="app-3-install-for-management-scripts-and-aliases">App 3: <code>install</code> for management scripts and aliases</h2>
<p>Once the service exists, you want a decent operator experience. Not “remember six long commands.” Real commands in <code>~/bin/my-service</code>, plus shell aliases for fish, zsh, and bash.</p>
<p>The basic shape looks like this:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">apps.install = flake-utils.lib.mkApp {
drv = pkgs.writeShellApplication {
name = "install";
text = ''
set -euo pipefail
INSTALL_DIR="${binDir}"
mkdir -p "$INSTALL_DIR"
cat > "$INSTALL_DIR/build" <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
cd "${projectDir}"
git pull --ff-only
nix run .#build-legacy
EOF
chmod +x "$INSTALL_DIR/build"
cat > "$INSTALL_DIR/tail-log" <<'EOF'
#!/usr/bin/env bash
exec journalctl --user -u my-service -f
EOF
chmod +x "$INSTALL_DIR/tail-log"
cat > "$INSTALL_DIR/bash_setup" <<EOF
echo "alias ${aliasPrefix}_build='${binDir}/build'"
echo "alias ${aliasPrefix}_tail_log='${binDir}/tail-log'"
EOF
cat > "$INSTALL_DIR/fish_setup" <<EOF
echo "alias ${aliasPrefix}_build '${binDir}/build'"
echo "alias ${aliasPrefix}_tail_log '${binDir}/tail-log'"
EOF
'';
};
};
</code></pre>
<p>In a real flake you’d factor that more cleanly, probably with a small helper that writes wrapped scripts and another that emits shell-specific alias syntax. The important part is the architecture:</p>
<ul>
<li><code>install</code> creates a stable operator-facing command set in <code>~/bin/my-service/</code></li>
<li>each command is a tiny wrapper around one task</li>
<li>shell integration is generated, not handwritten in three places</li>
</ul>
<p>That generated alias layer is worth more than it looks. It means the deploy user logs in and has <code>my_service_build</code>, <code>my_service_tail_log</code>, <code>my_service_dump_db</code>, <code>my_service_shell</code>, whatever else you provide, with no shell-specific maintenance burden.</p>
<h2 id="app-4-database-dumps-as-part-of-the-flake">App 4: database dumps as part of the flake</h2>
<p>If your “legacy deployment workflow” does not include a backup command, it is not a deployment workflow. It is an optimism framework.</p>
<p>Make a <code>dump-db</code> app and pin the toolchain with Nix just like everything else.</p>
<h3 id="sqlite">SQLite</h3>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">apps.dump-db = flake-utils.lib.mkApp {
drv = pkgs.writeShellApplication {
name = "dump-db";
runtimeInputs = with pkgs; [ sqlite bzip2 ];
text = ''
set -euo pipefail
${loadEnv}
DB_PATH="''${SERVICE_DB_PATH:-${projectDir}/.data/my-service.db}"
OUTPUT="db-export-$(date +%Y%m%d_%H%M%S).sql.bz2"
sqlite3 "$DB_PATH" .dump | bzip2 > "$OUTPUT"
echo "Exported to $OUTPUT"
'';
};
};
</code></pre>
<h3 id="postgresql">PostgreSQL</h3>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">runtimeInputs = with pkgs; [ postgresql bzip2 ];
text = ''
set -euo pipefail
${loadEnv}
OUTPUT="db-export-$(date +%Y%m%d_%H%M%S).sql.bz2"
pg_dump \
--host="''${DB_HOST:-localhost}" \
--port="''${DB_PORT:-5432}" \
--username="''${DB_USER}" \
--dbname="''${DB_NAME}" \
--no-owner --no-acl \
| bzip2 > "$OUTPUT"
'';
</code></pre>
<h3 id="mysql-or-mariadb">MySQL or MariaDB</h3>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">runtimeInputs = with pkgs; [ mariadb bzip2 ];
text = ''
set -euo pipefail
${loadEnv}
OUTPUT="db-export-$(date +%Y%m%d_%H%M%S).sql.bz2"
mysqldump \
--host="''${DB_HOST:-localhost}" \
--port="''${DB_PORT:-3306}" \
--user="''${DB_USER}" \
--single-transaction \
--routines \
"''${DB_NAME}" \
| bzip2 > "$OUTPUT"
'';
</code></pre>
<p>This is exactly the kind of thing people leave to tribal knowledge: “oh, just remember the right <code>pg_dump</code> flags.” Put it in the flake instead. Then your deploy workflow, your operator docs, and your actual tooling all agree.</p>
<h2 id="app-5-repl-access-for-the-running-service">App 5: REPL access for the running service</h2>
<p>The same pattern works for interactive operational tools.</p>
<p>For an Elixir release:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">replScript = mkScript "iex" ''
${loadEnv}
exec "${projectDir}/_build/prod/rel/my_app/bin/my_app" remote
'';
</code></pre>
<p>That connects an IEx shell to the running BEAM node. Same idea for Rails:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">replScript = mkScript "console" ''
${loadEnv}
cd "${projectDir}"
exec bundle exec rails console -e production
'';
</code></pre>
<p>Or Django:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">replScript = mkScript "shell" ''
${loadEnv}
cd "${projectDir}"
exec python manage.py shell
'';
</code></pre>
<p>The trick is not language-specific. Load the environment, move into the right directory, run the service-specific console command. Then wire that script into <code>install</code> and export an alias for it like any other operational command.</p>
<h2 id="the-motd-banner-is-worth-it">The MOTD banner is worth it</h2>
<p>If the deploy target has a dedicated service user, make login useful. A small generated MOTD script gives you a dashboard instead of a blank prompt:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">motdScript = mkScript "motd" ''
printf '\n'
printf ' \033[1;36m%s\033[0m\n' "My Service"
printf '\n'
printf ' Location: %s\n' "${projectDir}"
printf ' Environment file: %s/.env.production\n' "${projectDir}"
printf '\n'
printf ' \033[1mAliases:\033[0m\n'
printf ' %-28s %s\n' "${aliasPrefix}_build" "stop, pull, build, restart"
printf ' %-28s %s\n' "${aliasPrefix}_tail_log" "stream service logs"
printf ' %-28s %s\n' "${aliasPrefix}_dump_db" "export database backup"
printf ' %-28s %s\n' "${aliasPrefix}_db_shell" "open database shell"
printf ' %-28s %s\n' "${aliasPrefix}_iex" "attach remote console"
printf '\n'
'';
</code></pre>
<p>Then call it from the service user’s shell init after loading the alias file:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">eval "$(~/bin/my-service/bash_setup)"
~/bin/my-service/motd
</code></pre>
<p>Or in fish:</p>
<pre data-lang="fish" class="language-fish "><code class="language-fish" data-lang="fish">eval (~/bin/my-service/fish_setup)
~/bin/my-service/motd
</code></pre>
<p>This is not just cosmetic. The banner becomes a tiny operational contract: what service this account owns, where it lives, and which commands matter.</p>
<h2 id="the-environment-loading-pattern">The environment loading pattern</h2>
<p>The env-loading fragment from earlier is doing two jobs:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">if [ -f .env.production ]; then
set -a; source .env.production; set +a
elif [ -f .env.local ]; then
set -a; source .env.local; set +a
fi
</code></pre>
<p>First, it gives production and local environments a deterministic priority order.</p>
<p>Second, it keeps every app consistent. <code>setup</code>, <code>build-legacy</code>, <code>dump-db</code>, <code>db-shell</code>, <code>iex</code>, any future admin script: they all see the same environment variables from the same place.</p>
<p>That consistency is the real benefit. Once you have six or eight wrapper commands, the fastest way to create weird behavior is to let each one load config differently.</p>
<h2 id="the-contrast-with-nixos">The contrast with NixOS</h2>
<p>At this point the pattern should be clear: the legacy path is not a compromise build. It is a compromise orchestration model.</p>
<p>Here is the practical comparison:</p>
<table><thead><tr><th>Aspect</th><th>Legacy host via flake apps</th><th>NixOS host via module</th></tr></thead><tbody>
<tr><td>Service manager</td><td><code>systemd --user</code></td><td>system-level <code>systemd</code></td></tr>
<tr><td>Privileges</td><td>unprivileged service user</td><td>hardened service user / <code>DynamicUser</code></td></tr>
<tr><td>Secrets</td><td><code>.env.production</code></td><td>sops-nix / agenix / declarative injection</td></tr>
<tr><td>Updates</td><td><code>nix run .#build-legacy</code></td><td><code>nixos-rebuild switch</code></td></tr>
<tr><td>Reverse proxy</td><td>manual nginx or caddy config</td><td>declarative module options</td></tr>
<tr><td>Backups</td><td><code>nix run .#dump-db</code></td><td>timer or service declared in NixOS</td></tr>
<tr><td>Monitoring</td><td>manual integration</td><td>declarative Prometheus/Grafana wiring</td></tr>
</tbody></table>
<p>That is exactly the right relationship. NixOS should be the better deployment target. It gives you stronger isolation, stronger secret handling, stronger service definitions. But the legacy path still uses the same flake, same package graph, same env surface, and same operational commands.</p>
<p>That makes migration gradual instead of traumatic.</p>
<h2 id="daily-workflow-after-install">Daily workflow after <code>install</code></h2>
<p>Once you have installed the wrappers and sourced the aliases, operating the service should look boring:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">my_service_build
my_service_tail_log
my_service_dump_db
my_service_db_shell
my_service_iex
my_service_shell
</code></pre>
<p>That is the whole point. If the workflow on a non-NixOS box still feels bespoke, you have not pushed enough of it into the flake yet.</p>
<h2 id="why-this-pattern-holds-up">Why this pattern holds up</h2>
<p>The nice part of this approach is not that it is “fully declarative.” It isn’t. A legacy host is still a legacy host. You are still relying on a user account, a checked-out repo, a shell profile, and some one-time machine bootstrap.</p>
<p>The nice part is narrower and more useful:</p>
<ul>
<li>your deploy commands are versioned with the code</li>
<li>your build inputs stay pinned</li>
<li>your operational wrappers stay reproducible</li>
<li>your non-NixOS path does not fork into another toolchain</li>
</ul>
<p>You stop treating “legacy deployment” as a totally separate system. It becomes another interface on the same flake.</p>
<p>If you eventually migrate the box to NixOS, great. The service definition gets better, but the build and runtime assumptions stay familiar. If you do not migrate it yet, you still have something coherent: one repo, one flake, one set of commands, two orchestration targets.</p>
<p>That is usually enough to keep the infrastructure sane.</p>
Monitoring gpsd with Prometheus using prometheus-gpsd-exporter2026-04-03T00:04:00+00:002026-04-03T00:04:00+00:00
Ian Johannesen
https://perlpimp.net/blog/prometheus-gpsd-exporter-nixos/<p>GPS receivers spit out NMEA, gpsd parses it, and getting that data into Prometheus shouldn’t require babysitting a crashy exporter. Yet here we are.</p>
<p>The existing options for gpsd-to-Prometheus are a <a rel="noopener noreferrer" target="_blank" href="https://github.com/brendanbank/gpsd-prometheus-exporter">Python exporter</a> and a <a rel="noopener noreferrer" target="_blank" href="https://github.com/brendanbank/gpsd-prometheus-exporter/tree/master">Go exporter</a>. Both have the same fundamental problem: they don’t stay running. The Python one relies on the <code>gps</code> library, which throws <code>StopIteration</code> when the connection hiccups — unhandled, fatal, exporter dead. The Go one polls on a 10-second interval instead of streaming, calls <code>log.Fatal</code> on parse errors, has no reconnect logic, and uses different metric names from the Python exporter — so your Grafana dashboards break if you switch between them.</p>
<p>You end up restarting gpsd, restarting the exporter, and wondering why your satellite count graph has gaps every few hours.</p>
<h2 id="a-rust-rewrite-that-stays-up">A Rust rewrite that stays up</h2>
<p><a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/prometheus-gpsd-exporter">prometheus-gpsd-exporter</a> replaces both. It’s a single async Rust binary — Tokio runtime, one persistent TCP connection to gpsd with streaming JSON, one HTTP server on <code>/metrics</code>. The architecture is simple:</p>
<pre><code>gpsd (host A:2947) --TCP/JSON--> exporter (host B) --HTTP/metrics--> Prometheus
</code></pre>
<p>Direct TCP connection, no fragile library dependency. It reads gpsd’s JSON stream, parses every message type that matters — TPV, SKY, GST, TOFF, PPS, OSC — and exposes them as Prometheus metrics. Unknown or malformed messages get logged and skipped. Never crashes.</p>
<p>When the connection drops, it reconnects with exponential backoff — starting at 10 seconds, capping at 5 minutes. No manual intervention, no systemd restart loops, no gaps in your graphs.</p>
<h2 id="getting-gpsd-to-cooperate">Getting gpsd to cooperate</h2>
<p>A solid exporter is half the battle. The other half is getting gpsd to actually emit the data you want. Two things trip people up here: baud rate and network listening.</p>
<h3 id="the-baud-rate-problem">The baud rate problem</h3>
<p>gpsd truncates SKY messages and other verbose output unless you tell it the correct baud rate for your receiver. Most modern GPS modules run at 115200 baud. Without the <code>-s</code> flag, gpsd defaults to 9600 and silently drops data — you get position fixes but no satellite details, no DOP values, no signal strength per satellite.</p>
<p><strong>Traditional Linux (<code>/etc/default/gpsd</code>):</strong></p>
<pre><code>GPSD_OPTIONS="-G -s 115200"
</code></pre>
<p><strong>NixOS:</strong></p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.gpsd = {
enable = true;
devices = [ "/dev/ttyUSB0" ];
listenany = true; # equivalent of -G
extraArgs = [ "-s" "115200" ];
};
</code></pre>
<p>The <code>-G</code> flag (or <code>listenany</code> in NixOS) tells gpsd to listen on all interfaces instead of just localhost. You’ll need this if the exporter runs on a different host than the GPS receiver.</p>
<h3 id="making-gpsd-listen-on-the-network">Making gpsd listen on the network</h3>
<p>On traditional Linux distributions, gpsd uses systemd socket activation. The default socket only listens on localhost. To expose gpsd over the network, you need to override the socket unit.</p>
<p><strong>Traditional Linux (<code>systemctl edit gpsd.socket</code>):</strong></p>
<pre data-lang="ini" class="language-ini "><code class="language-ini" data-lang="ini">[Socket]
ListenStream=
ListenStream=/var/run/gpsd.sock
ListenStream=[::]:2947
ListenStream=0.0.0.0:2947
</code></pre>
<p>The empty <code>ListenStream=</code> clears the defaults first — without it, you’d be adding listeners on top of the existing ones. Then the subsequent lines add back the Unix socket plus all-interface TCP listeners. After editing, restart both units:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">sudo systemctl restart gpsd.socket gpsd.service
</code></pre>
<p><strong>NixOS:</strong></p>
<p>The NixOS gpsd module doesn’t use socket activation — it runs a forking service directly. The <code>listenany = true</code> option handles network listening. If you need socket activation with both Unix and TCP sockets for some reason, define it explicitly:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">systemd.sockets.gpsd = {
description = "GPS Daemon Sockets";
wantedBy = [ "sockets.target" ];
socketConfig = {
ListenStream = [
"/run/gpsd.sock"
"[::]:2947"
"0.0.0.0:2947"
];
SocketMode = "0600";
};
};
</code></pre>
<p>Most NixOS setups won’t need this — <code>listenany = true</code> is sufficient when the exporter and gpsd are on the same host or when you just need TCP access.</p>
<h2 id="setting-up-the-exporter">Setting up the exporter</h2>
<h3 id="as-a-nix-flake">As a Nix flake</h3>
<p>Add the exporter as a flake input and import the NixOS module:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
prometheus-gpsd-exporter.url = "github:ijohanne/prometheus-gpsd-exporter";
};
outputs = { nixpkgs, prometheus-gpsd-exporter, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
modules = [
prometheus-gpsd-exporter.nixosModules.default
./configuration.nix
];
};
};
}
</code></pre>
<h3 id="nixos-module-configuration">NixOS module configuration</h3>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.prometheus-gpsd-exporter = {
enable = true;
enableLocalScraping = true;
gpsdHost = "192.168.1.100";
ppsHistogram = true;
offsetFromGeopoint = true;
geopointLat = 51.5074;
geopointLon = -0.1278;
};
</code></pre>
<p><code>enableLocalScraping</code> adds a Prometheus scrape target automatically — no manual <code>scrapeConfigs</code> editing. <code>gpsdHost</code> points at whichever machine runs your GPS receiver. The geopoint options enable geo-offset histograms — set them to your receiver’s actual coordinates. You can read the initial values from the <code>gpsd_lat</code> and <code>gpsd_long</code> metrics once the exporter is running, then pin them in your config.</p>
<h3 id="standalone-binary">Standalone binary</h3>
<p>Not on NixOS? The binary works anywhere:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">prometheus-gpsd-exporter \
--hostname <gpsd-host> \
--port 2947 \
--exporter-port 9015 \
--listen-address :: \
--pps-histogram
</code></pre>
<h2 id="cli-reference">CLI reference</h2>
<pre><code>-H, --hostname <HOST> gpsd host [default: localhost]
-p, --port <PORT> gpsd port [default: 2947]
-E, --exporter-port <PORT> metrics port [default: 9015]
-L, --listen-address <ADDR> listen address [default: ::]
-t, --timeout <SECS> connection timeout [default: 10]
--retry-delay <SECS> initial retry delay [default: 10]
--max-retry-delay <SECS> max retry delay [default: 300]
-S, --disable-monitor-satellites
--pps-histogram enable PPS offset histograms
--pps-bucket-size <NS> [default: 250]
--pps-bucket-count <N> [default: 40]
--pps-time1 <FLOAT> PPS time1 offset [default: 0]
--offset-from-geopoint enable geo-offset tracking
--geopoint-lat <FLOAT>
--geopoint-lon <FLOAT>
--geo-bucket-size <M> [default: 0.5]
--geo-bucket-count <N> [default: 40]
</code></pre>
<p>The PPS options are for precision timing setups — if you’re running an NTP server disciplined by a GPS PPS signal, the histograms show pulse offset distribution. The geo-offset options track how far your reported position drifts from a known fixed point, which is useful for evaluating antenna placement or receiver quality.</p>
<h2 id="what-metrics-you-get">What metrics you get</h2>
<p>The exporter covers every gpsd message type that carries useful data:</p>
<ul>
<li><strong>TPV</strong> — latitude, longitude, altitude, speed, track, climb, and all the error estimates (epx, epy, epv, ept, eps, epc). The core positioning data.</li>
<li><strong>SKY</strong> — DOP values (gdop, hdop, pdop, tdop, vdop, xdop, ydop) and satellite counts. High DOP means bad geometry — you want these low.</li>
<li><strong>Per-satellite</strong> — signal strength, azimuth, elevation, and health status. Labeled by PRN, svid, gnssid, sigid, freqid, and whether the satellite is used in the fix. This is the data that tells you if your antenna has a clear sky view.</li>
<li><strong>GST</strong> — pseudorange noise and error ellipse data. The statistical quality of your position solution.</li>
<li><strong>TOFF</strong> — kernel-vs-GPS time offset. Essential if you’re running NTP with a GPS reference clock.</li>
<li><strong>PPS</strong> — pulse-per-second timing data, with optional histograms for offset distribution.</li>
<li><strong>OSC</strong> — oscillator status: running, reference, disciplined, delta. For GPSDO setups.</li>
<li><strong>Info</strong> — gpsd version and connected device metadata.</li>
</ul>
<p>Metric names are compatible with the Python exporter (<code>gpsd_lat</code>, <code>gpsd_hdop</code>, <code>gpsd_satellites_used</code>, etc.), so existing Grafana dashboards work without changes. Switching from the Python exporter is a drop-in replacement.</p>
<h2 id="why-this-over-the-alternatives">Why this over the alternatives</h2>
<table><thead><tr><th></th><th>Python [1]</th><th>Go [2]</th><th>Rust [3]</th></tr></thead><tbody>
<tr><td>Crash resilient</td><td>no</td><td>no</td><td><strong>yes</strong></td></tr>
<tr><td>Streaming</td><td>yes</td><td>no</td><td><strong>yes</strong></td></tr>
<tr><td>Reconnect</td><td>buggy</td><td>no</td><td><strong>yes</strong></td></tr>
<tr><td>No runtime deps</td><td>no</td><td>yes</td><td><strong>yes</strong></td></tr>
<tr><td>Dashboard compat</td><td>yes</td><td>no</td><td><strong>yes</strong></td></tr>
<tr><td>GST</td><td>no</td><td>yes</td><td><strong>yes</strong></td></tr>
<tr><td>TOFF</td><td>no</td><td>yes</td><td><strong>yes</strong></td></tr>
<tr><td>OSC</td><td>no</td><td>yes</td><td><strong>yes</strong></td></tr>
<tr><td>PPS histograms</td><td>yes</td><td>no</td><td><strong>yes</strong></td></tr>
<tr><td>Geo-offset</td><td>yes</td><td>no</td><td><strong>yes</strong></td></tr>
<tr><td>Per-sat labels</td><td>yes</td><td>partial</td><td><strong>yes</strong></td></tr>
<tr><td>Per-sat sigid</td><td>no</td><td>yes</td><td><strong>yes</strong></td></tr>
</tbody></table>
<p>[1] <a rel="noopener noreferrer" target="_blank" href="https://github.com/brendanbank/gpsd-prometheus-exporter">brendanbank/gpsd-prometheus-exporter</a><br>
[2] <a rel="noopener noreferrer" target="_blank" href="https://github.com/brendanbank/gpsd-prometheus-exporter/tree/master">natesales/gpsd-exporter</a><br>
[3] <a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/prometheus-gpsd-exporter">ijohanne/prometheus-gpsd-exporter</a></p>
<p>The Python exporter is the one most people start with. It has the right metric names and decent coverage, but the <code>gps</code> library dependency is a liability — <code>StopIteration</code> crashes are a known issue with no upstream fix. The Go exporter covers more message types but uses polling instead of streaming, crashes on parse errors, and renames every metric so your dashboards need rewriting.</p>
<p>The Rust exporter takes the superset of both feature sets and adds the one thing neither has: reliability. It stays connected, reconnects when it can’t, and never exits on bad data.</p>
<h2 id="wrapping-up">Wrapping up</h2>
<p>The setup is a flake input, a module configuration, and making sure gpsd is configured to actually emit the data you want — which means getting the baud rate right and, if the exporter runs remotely, opening up the network socket. Once that’s wired, your GPS receiver’s position accuracy, satellite coverage, timing offsets, and signal quality all live in Prometheus alongside everything else. The <a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/prometheus-gpsd-exporter">project is on GitHub</a> — contributions welcome.</p>
Running Hickory-DNS as a Full Authoritative + Recursive DNS Server on NixOS2026-04-02T00:05:00+00:002026-04-02T00:05:00+00:00
Ian Johannesen
https://perlpimp.net/blog/hickory-dns-nixos/<p>BIND has been around since the 1980s. Unbound does recursive resolution well but doesn’t serve authoritative zones. CoreDNS is fine until you want DDNS with TSIG authentication and it’s suddenly less fine. Then there’s hickory-dns — a Rust DNS server that handles authoritative zones, recursive resolution, sqlite-backed DDNS zones, TSIG key verification, and Prometheus metrics. All in one binary.</p>
<p>The catch is it’s not in nixpkgs with the features you need. So you build it from source, wire it into a NixOS module, and generate your zone files declaratively from a host registry. This post walks through the entire working setup — not a toy example, but a production config with multiple VLANs, IPv4 and IPv6 reverse DNS, Kea DHCP-DDNS integration, systemd hardening, and a custom Grafana dashboard built from scratch because nobody’s made one yet.</p>
<h2 id="why-hickory-dns">Why hickory-dns</h2>
<p>The pitch is short:</p>
<ul>
<li><strong>Rust</strong> — memory-safe, single static-ish binary, no garbage collector pauses</li>
<li><strong>Built-in recursor</strong> — one process handles both authoritative and recursive queries</li>
<li><strong>Prometheus metrics</strong> — native <code>/metrics</code> endpoint, no sidecar exporter</li>
<li><strong>SQLite-backed DDNS zones</strong> — RFC 2136 dynamic updates with TSIG authentication, journal files for durability</li>
<li><strong>DNSSEC support</strong> — ring-based crypto for TSIG key verification</li>
</ul>
<p>One binary replaces what would otherwise be BIND (or Unbound + a separate authoritative server) plus a Prometheus exporter. The config is TOML, which is a nice change from BIND’s syntax.</p>
<h2 id="building-from-source-in-nix">Building from source in Nix</h2>
<p>Hickory-dns isn’t packaged in nixpkgs with the feature combination you need — sqlite, recursor, prometheus-metrics, and dnssec-ring. Build it inline:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">hickory-dns = pkgs.rustPlatform.buildRustPackage rec {
pname = "hickory-dns";
version = "0.26.0-beta.2";
src = pkgs.fetchFromGitHub {
owner = "hickory-dns";
repo = "hickory-dns";
hash = "sha256-7kra6MbLcv0P6iiUJ+hQ0ezqgXh/1KskCrZvFYDqiXQ=";
rev = "v${version}";
};
cargoHash = "sha256-FfckN+qhSqbc8jnL0xThdAMQEgluocSY1ksEyT8rFFY=";
buildAndTestSubdir = "bin";
buildFeatures = [
"sqlite" "resolver" "recursor"
"prometheus-metrics" "dnssec-ring"
];
nativeBuildInputs = [ pkgs.pkg-config ];
buildInputs = [ pkgs.openssl pkgs.sqlite ];
doCheck = false;
meta.mainProgram = "hickory-dns";
};
</code></pre>
<p>A few things to note. <code>buildAndTestSubdir = "bin"</code> is critical — the workspace has many crates and you only want the server binary. The <code>buildFeatures</code> list is the whole reason you’re building from source: <code>sqlite</code> for DDNS journal zones, <code>recursor</code> for upstream resolution, <code>prometheus-metrics</code> for monitoring, and <code>dnssec-ring</code> for TSIG key verification on dynamic updates. Tests are disabled because they require network access — <code>doCheck = false</code> and move on.</p>
<h2 id="generating-zone-files-from-a-host-registry">Generating zone files from a host registry</h2>
<p>Hard-coding zone files is fine for five hosts. At twenty, with multiple VLANs, reverse zones, and IPv6, it’s a maintenance disaster. The better approach is generating everything from a single Nix attrset — a host registry:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">hosts = {
myhost = { ip = "10.0.1.10"; mac = "aa:bb:cc:dd:ee:ff"; };
another = { ip = "10.0.2.20"; ip6 = "fd00:255:101::20";
dns = [ "another" "alias" ]; };
};
</code></pre>
<p>Every host has an IP and optional fields — MAC for DHCP reservations, <code>ip6</code> for IPv6, <code>dns</code> for additional names. From this, you generate forward zones, reverse zones, and DHCP configs. One source of truth, zero drift.</p>
<h3 id="forward-zone-a-and-aaaa-records">Forward zone (A and AAAA records)</h3>
<p>The SOA boilerplate goes into a helper:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">mkSoa = zone: ''
$ORIGIN ${zone}.
$TTL 3600
@ IN SOA ns1.${domain}. admin.${domain}. (
1 ; serial
3600 ; refresh
900 ; retry
604800 ; expire
300 ; minimum
)
@ IN NS ns1.${domain}.
'';
</code></pre>
<p>Then you filter and map:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">isInZone = name: !(lib.hasInfix "." name);
hostARecords = lib.flatten (lib.mapAttrsToList (name: host:
let names = builtins.filter isInZone (hostDnsNames name host);
in map (n: "${n} IN A ${host.ip}") names
) network.hosts);
hostAAAARecords = lib.optionals network.enableIPv6ULA (lib.flatten (
lib.mapAttrsToList (name: host:
let names = builtins.filter isInZone (hostDnsNames name host);
in map (n: "${n} IN AAAA ${host.ip6}") names
) hostsWithIp6));
forwardZoneContent =
mkSoa domain
+ "ns1 IN A ${gateway.ip}\n"
+ lib.concatStringsSep "\n" (hostARecords ++ extraARecords ++ hostAAAARecords)
+ "\n";
forwardZoneFile = pkgs.writeText "${domain}.zone" forwardZoneContent;
</code></pre>
<p>The <code>isInZone</code> filter catches an easy mistake — names with dots in them (like <code>k8s-master.local</code>) belong in other domains and can’t be served from this authoritative zone. Filter them out at the Nix level instead of debugging broken DNS resolution later.</p>
<h3 id="ipv4-reverse-zones-ptr-records-per-24-subnet">IPv4 reverse zones (PTR records per /24 subnet)</h3>
<p>Reverse zones are per-subnet, so you group hosts by their first three octets:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">hostsBySubnet = lib.groupBy (h:
let parts = lib.splitString "." h.ip;
in "${builtins.elemAt parts 0}.${builtins.elemAt parts 1}.${builtins.elemAt parts 2}"
) allHostEntries;
mkReverseZone = subnet: entries:
let
parts = lib.splitString "." subnet;
revSubnet = "${builtins.elemAt parts 2}.${builtins.elemAt parts 1}.${builtins.elemAt parts 0}";
zoneName = "${revSubnet}.in-addr.arpa";
records = map (h:
let lastOctet = lib.last (lib.splitString "." h.ip);
in "${lastOctet} IN PTR ${h.dnsName}.${domain}."
) entries;
in {
name = zoneName;
content = mkSoa zoneName
+ "ns1.${domain}. IN A ${gateway.ip}\n"
+ lib.concatStringsSep "\n" records + "\n";
};
reverseZones = lib.mapAttrsToList mkReverseZone hostsBySubnet;
</code></pre>
<p>Each /24 subnet gets its own <code>.in-addr.arpa</code> zone with PTR records pointing back to the canonical hostname. Add a host to the registry, rebuild, and forward and reverse DNS update together.</p>
<h3 id="ipv6-reverse-zones-nibble-based-ptr-records">IPv6 reverse zones (nibble-based PTR records)</h3>
<p>IPv6 reverse DNS is where things get tedious. Each address expands to individual hex nibbles, reversed and dot-separated under <code>.ip6.arpa</code>. A helper does the expansion:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">ip6Nibbles = addr:
let expanded = network.expandIp6 addr;
in lib.stringToCharacters (lib.replaceStrings [":"] [""] expanded);
ip6ZoneName = addr:
let
nibbles = ip6Nibbles addr;
rev12 = lib.concatStringsSep "." (lib.reverseList (lib.take 12 nibbles));
in "${rev12}.ip6.arpa";
</code></pre>
<p>The first 12 nibbles (48 bits) form the zone name — this covers a /48 prefix. The remaining 20 nibbles become the host part of each PTR record:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">mkIp6ReverseZone = zoneName: entries:
let
records = map (h:
let
nibbles = ip6Nibbles h.ip6;
allRev = lib.reverseList nibbles;
relName = lib.concatStringsSep "." (lib.take 20 allRev);
in "${relName} IN PTR ${h.dnsName}.${domain}."
) entries;
in {
name = zoneName;
content = mkSoa zoneName
+ "ns1.${domain}. IN A ${gateway.ip}\n"
+ lib.concatStringsSep "\n" records + "\n";
};
ip6ReverseZones = lib.optionals network.enableIPv6ULA
(lib.mapAttrsToList mkIp6ReverseZone hostsByIp6Zone);
</code></pre>
<p>Manually maintaining nibble-format PTR records for even a dozen IPv6 hosts is a recipe for typos. Generating them from the registry makes it mechanical.</p>
<h3 id="splitting-zones-static-vs-ddns-updatable">Splitting zones: static vs DDNS-updatable</h3>
<p>Here’s the key architectural decision. Reverse zones for subnets with DHCP clients need to accept dynamic updates — Kea D2 will send RFC 2136 updates to create PTR records when leases are handed out. But reverse zones for static-only subnets should be plain zone files, read-only. You split them at the Nix level:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">ddnsSubnets = [ "10.0.1" "10.0.2" "10.0.3" ];
zoneToSubnet = zoneName:
let
stripped = lib.removeSuffix ".in-addr.arpa" zoneName;
parts = lib.splitString "." stripped;
in "${builtins.elemAt parts 2}.${builtins.elemAt parts 1}.${builtins.elemAt parts 0}";
isDdnsReverseZone = z: builtins.elem (zoneToSubnet z.name) ddnsSubnets;
ddnsReverseZones = builtins.filter isDdnsReverseZone reverseZones;
staticReverseZones = builtins.filter (z: !(isDdnsReverseZone z)) reverseZones;
</code></pre>
<p>Same split for IPv6 — DHCP-served subnets get sqlite-backed DDNS zones, management subnets get static files. This matters because DDNS zones require sqlite storage, TSIG key configuration, and journal files. You don’t want that overhead on zones that never change.</p>
<p>The DDNS forward zones start empty — they’re populated at runtime by Kea:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">ddnsZoneContent =
mkSoa "dhcp.${domain}"
+ "ns1.${domain}. IN A ${gateway.ip}\n";
guestZoneContent =
mkSoa "guest.${domain}"
+ "ns1.${domain}. IN A ${gateway.ip}\n";
</code></pre>
<h2 id="toml-configuration">TOML configuration</h2>
<p>With all the zone files generated, the TOML config ties everything together. Four categories of zones, plus a recursor for upstream resolution:</p>
<pre data-lang="toml" class="language-toml "><code class="language-toml" data-lang="toml">listen_addrs_ipv4 = ["10.0.0.1", "10.0.1.1", "10.0.2.1", "127.0.0.1"]
listen_addrs_ipv6 = ["fd00:255:100::1", "::1"]
listen_port = 53
directory = "/var/lib/hickory-dns"
tcp_request_timeout = 5
allow_networks = ["10.0.0.0/8", "127.0.0.0/8", "fd00::/8", "::1/128"]
prometheus_listen_addr = "127.0.0.1:9153"
# Static authoritative forward zone
[[zones]]
zone = "example.net."
zone_type = "Primary"
file = "/nix/store/...-example.net.zone"
# DDNS forward zone (sqlite + TSIG)
[[zones]]
zone = "dhcp.example.net."
zone_type = "Primary"
[zones.stores]
type = "sqlite"
zone_path = "/nix/store/...-dhcp.example.net.zone"
journal_path = "/var/lib/hickory-dns/dhcp.example.net.jrnl"
allow_update = true
[[zones.stores.tsig_keys]]
name = "kea-ddns-key."
algorithm = "hmac-sha256"
key_file = "/var/lib/hickory-dns/tsig-key.bin"
# Root hints recursor (upstream resolution)
[[zones]]
zone = "."
zone_type = "External"
[zones.stores]
type = "recursor"
roots = "/nix/store/...-root.hints"
ns_cache_size = 1024
record_cache_size = 1048576
</code></pre>
<p>A few important details. You must list every interface IP explicitly in <code>listen_addrs_ipv4</code> — hickory-dns doesn’t support <code>0.0.0.0</code> binding reliably. The <code>allow_networks</code> list restricts which clients can query. DDNS zones use <code>type = "sqlite"</code> with <code>allow_update = true</code> and a TSIG key for authentication. The root hints file comes from <code>pkgs.dns-root-data</code>. And <code>record_cache_size = 1048576</code> gives the recursor a generous cache — one million entries.</p>
<p>The Nix side generates this TOML with string interpolation, rendering static reverse zones as plain file zones and DDNS reverse zones with the sqlite/TSIG configuration:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix"># Static reverse zones — plain file, read-only
staticReverseZoneToml = lib.concatMapStringsSep "\n" (z: ''
[[zones]]
zone = "${z.name}."
zone_type = "Primary"
file = "${reverseZoneFilesByName.${z.name}}"
'') staticReverseZones;
# DDNS reverse zones — sqlite + journal + TSIG key
ddnsReverseZoneToml = lib.concatMapStringsSep "\n" (z: ''
[[zones]]
zone = "${z.name}."
zone_type = "Primary"
[zones.stores]
type = "sqlite"
zone_path = "${reverseZoneFilesByName.${z.name}}"
journal_path = "${dataDir}/${z.name}.jrnl"
allow_update = true
${tsigKeyToml}
'') ddnsReverseZones;
</code></pre>
<p>Same pattern for IPv6 — static and DDNS variants, generated from the zone split you did earlier.</p>
<h2 id="systemd-service-with-hardening">SystemD service with hardening</h2>
<p>The service definition is straightforward but the hardening is thorough:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">users.users.hickory-dns = {
isSystemUser = true;
group = "hickory-dns";
};
users.groups.hickory-dns = {};
systemd.services.hickory-dns = {
description = "hickory-dns DNS server";
after = [ "network-online.target" ];
wants = [ "network-online.target" ];
wantedBy = [ "multi-user.target" ];
serviceConfig = {
ExecStartPre = "${pkgs.bash}/bin/bash -c '${pkgs.coreutils}/bin/base64 -d \
< ${config.sops.secrets.hickory_dns_private_key.path} \
> ${tsigKeyRawPath}'";
ExecStart = "${hickory-dns}/bin/hickory-dns -c ${configFile}";
User = "hickory-dns";
Group = "hickory-dns";
StateDirectory = "hickory-dns";
AmbientCapabilities = [ "CAP_NET_BIND_SERVICE" ];
CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" ];
LockPersonality = true;
MemoryDenyWriteExecute = true;
NoNewPrivileges = true;
PrivateDevices = true;
PrivateTmp = true;
ProtectClock = true;
ProtectControlGroups = true;
ProtectHome = true;
ProtectHostname = true;
ProtectKernelLogs = true;
ProtectKernelModules = true;
ProtectKernelTunables = true;
ProtectSystem = "strict";
ReadWritePaths = [ dataDir ];
RestrictAddressFamilies = [ "AF_INET" "AF_INET6" "AF_UNIX" ];
RestrictNamespaces = true;
RestrictRealtime = true;
SystemCallArchitectures = "native";
};
};
</code></pre>
<p>The <code>ExecStartPre</code> step decodes the TSIG key from a sops-nix secret before the server starts — the raw key file lives in the state directory and is readable only by the service user. <code>CAP_NET_BIND_SERVICE</code> is the only capability needed (port 53). <code>StateDirectory = "hickory-dns"</code> tells systemd to create <code>/var/lib/hickory-dns</code> owned by the service user. <code>ProtectSystem = "strict"</code> plus <code>ReadWritePaths</code> means only the state dir is writable — for sqlite journals and the decoded TSIG key. Everything else is locked down.</p>
<h2 id="kea-dhcp-ddns-integration">Kea DHCP-DDNS integration</h2>
<p>This is where most of the debugging happens. Three pieces need to align: Kea’s DHCP servers, the D2 daemon, and hickory-dns. Get one wrong and you’ll have hostnames that resolve for some devices but not others, or PTR records that point to the wrong zone.</p>
<h3 id="skipping-static-reservations">Skipping static reservations</h3>
<p>Hosts with static DNS entries in the forward zone must not get DDNS records. Otherwise Kea creates <code>myhost.dhcp.example.net</code> entries that shadow the authoritative <code>myhost.example.net</code> records — or worse, you get duplicate names in different zones with different TTLs and spend an evening figuring out why <code>dig</code> returns different answers depending on which resolver cache you hit.</p>
<p>The fix: tag all static reservations with a <code>SKIP_DDNS</code> client class:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">skipDdnsReservations = rs: map (r: r // { client-classes = [ "SKIP_DDNS" ]; }) rs;
# Applied to both DHCPv4 and DHCPv6 reservations
reservations = skipDdnsReservations network.dhcpReservations;
</code></pre>
<p>This works because Kea’s <code>ddns_tuning</code> hooks library respects the class and suppresses DDNS updates for matching clients:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">hooks-libraries = [{
library = "${pkgs.kea}/lib/kea/hooks/libdhcp_ddns_tuning.so";
parameters = {};
}];
</code></pre>
<p>Load this hook in both <code>dhcp4</code> and <code>dhcp6</code> settings.</p>
<h3 id="d2-config-with-tsig-and-reverse-dns">D2 config with TSIG and reverse DNS</h3>
<p>The Kea D2 daemon handles the actual RFC 2136 updates. Its config contains the TSIG key secret, so it’s rendered via a sops template:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">systemd.services.kea-dhcp-ddns-server = {
after = [ "hickory-dns.service" ];
wants = [ "hickory-dns.service" ];
serviceConfig.ExecStart = lib.mkForce
"${pkgs.kea}/bin/kea-dhcp-ddns -c ${config.sops.templates."kea-dhcp-ddns.conf".path}";
};
</code></pre>
<p>The template itself defines forward and reverse DDNS domains:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">sops.templates."kea-dhcp-ddns.conf" = {
mode = "0444";
restartUnits = [ "kea-dhcp-ddns-server.service" ];
content = builtins.toJSON {
DhcpDdns = {
ip-address = "127.0.0.1";
port = 53001;
dns-server-timeout = 3000;
tsig-keys = [{
name = "kea-ddns-key.";
algorithm = "HMAC-SHA256";
secret = config.sops.placeholder.hickory_dns_private_key;
}];
forward-ddns = {
ddns-domains = [
{
name = "dhcp.example.net.";
key-name = "kea-ddns-key.";
dns-servers = [{ ip-address = "127.0.0.1"; port = 53; }];
}
{
name = "guest.example.net.";
key-name = "kea-ddns-key.";
dns-servers = [{ ip-address = "127.0.0.1"; port = 53; }];
}
];
};
reverse-ddns = {
ddns-domains = [
# IPv4 reverse zones for DHCP subnets
{
name = "1.0.10.in-addr.arpa.";
key-name = "kea-ddns-key.";
dns-servers = [{ ip-address = "127.0.0.1"; port = 53; }];
}
{
name = "2.0.10.in-addr.arpa.";
key-name = "kea-ddns-key.";
dns-servers = [{ ip-address = "127.0.0.1"; port = 53; }];
}
# IPv6 reverse zones (generated from ULA prefix)
{
name = wiredIp6RevZone;
key-name = "kea-ddns-key.";
dns-servers = [{ ip-address = "127.0.0.1"; port = 53; }];
}
{
name = wifiIp6RevZone;
key-name = "kea-ddns-key.";
dns-servers = [{ ip-address = "127.0.0.1"; port = 53; }];
}
];
};
};
};
};
</code></pre>
<p><strong>Gotcha worth highlighting:</strong> because <code>ExecStart</code> is overridden with <code>lib.mkForce</code> to point at the sops template, the NixOS kea module’s own restart triggers no longer cover the actual config. Without <code>restartUnits</code> on the sops template, changing the D2 config deploys a new template file but the running D2 process keeps the old one. The <code>restartUnits = [ "kea-dhcp-ddns-server.service" ]</code> line tells sops-nix to restart D2 whenever the rendered template content changes. This pattern is needed any time you <code>mkForce</code> a service’s <code>ExecStart</code> to use a sops template.</p>
<h3 id="per-subnet-ddns-settings">Per-subnet DDNS settings</h3>
<p>Each VLAN gets its own qualifying suffix — or no DDNS at all:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix"># WiFi and wired subnets — register in dhcp.example.net
ddns-send-updates = true;
ddns-qualifying-suffix = "dhcp.example.net.";
# Guest subnet — register in guest.example.net
ddns-send-updates = true;
ddns-qualifying-suffix = "guest.example.net.";
# Camera and management subnets — no DDNS
ddns-send-updates = false;
</code></pre>
<p>Global DDNS settings that apply to both DHCPv4 and DHCPv6:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">ddns-override-client-update = true;
ddns-replace-client-name = "never";
ddns-update-on-renew = true;
hostname-char-set = "[^A-Za-z0-9.-]";
hostname-char-replacement = "-";
</code></pre>
<p>The <code>hostname-char-set</code> and replacement ensure that devices sending garbage hostnames — and they will — get sanitized into valid DNS labels.</p>
<h3 id="search-domains-per-vlan">Search domains per VLAN</h3>
<p>Each network gets appropriate search domains so short names resolve without qualification:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix"># WiFi and wired: dhcp.example.net, example.net, parent.net
searchDomainWifiWired = {
code = 119;
data = "dhcp.example.net, example.net, parent.net";
name = "domain-search";
space = "dhcp4";
};
# Management: example.net, parent.net (no dhcp subdomain)
searchDomainMgnt = {
code = 119;
data = "example.net, parent.net";
name = "domain-search";
space = "dhcp4";
};
# Guest: only guest.example.net
searchDomainGuest = {
code = 119;
data = "guest.example.net";
name = "domain-search";
space = "dhcp4";
};
</code></pre>
<p>Trusted VLANs search both the DDNS zone and the authoritative zone — so <code>ssh myhost</code> resolves whether <code>myhost</code> got its name from a static zone entry or a DHCP lease. Guest gets only its own zone. Management doesn’t need the DDNS subdomain because everything on that VLAN has a static reservation.</p>
<h3 id="the-complete-data-flow">The complete data flow</h3>
<p>When it all comes together:</p>
<ol>
<li>Client gets a DHCP lease from Kea (v4 or v6)</li>
<li>If <code>ddns-send-updates = true</code> for that subnet and the client isn’t tagged <code>SKIP_DDNS</code>:
<ul>
<li>Kea sends an RFC 2136 UPDATE to the D2 daemon (port 53001)</li>
<li>D2 sends a TSIG-authenticated forward update to hickory-dns (A/AAAA in <code>dhcp.</code> or <code>guest.</code> zone)</li>
<li>D2 sends a TSIG-authenticated reverse update to hickory-dns (PTR in <code>.in-addr.arpa</code> or <code>.ip6.arpa</code> zone)</li>
</ul>
</li>
<li>Hickory-dns validates the TSIG signature and writes to the sqlite journal</li>
<li>Static reservations — servers with known IPs — only appear in the authoritative forward zone, never duplicated in the DDNS zone</li>
</ol>
<h2 id="prometheus-monitoring">Prometheus monitoring</h2>
<p>The <code>prometheus-metrics</code> build feature exposes metrics on the address you configured. Scrape config is minimal:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.prometheus.scrapeConfigs = [{
job_name = "hickory-dns";
honor_labels = true;
static_configs = [{
targets = [ "127.0.0.1:9153" ];
}];
}];
</code></pre>
<p>The metrics you get are genuinely useful:</p>
<ul>
<li><code>hickory_request_record_types_total</code> — query types (A, AAAA, PTR, HTTPS, MX, SRV, TXT, DS, DNSKEY, SOA, NS)</li>
<li><code>hickory_response_codes_total</code> — response codes (NOERROR, NXDOMAIN, SERVFAIL, etc.)</li>
<li><code>hickory_request_protocols_total</code> — TCP vs UDP split</li>
<li><code>hickory_recursor_cache_hit_total</code> / <code>hickory_recursor_cache_miss_total</code> — cache effectiveness</li>
<li><code>hickory_recursor_cache_hit_duration_seconds_bucket</code> / <code>hickory_recursor_cache_miss_duration_seconds_bucket</code> — latency histograms</li>
<li><code>hickory_recursor_response_cache_size</code> / <code>hickory_recursor_name_server_cache_size</code> — cache fill levels</li>
<li><code>hickory_recursor_in_flight_queries</code> — concurrent query count</li>
<li><code>hickory_zone_lookups_total</code> — per-zone lookups by handler and success</li>
<li><code>hickory_zone_records_total</code> — record count per zone</li>
<li>Standard process metrics — RSS, virtual memory, CPU seconds, threads, open FDs</li>
</ul>
<h2 id="grafana-dashboard">Grafana dashboard</h2>
<p>There is no pre-made Grafana dashboard for hickory-dns. Had to build one from scratch by reading the metrics endpoint. It has four sections and fourteen panels.</p>
<h3 id="overview">Overview</h3>
<ul>
<li><strong>Query Rate (by type)</strong> — stacked area chart: <code>sum by (type) (rate(hickory_request_record_types_total{job="hickory-dns", type=~"a|aaaa|ptr|https|mx|srv|txt|ds|dnskey|soa|ns"}[$__rate_interval]))</code></li>
<li><strong>Response Rate (by rcode)</strong> — <code>sum by (code) (rate(hickory_response_codes_total{job="hickory-dns"}[$__rate_interval])) > 0</code></li>
<li><strong>Recursor Latency</strong> — p50/p95 from <code>histogram_quantile</code> on both <code>hickory_recursor_cache_miss_duration_seconds_bucket</code> and <code>hickory_recursor_cache_hit_duration_seconds_bucket</code></li>
<li><strong>Requests by Protocol</strong> — <code>sum by (protocol) (rate(hickory_request_protocols_total{job="hickory-dns"}[$__rate_interval]))</code></li>
</ul>
<h3 id="cache-and-recursor">Cache and recursor</h3>
<ul>
<li><strong>Cache Hit/Miss Rate</strong> — <code>rate(hickory_recursor_cache_hit_total{...}[$__rate_interval])</code> and the miss equivalent</li>
<li><strong>Cache Hit Ratio</strong> — gauge panel: <code>hit_rate / (hit_rate + miss_rate)</code> with thresholds: red below 50%, yellow 50–80%, green above 80%</li>
<li><strong>Cache Sizes & In-Flight</strong> — <code>hickory_recursor_response_cache_size</code>, <code>hickory_recursor_name_server_cache_size</code>, <code>hickory_recursor_in_flight_queries</code></li>
</ul>
<h3 id="zones">Zones</h3>
<ul>
<li><strong>Zone Lookups (by handler)</strong> — <code>sum by (zone_handler, success) (rate(hickory_zone_lookups_total{...}[$__rate_interval]))</code></li>
<li><strong>Zone Records Count</strong> — <code>hickory_zone_records_total</code> with legend format <code>{{zone_handler}} {{type}} {{role}}</code></li>
</ul>
<h3 id="process">Process</h3>
<ul>
<li><strong>Memory Usage</strong> — RSS vs virtual</li>
<li><strong>CPU / Threads / FDs</strong> — <code>rate(process_cpu_seconds_total{...}[$__rate_interval])</code>, <code>process_threads</code>, <code>process_open_fds</code></li>
</ul>
<p>Design decisions: a <code>${datasource}</code> template variable for Prometheus datasource selection, <code>$__rate_interval</code> everywhere for proper rate calculations, stacked area charts for rates, line charts for latencies, table legends with mean/max calcs. Default time range is six hours.</p>
<h2 id="firewall-rules">Firewall rules</h2>
<p>DNS needs to be reachable from every VLAN that should resolve. For nftables:</p>
<pre><code># Trusted networks (full access)
ip saddr 10.0.0.0/8 tcp dport 53 counter accept comment "lan dns tcp"
ip saddr 10.0.0.0/8 udp dport 53 counter accept comment "lan dns udp"
# Isolated networks (DNS + DHCP only, everything else dropped)
iifname "guest" udp dport { 53, 67, 68 } counter accept comment "guest dns+dhcp"
iifname "guest" tcp dport 53 counter accept comment "guest dns tcp"
iifname "camera" udp dport { 53, 67, 68 } counter accept comment "camera dns+dhcp"
iifname "camera" tcp dport 53 counter accept comment "camera dns tcp"
</code></pre>
<p>Guest and camera networks are isolated — they can reach DNS and DHCP, nothing else. Trusted interfaces get full access. The DNS server is the gateway, so it’s reachable on every VLAN interface without extra routing.</p>
<h2 id="wrapping-up">Wrapping up</h2>
<p>The whole setup is one Nix module that generates zone files from a host registry, builds hickory-dns with the right features, renders a TOML config, runs a hardened systemd service, and wires up Kea D2 for dynamic updates. Add a host to the registry, rebuild, and forward DNS, reverse DNS, and DHCP reservations all update together. The Prometheus metrics are there from day one, and the Grafana dashboard — since nobody else has built one yet — gives you query rates, cache hit ratios, recursor latency, and per-zone lookup breakdowns. It’s a lot of moving parts, but once it’s declarative, the moving parts stop being your problem.</p>
NixOS on a Budget VPS: Getting Under the 2GB RAM Floor2026-04-01T12:00:00+00:002026-04-01T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/nixos-low-memory-vps-nixos-infect-kexec/<p>You found a VPS deal. Three bucks a month, 1GB RAM, somewhere in Eastern Europe. You want NixOS on it. You run nixos-anywhere and it dies halfway through because the kexec installer needs 2GB to do its thing. Your three-dollar dream just hit a wall.</p>
<p>This is the RAM floor problem. The standard nixos-anywhere workflow boots a kexec image that replaces the running kernel with a minimal NixOS installer. That installer needs enough memory to hold itself, the Nix store, and the build artifacts — and that comes out to roughly 2GB. Below that, it either OOM-kills itself or hangs.</p>
<p>Two approaches get you past it:</p>
<ul>
<li><strong>nixos-infect</strong> converts the running OS in-place without kexec. Works down to <strong>768MB RAM</strong>.</li>
<li><strong>A custom kexec image</strong> strips the installer down to fit in ~<strong>1GB RAM</strong>, keeping the nixos-anywhere workflow intact.</li>
</ul>
<h2 id="nixos-infect-the-in-place-conversion">nixos-infect: the in-place conversion</h2>
<p><a rel="noopener noreferrer" target="_blank" href="https://github.com/elitak/nixos-infect">nixos-infect</a> takes whatever Linux distro your VPS booted — usually Ubuntu or Debian — and converts it to NixOS while it’s running. No kexec, no installer image, no second OS in memory. It downloads Nix, builds a NixOS system closure, installs the bootloader, and reboots into your new system.</p>
<p>The tradeoff is that it keeps the existing partition layout. No disko, no declarative disk formatting. Whatever filesystem your provider set up is what you get. For a cheap VPS where the disk layout is “one ext4 partition and a prayer,” that’s usually fine.</p>
<p>Here’s the invocation that actually works on budget providers:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">ssh root@your-vps "curl -sL https://github.com/elitak/nixos-infect/raw/master/nixos-infect \
| sed 's|mktemp /tmp/nixos-infect|mktemp /var/tmp/nixos-infect|g' \
| doNetConf=y NIX_CHANNEL=nixos-unstable bash -x"
</code></pre>
<p>Three things are happening beyond the obvious <code>curl | bash</code>.</p>
<h3 id="the-sed-patch-surviving-small-tmp">The <code>sed</code> patch: surviving small <code>/tmp</code></h3>
<p>Many budget providers mount <code>/tmp</code> as a small tmpfs — sometimes just tens of megabytes. nixos-infect creates temporary files via <code>mktemp /tmp/nixos-infect.*</code>, and on these providers that write hits the tiny tmpfs and fails silently or with a cryptic “no space left on device” error.</p>
<p>The <code>sed</code> rewrites the path from <code>/tmp/nixos-infect</code> to <code>/var/tmp/nixos-infect</code>, which lives on the real disk. It’s one substitution that turns a mysterious failure into a working install.</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">sed 's|mktemp /tmp/nixos-infect|mktemp /var/tmp/nixos-infect|g'
</code></pre>
<p>If your provider gives <code>/tmp</code> real disk space, the patch is harmless. If it doesn’t, the patch is mandatory.</p>
<h3 id="donetconf-y-keeping-your-network-alive"><code>doNetConf=y</code>: keeping your network alive</h3>
<p>Most budget VPS hosts use static IP addressing. Without <code>doNetConf=y</code>, nixos-infect doesn’t capture the current network configuration before rebooting. The machine comes back up with NixOS, but NixOS has no idea what its IP, gateway, or DNS servers are. You’re locked out.</p>
<p><code>doNetConf=y</code> tells nixos-infect to snapshot the active network config — IPs, gateways, DNS resolvers — and bake it into the generated NixOS configuration. For any host that isn’t running DHCP, this flag is not optional.</p>
<h3 id="what-survives-the-reboot">What survives the reboot</h3>
<p>nixos-infect preserves your root SSH authorized keys, so you keep access after the conversion. The existing partition layout stays as-is. What you get after reboot is a bare NixOS system with your SSH keys and a working network — a blank canvas to deploy your actual config onto.</p>
<p>After reboot, deploy your real NixOS configuration with <code>nixos-rebuild switch --flake</code> or whatever your deployment tool of choice is. nixos-infect just gets you a bootable NixOS base.</p>
<h2 id="custom-kexec-nixos-anywhere-on-a-diet">Custom kexec: nixos-anywhere on a diet</h2>
<p>If you want the full nixos-anywhere experience — declarative partitioning with disko, a clean install, the works — but your VPS only has around 1GB of RAM, you need a slimmer kexec image.</p>
<p>The <a rel="noopener noreferrer" target="_blank" href="https://github.com/nix-community/nixos-anywhere">standard nixos-anywhere kexec</a> bundles a comfortable installer environment. Comfortable costs memory. A <a rel="noopener noreferrer" target="_blank" href="https://gitlab.com/-/snippets/5973159">custom kexec image</a> strips that down to the bare minimum.</p>
<p>Here’s what that minimal kexec module looks like:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
self,
modulesPath,
lib,
config,
pkgs,
...
}: {
imports = [
"${modulesPath}/installer/netboot/netboot.nix"
"${modulesPath}/profiles/minimal.nix"
"${modulesPath}/profiles/qemu-guest.nix"
];
system.nixos.variant_id = "kexec";
system.build.kexec_compressed =
pkgs.runCommand "kexec-compressed"
{ buildInputs = [ pkgs.gnutar ]; }
''
mkdir $out
tar -cvzhf $out/kexec.tar.gz -C ${config.system.build.kexecTree} \
bzImage \
initrd.gz \
kexec-boot
'';
boot = {
initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" ];
kernelParams = [
"panic=30"
"boot.panic_on_fail"
"console=ttyS0"
"console=tty1"
];
kernel.sysctl."vm.overcommit_memory" = lib.mkForce "1";
supportedFilesystems = [ "zfs" ];
};
environment.variables.GC_INITIAL_HEAP_SIZE = "1M";
documentation.enable = false;
documentation.nixos.enable = false;
fonts.fontconfig.enable = false;
programs.bash.completion.enable = false;
programs.command-not-found.enable = false;
security.polkit.enable = false;
security.rtkit.enable = pkgs.lib.mkForce false;
services.udisks2.enable = false;
i18n.supportedLocales = [
(config.i18n.defaultLocale + "/UTF-8")
];
networking.hostName = "kexec";
networking.hostId = "88ad2520";
services = {
getty.autologinUser = lib.mkForce "root";
openssh = {
enable = true;
settings.KbdInteractiveAuthentication = false;
settings.PasswordAuthentication = false;
};
};
environment.etc.is_kexec.text = "";
system.stateVersion = "23.11";
users.users.root.openssh.authorizedKeys.keys = [
# your SSH public key here
];
}
</code></pre>
<p>The key decisions that shave off memory:</p>
<ul>
<li><strong><code>profiles/minimal.nix</code></strong> instead of the full installer profile — no man pages, no extra tooling</li>
<li><strong><code>vm.overcommit_memory = "1"</code></strong> — tells the kernel to always say yes to memory allocations and sort it out later via OOM, rather than refusing allocations conservatively. Aggressive, but it’s a throwaway installer environment</li>
<li><strong><code>GC_INITIAL_HEAP_SIZE = "1M"</code></strong> — keeps the Nix garbage collector’s initial heap small</li>
<li><strong>Everything non-essential disabled</strong> — documentation, fonts, bash completion, polkit, udisks, command-not-found. Every service you don’t load is memory you keep</li>
<li><strong><code>/etc/is_kexec</code></strong> marker file — tells nixos-anywhere it’s already in a kexec environment so it doesn’t try to kexec again</li>
</ul>
<p>Build the image and point nixos-anywhere at it:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix build .#kexec-image
nixos-anywhere --flake .#my-vps --kexec ./result/kexec.tar.gz root@your-vps
</code></pre>
<p>You get the full nixos-anywhere workflow — disko partitioning, clean install, proper host keys — on a box that would choke on the default kexec image.</p>
<h2 id="when-to-use-which">When to use which</h2>
<table><thead><tr><th>RAM</th><th>Approach</th><th>What you get</th></tr></thead><tbody>
<tr><td>< 768MB</td><td>Neither — upgrade or look elsewhere</td><td>Pain</td></tr>
<tr><td>768MB – 1GB</td><td>nixos-infect</td><td>NixOS, existing partitions, no disko</td></tr>
<tr><td>~1GB</td><td>Custom kexec + nixos-anywhere</td><td>Full nixos-anywhere with disko</td></tr>
<tr><td>2GB+</td><td>Standard nixos-anywhere</td><td>Everything works out of the box</td></tr>
</tbody></table>
<p>nixos-infect is the safer bet at the low end. It doesn’t need to hold an entire installer OS in memory — it mutates the running system. The downside is you inherit whatever disk layout your provider gave you.</p>
<p>The custom kexec approach is better if you care about declarative partitioning or need a clean-slate install. But it’s cutting it close at 1GB. If the provider is generous with their RAM accounting and doesn’t have other processes eating into it, it works. If they overcommit heavily on the host side, you might still hit the wall.</p>
<h2 id="provider-gotchas">Provider gotchas</h2>
<p>A few things to watch for on the LowEndBox-tier providers where this matters:</p>
<p><strong><code>/tmp</code> as tmpfs.</strong> Already covered above, but worth repeating. If <code>df -h /tmp</code> shows a tmpfs mount of 50–100MB, the <code>sed</code> patch for nixos-infect is mandatory.</p>
<p><strong>Static networking.</strong> The vast majority of budget providers don’t use DHCP. <code>doNetConf=y</code> is not optional. Forget it once, learn the lesson permanently.</p>
<p><strong>Swap.</strong> Some providers give you swap, some don’t. If your 1GB VPS has no swap and you’re trying the custom kexec route, you’re working without a safety net. Consider adding a swap file to your NixOS config as one of the first things you deploy.</p>
<p><strong>OpenVZ vs KVM.</strong> OpenVZ containers can’t kexec at all — the host kernel is shared. nixos-infect is your only option on OpenVZ, and even then it’s hit-or-miss depending on what the container exposes. KVM is the safe choice for NixOS.</p>
<h2 id="after-the-install">After the install</h2>
<p>Whichever path you took, you now have a minimal NixOS system with SSH access. From here:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nixos-rebuild switch --flake .#my-vps --target-host root@your-vps
</code></pre>
<p>Push your real configuration. Set up your services. The hard part — getting NixOS onto a box that wasn’t designed for it — is done. Everything from here is just regular NixOS.</p>
<p>Three dollars a month, 1GB of RAM, and a fully declarative operating system. Not bad for a provider that only officially supports Ubuntu.</p>
Self-Hosting a Private Nix Binary Cache with Attic and Garage2026-03-31T12:00:00+00:002026-03-31T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/self-hosted-nix-binary-cache-attic-garage/<p>You want a private Nix binary cache. Not “throw artifacts into S3 and hope for the best” private. A real cache server, scoped credentials, multiple teams, and CI that pushes build outputs automatically. Basically the Cachix model, but self-hosted and under your control.</p>
<p>This stack does that cleanly:</p>
<ul>
<li><strong>Garage</strong> provides S3-compatible object storage for NAR files</li>
<li><strong>Attic</strong> provides the cache API, signing keys, cache metadata, and auth tokens</li>
<li><strong>PostgreSQL</strong> stores Attic metadata</li>
<li><strong>sops-nix</strong> wires in secrets without hardcoding credentials in your Nix config</li>
<li><strong>GitHub Actions</strong> pushes build outputs into the cache on trusted builds</li>
</ul>
<p>The useful twist is Attic’s token model. You can hand a team one token scoped to <code>teamname-*</code>, and they can create and manage <code>teamname-dev</code>, <code>teamname-prod</code>, <code>teamname-whatever</code> without touching anyone else’s caches. That’s the self-hosted private Cachix pitch.</p>
<p>This post walks through the whole setup on NixOS. It does <strong>not</strong> cover per-cache vanity hostnames through nginx. Attic still exposes one global substituter endpoint, so path-based cache URLs are the model today.</p>
<h2 id="why-split-attic-and-garage">Why split Attic and Garage</h2>
<p>Attic wants durable blob storage plus a relational database. Garage gives you the blob layer without needing MinIO, Ceph, or a cloud bucket. PostgreSQL handles Attic’s metadata. That separation matters:</p>
<ul>
<li>NAR files live in S3-compatible object storage</li>
<li>Cache definitions, ACLs, and token state live in PostgreSQL</li>
<li>The cache server stays stateless apart from its DB and signing key</li>
</ul>
<p>For a single-node deployment, Garage with SQLite is enough. If you need more later, you can grow from there without replacing the cache server.</p>
<h2 id="garage-s3-backend-for-nar-storage">Garage: S3 backend for NAR storage</h2>
<p>Garage is the object store Attic writes to. The NixOS side is straightforward: run Garage locally, expose the S3 API through nginx, and bootstrap the initial layout, S3 key, and bucket with an idempotent oneshot service.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{ config, pkgs, ... }:
let
garageS3Host = "s3.example.com";
garageZone = "est1";
garageCapacity = "1T";
garageBootstrap = pkgs.writeShellScript "garage-bootstrap" ''
set -euo pipefail
for _ in $(seq 1 30); do
if node_id="$(${config.services.garage.package}/bin/garage node id 2>/dev/null | cut -d@ -f1)" && [ -n "$node_id" ]; then
break
fi
sleep 1
done
if [ -z "''${node_id:-}" ]; then
echo "garage node id did not become available in time" >&2
exit 1
fi
if ${config.services.garage.package}/bin/garage status | grep -q 'NO ROLE ASSIGNED'; then
${config.services.garage.package}/bin/garage layout assign -z ${garageZone} -c ${garageCapacity} "$node_id"
current_version="$(${config.services.garage.package}/bin/garage layout show | sed -n 's/^Current cluster layout version: \([0-9][0-9]*\)$/\1/p')"
if [ -n "$current_version" ]; then
next_version=$((current_version + 1))
else
next_version=1
fi
${config.services.garage.package}/bin/garage layout apply --version "$next_version"
fi
if ! ${config.services.garage.package}/bin/garage key info "$(cat ${config.sops.secrets.garage_attic_key_id.path})" >/dev/null 2>&1; then
${config.services.garage.package}/bin/garage key import \
"$(cat ${config.sops.secrets.garage_attic_key_id.path})" \
"$(cat ${config.sops.secrets.garage_attic_secret_key.path})" \
-n attic \
--yes
fi
if ! ${config.services.garage.package}/bin/garage bucket info attic >/dev/null 2>&1; then
${config.services.garage.package}/bin/garage bucket create attic
fi
${config.services.garage.package}/bin/garage bucket allow \
--read \
--write \
--owner \
attic \
--key "$(cat ${config.sops.secrets.garage_attic_key_id.path})"
'';
in
{
users.groups.garage = { };
users.users.garage = {
isSystemUser = true;
group = "garage";
description = "Garage object storage service";
};
sops.secrets.garage_rpc_secret = {
mode = "0400";
owner = "garage";
group = "garage";
};
sops.secrets.garage_admin_token = {
mode = "0400";
owner = "garage";
group = "garage";
};
sops.secrets.garage_metrics_token = {
mode = "0400";
owner = "garage";
group = "garage";
};
sops.secrets.garage_attic_key_id = { };
sops.secrets.garage_attic_secret_key = { };
services.garage = {
enable = true;
package = pkgs.garage;
logLevel = "info";
settings = {
replication_factor = 1;
db_engine = "sqlite";
rpc_bind_addr = "127.0.0.1:3901";
rpc_public_addr = "127.0.0.1:3901";
rpc_secret_file = config.sops.secrets.garage_rpc_secret.path;
allow_world_readable_secrets = false;
s3_api = {
api_bind_addr = "127.0.0.1:3900";
s3_region = "garage";
};
admin = {
api_bind_addr = "127.0.0.1:3903";
admin_token_file = config.sops.secrets.garage_admin_token.path;
metrics_token_file = config.sops.secrets.garage_metrics_token.path;
};
};
};
systemd.services.garage.serviceConfig = {
DynamicUser = false;
User = "garage";
Group = "garage";
};
services.nginx.virtualHosts.${garageS3Host} = {
forceSSL = true;
enableACME = true;
acmeRoot = null;
locations."/" = {
proxyPass = "http://127.0.0.1:3900";
extraConfig = ''
client_max_body_size 0;
proxy_request_buffering off;
proxy_buffering off;
'';
};
};
systemd.services.garage-bootstrap = {
description = "Bootstrap Garage layout and Attic bucket";
wantedBy = [ "multi-user.target" ];
after = [ "garage.service" ];
requires = [ "garage.service" ];
path = with pkgs; [
coreutils
gnugrep
gnused
];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
ExecStart = garageBootstrap;
};
};
}
</code></pre>
<p>The important part is the bootstrap script. It makes the single-node layout assignment, imports the access key Attic will use, creates the <code>attic</code> bucket, and grants that key ownership of the bucket. Because it’s idempotent, you can leave it in the boot path without fear.</p>
<h2 id="attic-cache-server-metadata-and-tokens">Attic: cache server, metadata, and tokens</h2>
<p>Attic sits in front of Garage and PostgreSQL. It serves the binary cache API, signs cache metadata, and issues scoped JWTs for pushing and administration.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{ config, pkgs, ... }:
let
atticApiHost = "nix-cache.example.com";
atticBootstrapCache = "myuser";
atticClient = "${pkgs.attic-client}/bin/attic";
atticEnv = config.sops.templates."atticd-env";
atticBootstrap = pkgs.writeShellScript "attic-bootstrap" ''
set -euo pipefail
state_root=/var/lib/attic-bootstrap
export HOME="$state_root"
export XDG_CONFIG_HOME="$state_root/xdg"
rm -rf "$XDG_CONFIG_HOME"
mkdir -p "$XDG_CONFIG_HOME"
token=$(/run/current-system/sw/bin/atticd-atticadm make-token \
--sub bootstrap \
--validity '10 years' \
--pull '*' \
--push '*' \
--delete '*' \
--create-cache '*' \
--configure-cache '*' \
--configure-cache-retention '*' \
--destroy-cache '*')
${atticClient} login bootstrap http://127.0.0.1:8080/ "$token"
if ! ${atticClient} cache info bootstrap:${atticBootstrapCache} >/dev/null 2>&1; then
${atticClient} cache create bootstrap:${atticBootstrapCache} --public
fi
${atticClient} cache configure bootstrap:${atticBootstrapCache} --public
${atticClient} cache info bootstrap:${atticBootstrapCache} > "$state_root/${atticBootstrapCache}.info"
'';
in
{
sops.secrets.attic_token_rs256_secret_base64 = {
mode = "0400";
owner = "root";
group = "root";
};
sops.templates."atticd-env" = {
content = ''
ATTIC_SERVER_TOKEN_RS256_SECRET_BASE64=${config.sops.placeholder.attic_token_rs256_secret_base64}
AWS_ACCESS_KEY_ID=${config.sops.placeholder.garage_attic_key_id}
AWS_SECRET_ACCESS_KEY=${config.sops.placeholder.garage_attic_secret_key}
'';
mode = "0400";
owner = "root";
group = "root";
};
services.postgresql.ensureDatabases = [ "atticd" ];
services.postgresql.ensureUsers = [
{
name = "atticd";
ensureDBOwnership = true;
}
];
services.atticd = {
enable = true;
package = pkgs.attic-server;
environmentFile = atticEnv.path;
settings = {
listen = "127.0.0.1:8080";
allowed-hosts = [
atticApiHost
"${atticBootstrapCache}.${atticApiHost}"
"127.0.0.1:8080"
"localhost:8080"
];
api-endpoint = "https://${atticApiHost}/";
substituter-endpoint = "https://${atticApiHost}/";
database.url = "postgres:///atticd?host=/run/postgresql&user=atticd";
storage = {
type = "s3";
region = "garage";
bucket = "attic";
endpoint = "https://s3.example.com";
};
};
};
systemd.services.atticd = {
after = [ "garage-bootstrap.service" ];
requires = [ "garage-bootstrap.service" ];
};
systemd.services.attic-bootstrap = {
description = "Bootstrap the initial public Attic cache";
wantedBy = [ "multi-user.target" ];
after = [ "atticd.service" ];
requires = [ "atticd.service" ];
path = with pkgs; [
coreutils
findutils
];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
ExecStart = atticBootstrap;
StateDirectory = "attic-bootstrap";
};
};
}
</code></pre>
<p>Three details matter here:</p>
<ol>
<li><code>services.atticd.settings.storage</code> points at Garage’s S3 endpoint, not local disk.</li>
<li>The Attic signing key and S3 credentials come from a <code>sops.templates</code> environment file, so the unit gets exactly the secrets it needs.</li>
<li><code>attic-bootstrap</code> logs into the local API with an all-powerful bootstrap token and creates the first cache declaratively.</li>
</ol>
<p>That last point is the difference between “deployed a service” and “deployed a usable cache.”</p>
<h2 id="the-boot-dependency-chain">The boot dependency chain</h2>
<p>The full boot ordering looks like this:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">garage.service
└─ garage-bootstrap.service
└─ atticd.service
└─ attic-bootstrap.service
</code></pre>
<p>Garage has to be up before the S3 bucket can be created. The bucket and access key have to exist before Attic can start cleanly. Attic has to be live before you can create the initial cache through its API. All of those bootstrap steps are <code>oneshot</code> units with <code>RemainAfterExit = true</code>, so they run once, stay “active”, and don’t flap on every check.</p>
<h2 id="secrets-with-sops-nix">Secrets with sops-nix</h2>
<p>This stack has six secrets:</p>
<table><thead><tr><th>Secret</th><th>Purpose</th></tr></thead><tbody>
<tr><td><code>garage_rpc_secret</code></td><td>Garage RPC authentication</td></tr>
<tr><td><code>garage_admin_token</code></td><td>Garage admin API authentication</td></tr>
<tr><td><code>garage_metrics_token</code></td><td>Garage metrics API authentication</td></tr>
<tr><td><code>garage_attic_key_id</code></td><td>S3 access key for Attic</td></tr>
<tr><td><code>garage_attic_secret_key</code></td><td>S3 secret key for Attic</td></tr>
<tr><td><code>attic_token_rs256_secret_base64</code></td><td>Attic JWT signing key</td></tr>
</tbody></table>
<p>The nice part is the direction of flow:</p>
<ul>
<li>Garage consumes the RPC and admin secrets directly</li>
<li>Garage bootstrap consumes the S3 key pair to import it and grant bucket access</li>
<li>Attic consumes the JWT secret and the same S3 key pair through one rendered environment file</li>
</ul>
<p>No plaintext credentials in git. No hand-written <code>Environment=</code> lines in systemd units.</p>
<h2 id="multi-tenant-token-scoping-is-the-whole-point">Multi-tenant token scoping is the whole point</h2>
<p>If all you wanted was a single shared cache, you could stop at “CI can push and clients can pull.” The more interesting setup is when multiple teams or projects share one Attic server without sharing one namespace.</p>
<p>Attic’s token generator gives you that directly.</p>
<p>For one personal cache:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">atticd-atticadm make-token \
--sub myuser-push \
--validity '1 year' \
--pull myuser \
--push myuser
</code></pre>
<p>That token can read and push exactly one cache: <code>myuser</code>.</p>
<p>For a team namespace:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">atticd-atticadm make-token \
--sub teamname \
--validity '1 year' \
--pull 'teamname-*' \
--push 'teamname-*' \
--create-cache 'teamname-*' \
--configure-cache 'teamname-*' \
--configure-cache-retention 'teamname-*'
</code></pre>
<p>This is the private Cachix pattern. One token, one namespace prefix. The team can create <code>teamname-dev</code>, <code>teamname-staging</code>, and <code>teamname-prod</code> on demand. They can push to them, configure them, and rotate retention settings. They cannot touch <code>otherteam-*</code>.</p>
<p>That gives you one shared Attic deployment with isolation by scoped capability rather than by running one cache server per team.</p>
<h2 id="client-side-nix-configuration">Client-side Nix configuration</h2>
<p>Clients only need the global Attic endpoint and the public key for the cache they consume:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{ ... }:
{
nix.settings = {
substituters = [
"https://nix-cache.example.com/myuser"
];
trusted-public-keys = [
"myuser:55EJTBFbq5pCYx2tf+aR8pmVPvCmP7QlafHH90/kikw="
];
};
}
</code></pre>
<p>That’s valid for NixOS and nix-darwin alike. If you have multiple team caches, add multiple path-based substituters and their public keys.</p>
<p>The thing to remember is that the cache name lives in the path. Attic does not currently give you a first-class “one hostname per cache” model through <code>attic use</code>, so don’t design around vanity subdomains here.</p>
<h2 id="ci-push-trusted-build-outputs-never-pr-outputs">CI: push trusted build outputs, never PR outputs</h2>
<p>The cache gets interesting when CI writes to it automatically. The safe pattern is:</p>
<ul>
<li>pull from the cache on every build</li>
<li>push to the cache only on trusted events</li>
<li>keep pull requests read-only to avoid cache poisoning</li>
</ul>
<p>Here’s a GitHub Actions workflow that does exactly that:</p>
<pre data-lang="yaml" class="language-yaml "><code class="language-yaml" data-lang="yaml">name: "Build and populate cache"
on:
pull_request:
push:
workflow_dispatch:
schedule:
- cron: '42 5 * * *'
jobs:
tests:
strategy:
matrix:
nurRepo:
- myuser
nixPath:
- nixpkgs=channel:nixos-unstable
- nixpkgs=channel:nixpkgs-unstable
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Install nix
uses: cachix/install-nix-action@v31
with:
nix_path: "${{ matrix.nixPath }}"
extra_nix_config: |
experimental-features = nix-command flakes
access-tokens = github.com=${{ secrets.GITHUB_TOKEN }}
extra-substituters = https://nix-cache.example.com/myuser
extra-trusted-public-keys = myuser:55EJTBFbq5pCYx2tf+aR8pmVPvCmP7QlafHH90/kikw=
- name: Show nixpkgs version
run: nix-instantiate --eval -E '(import <nixpkgs> {}).lib.version'
- name: Login to Attic
if: github.event_name != 'pull_request'
run: nix shell nixpkgs#attic-client -c attic login ci https://nix-cache.example.com/ "${{ secrets.ATTIC_TOKEN }}"
- name: Build nix packages
run: nix shell nixpkgs#nix-build-uncached -c nix-build-uncached ci.nix -A cacheOutputs
- name: Push build outputs to Attic
if: github.event_name != 'pull_request'
run: nix shell nixpkgs#attic-client -c sh -lc 'attic push ci:myuser result*'
- name: Trigger NUR update
if: ${{ matrix.nurRepo != '<YOUR_REPO_NAME>' }}
run: curl -XPOST "https://nur-update.nix-community.org/update?repo=${{ matrix.nurRepo }}"
</code></pre>
<p>The key behavior:</p>
<ul>
<li>PRs can use the cache as a substituter, but they do not get an Attic login token</li>
<li>pushes, scheduled builds, and manual runs can log in and push results</li>
<li><code>ATTIC_TOKEN</code> should be a scoped token, not a bootstrap token</li>
<li><code>nix-build-uncached</code> avoids pointlessly rebuilding outputs that are already available from substituters</li>
</ul>
<p>If you’re handing out CI tokens per repository or per team, use the same wildcard scoping model as the human tokens. <code>repo-a-*</code> and <code>repo-b-*</code> stay isolated even though they hit the same server.</p>
<h2 id="operating-model">Operating model</h2>
<p>Once this is deployed, the workflow is simple:</p>
<ol>
<li>Garage stores the blobs.</li>
<li>Attic serves the cache API and tracks metadata in PostgreSQL.</li>
<li>Teams get scoped tokens limited to their namespace.</li>
<li>Developers add the cache URL and public key to <code>nix.settings</code>.</li>
<li>CI logs in on trusted runs and pushes build outputs.</li>
</ol>
<p>That gives you most of what people actually want from Cachix:</p>
<ul>
<li>faster CI</li>
<li>faster local builds</li>
<li>a shared cache for private code</li>
<li>scoped write access</li>
<li>one central service instead of ad-hoc per-project buckets</li>
</ul>
<p>But you keep control of the storage, keys, and tenancy model.</p>
<h2 id="one-limitation-worth-planning-around">One limitation worth planning around</h2>
<p>Attic still exposes one global <code>substituter-endpoint</code>. <code>attic use <cache></code> advertises a path-based URL like <code>https://nix-cache.example.com/myuser</code>, not a dedicated hostname per cache. So if you’re thinking “I’ll front each cache with its own vanity nginx vhost,” stop there. That’s not the product surface Attic exposes today.</p>
<p>Path-based cache names are the stable interface. Design your client config, CI config, and team docs around that and the setup stays boring in the good way.</p>
<h2 id="final-thought">Final thought</h2>
<p>This stack hits a nice balance. Garage is lightweight enough for single-node self-hosting. Attic gives you a real multi-user cache server instead of raw object storage. PostgreSQL handles the metadata cleanly. And scoped tokens let you share one service across teams without turning it into a free-for-all.</p>
<p>If you already run NixOS and sops-nix, the whole thing fits naturally into the rest of your infrastructure. Which is the main reason to do it this way in the first place.</p>
Nix-Wrapped Android Release Builds with Out-of-Repo Signing2026-03-30T12:00:00+00:002026-03-30T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/nix-wrapped-android-release-builds/<p>Android release builds have three annoyances that like to get tangled together:</p>
<ol>
<li>The Android SDK is huge, version-sensitive, and annoying to install consistently.</li>
<li>Release signing needs a keystore and credentials that absolutely should not live in the repo.</li>
<li>Flutter build commands tend to accrete tribal shell setup until nobody remembers what is actually required.</li>
</ol>
<p>Nix solves the first and third problems nicely. For the second, the pattern I like is simple: keep signing material under <code>~/.config/your-app/</code>, and have the flake symlink it into the place Gradle expects at build time.</p>
<p>That gives you a release workflow where:</p>
<ul>
<li>the SDK version is pinned</li>
<li>the build command is repeatable</li>
<li>the signing config stays outside git</li>
<li>new machines bootstrap from <code>nix develop</code> instead of a README full of imperative setup</li>
</ul>
<p>Here’s the pattern.</p>
<h2 id="the-shape-of-it">The shape of it</h2>
<p>The repo stays clean. Secrets live in your home directory:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">~/.config/myapp/
├── android-signing.properties
└── upload.jks
myapp/
├── flake.nix
├── android/
│ ├── .gitignore
│ └── app/build.gradle.kts
└── ...
</code></pre>
<p><code>android-signing.properties</code> contains the passwords and points at the keystore using an absolute path. The repo never stores either file. The build script links the properties file into <code>android/key.properties</code> right before running the release build.</p>
<p>That one decision removes most of the risk. You stop playing games with encrypted blobs in app repos, and Gradle still gets the file layout it expects.</p>
<h2 id="step-1-pin-the-android-sdk-in-the-flake">Step 1: pin the Android SDK in the flake</h2>
<p>The core idea is to compose the Android SDK declaratively in <code>flake.nix</code>, then export the usual Android environment variables from the dev shell:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">devShells = forAllSystems (system:
let
pkgs = import nixpkgs {
inherit system;
config = {
allowUnfree = true;
android_sdk.accept_license = true;
};
};
androidComposition =
pkgs.androidenv.composeAndroidPackages {
buildToolsVersions = [ "28.0.3" "35.0.0" ];
platformVersions = [ "36" "35" "34" "33" ];
includeNDK = true;
ndkVersions = [ "28.2.13676358" ];
cmakeVersions = [ "3.22.1" ];
includeEmulator = true;
includeSystemImages = true;
abiVersions = [ "arm64-v8a" "x86_64" ];
systemImageTypes = [ "google_apis" ];
includeSources = false;
};
androidSdk = androidComposition.androidsdk;
in {
default = pkgs.mkShellNoCC {
packages = [
pkgs.flutter
androidSdk
pkgs.jdk17
release-android
];
shellHook = ''
export ANDROID_HOME="${androidSdk}/libexec/android-sdk"
export ANDROID_SDK_ROOT="$ANDROID_HOME"
export JAVA_HOME="${pkgs.jdk17}"
'';
};
});
</code></pre>
<p>The important bits:</p>
<ul>
<li><code>android_sdk.accept_license = true</code> has to be set in <code>config</code>, or the SDK composition fails</li>
<li><code>ANDROID_HOME</code>, <code>ANDROID_SDK_ROOT</code>, and <code>JAVA_HOME</code> should all be exported explicitly</li>
<li><code>buildToolsVersions</code>, <code>platformVersions</code>, <code>ndkVersions</code>, and <code>cmakeVersions</code> should be pinned instead of left implicit</li>
</ul>
<p>This is the whole point of using Nix here. You are replacing “install whatever Android Studio pulls today” with a concrete, reviewable SDK definition.</p>
<h2 id="step-2-keep-signing-config-outside-the-repo">Step 2: keep signing config outside the repo</h2>
<p>Under <code>~/.config/myapp/</code>, create a properties file like this:</p>
<pre data-lang="properties" class="language-properties "><code class="language-properties" data-lang="properties">storePassword=your-store-password
keyPassword=your-key-password
keyAlias=upload
storeFile=/Users/yourname/.config/myapp/upload.jks
</code></pre>
<p>Then generate the keystore once:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">keytool -genkey -v -keystore ~/.config/myapp/upload.jks \
-keyalg RSA -keysize 2048 -validity 10000 -alias upload
</code></pre>
<p>Two details matter here:</p>
<ul>
<li><code>storeFile</code> should be an absolute path</li>
<li>both files should live somewhere user-private, not under the project tree</li>
</ul>
<p>You can move these wherever you like, but <code>~/.config/myapp/</code> is an easy convention to remember and document.</p>
<h2 id="step-3-let-gradle-read-key-properties-if-it-exists">Step 3: let Gradle read <code>key.properties</code> if it exists</h2>
<p>On the Gradle side, the app should load <code>android/key.properties</code> when present and define the release signing config from it.</p>
<p>In <code>android/app/build.gradle.kts</code>:</p>
<pre data-lang="kotlin" class="language-kotlin "><code class="language-kotlin" data-lang="kotlin">import java.io.FileInputStream
import java.util.Properties
val keystoreProperties = Properties()
val keystorePropertiesFile = rootProject.file("key.properties")
if (keystorePropertiesFile.exists()) {
keystoreProperties.load(FileInputStream(keystorePropertiesFile))
}
android {
signingConfigs {
if (keystorePropertiesFile.exists()) {
create("release") {
keyAlias = keystoreProperties["keyAlias"] as String
keyPassword = keystoreProperties["keyPassword"] as String
storeFile = file(keystoreProperties["storeFile"] as String)
storePassword = keystoreProperties["storePassword"] as String
}
}
}
buildTypes {
release {
signingConfig = if (keystorePropertiesFile.exists()) {
signingConfigs.getByName("release")
} else {
signingConfigs.getByName("debug")
}
}
}
}
</code></pre>
<p>And in <code>android/.gitignore</code>:</p>
<pre data-lang="gitignore" class="language-gitignore "><code class="language-gitignore" data-lang="gitignore">key.properties
</code></pre>
<p>The fallback to <code>debug</code> signing is optional, but useful. It means the project still builds for local testing even when the real release credentials are absent.</p>
<h2 id="step-4-wrap-the-release-build-in-nix">Step 4: wrap the release build in Nix</h2>
<p>Now the useful part: hide the shell setup and symlink dance behind a single command.</p>
<p>I like doing this with <code>writeShellScriptBin</code>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">release-android =
pkgs.writeShellScriptBin "release-android" ''
set -euo pipefail
export ANDROID_HOME="${androidSdk}/libexec/android-sdk"
export ANDROID_SDK_ROOT="$ANDROID_HOME"
export JAVA_HOME="${pkgs.jdk17}"
root="$(${pkgs.git}/bin/git rev-parse --show-toplevel 2>/dev/null || pwd)"
signing="$HOME/.config/myapp/android-signing.properties"
if [ ! -f "$signing" ]; then
echo "error: signing config not found" >&2
echo >&2
echo "Create $signing with:" >&2
echo " storePassword=<your store password>" >&2
echo " keyPassword=<your key password>" >&2
echo " keyAlias=upload" >&2
echo " storeFile=/absolute/path/to/keystore.jks" >&2
exit 1
fi
ln -sf "$signing" "$root/android/key.properties"
echo "linked signing config"
cd "$root"
flutter clean
flutter pub get
flutter build appbundle --release \
--dart-define=GRPC_HOST=api.prod.example.com \
--dart-define=GRPC_PORT=443 \
--dart-define=GRPC_USE_TLS=true \
"$@"
echo
echo "Bundle: build/app/outputs/bundle/release/app-release.aab"
'';
</code></pre>
<p>This script does four things:</p>
<ol>
<li>Exports the Android and Java paths expected by Flutter and Gradle.</li>
<li>Verifies that the signing config exists before doing any expensive work.</li>
<li>Symlinks the external properties file into <code>android/key.properties</code>.</li>
<li>Runs the release build with whatever <code>--dart-define</code> values production needs.</li>
</ol>
<p>That is a much better interface than “first open the right shell, then export three variables, then remember the path to the signing file, then run the exact build incantation.”</p>
<p>It also means CI or another developer can run the same command and get the same behaviour, assuming they have their own signing material configured.</p>
<h2 id="step-5-add-emulator-helpers-while-you-re-here">Step 5: add emulator helpers while you’re here</h2>
<p>Once you’re already pinning the SDK, adding emulator launchers is cheap and useful.</p>
<p>Here’s a small helper that creates an AVD only if it doesn’t already exist, then launches it:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">abi = if pkgs.stdenv.hostPlatform.isAarch64
then "arm64-v8a"
else "x86_64";
mkEmulator = { name, avdName, device, apiLevel, desc }:
pkgs.writeShellScriptBin name ''
set -euo pipefail
export ANDROID_HOME="${androidSdk}/libexec/android-sdk"
export ANDROID_SDK_ROOT="$ANDROID_HOME"
export PATH="${pkgs.jdk17}/bin:$PATH"
AVD_NAME="${avdName}"
SYSTEM_IMAGE="system-images;android-${apiLevel};google_apis;${abi}"
if ! "$ANDROID_HOME/emulator/emulator" -list-avds 2>/dev/null \
| grep -qx "$AVD_NAME"; then
echo "Creating ${desc}..."
echo "no" | "$ANDROID_HOME"/cmdline-tools/*/bin/avdmanager \
create avd -n "$AVD_NAME" -k "$SYSTEM_IMAGE" \
-d "${device}" --force
fi
echo "Launching ${desc}..."
exec "$ANDROID_HOME/emulator/emulator" -avd "$AVD_NAME" "$@"
'';
emu-phone = mkEmulator {
name = "emu-phone";
avdName = "myapp_phone";
device = "pixel_7";
apiLevel = "35";
desc = "Pixel 7 API 35";
};
emu-tablet = mkEmulator {
name = "emu-tablet";
avdName = "myapp_tablet";
device = "pixel_tablet";
apiLevel = "34";
desc = "Pixel Tablet API 34";
};
</code></pre>
<p>The nice thing here is not just convenience. It’s that the emulator definition becomes part of the same reproducible environment as the SDK itself. Team members are not manually creating mystery AVDs with slightly different images and wondering why behaviour differs.</p>
<h2 id="what-usage-looks-like">What usage looks like</h2>
<p>From a clean machine, the workflow becomes:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># Enter the project environment
nix develop
# Run an emulator
emu-phone
# Build the signed Android app bundle
release-android
</code></pre>
<p>The result lands at:</p>
<pre data-lang="text" class="language-text "><code class="language-text" data-lang="text">build/app/outputs/bundle/release/app-release.aab
</code></pre>
<p>That’s the artifact you upload to Google Play.</p>
<h2 id="why-this-pattern-is-worth-keeping">Why this pattern is worth keeping</h2>
<p>The main win is not “you can build Android with Nix.” You can do that a dozen messy ways. The win is that responsibilities get separated cleanly:</p>
<ul>
<li>Nix owns SDK provisioning and wrapper scripts</li>
<li>Gradle owns signing configuration and release build logic</li>
<li>your home directory owns secrets</li>
<li>the repo stays free of machine-local credentials</li>
</ul>
<p>That separation scales better than the usual alternative, where every developer has a slightly different Android setup and the release process is half shell history, half folklore.</p>
<p>If you’re already using flakes for your dev environment, Android release builds fit into that model surprisingly well. Put the SDK in the flake, put the signing material under <code>~/.config</code>, wrap the build in one script, and stop treating release day as a special machine ceremony.</p>
Prometheus Ecowitt Exporter on NixOS2026-03-29T12:00:00+00:002026-03-29T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/prometheus-ecowitt-exporter-nixos/<p>Ecowitt weather stations are solid hardware. The sensors are reliable, the gateways are cheap, and the ecosystem covers everything from soil moisture to lightning detection. The cloud platform, though — that’s where it falls apart. Limited retention, clunky dashboards, no alerting worth mentioning, and your data sitting on someone else’s servers. You already run Prometheus and Grafana for everything else. You just need a bridge.</p>
<p><a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/prometheus-ecowitt-exporter">prometheus-ecowitt-exporter</a> is that bridge. It’s a Rust service that receives HTTP POSTs from your Ecowitt gateway, converts the readings into Prometheus metrics, and exposes them on a <code>/metrics</code> endpoint. It can also forward the raw data to other HTTP endpoints — Home Assistant, a second exporter, anything that accepts an HTTP POST — working around the gateway’s single-destination limitation. It ships with a NixOS module that wires everything up — including optional Prometheus scrape config and a Grafana dashboard.</p>
<h2 id="how-the-data-flows">How the data flows</h2>
<p>The path from sensor to graph looks like this:</p>
<pre><code>Ecowitt sensors
↓ RF 433/868MHz
Gateway (GW1100, etc.)
↓ HTTP POST
prometheus-ecowitt-exporter ──→ Home Assistant, other receivers
↓ /metrics
Prometheus
↓ query
Grafana
</code></pre>
<p>Your sensors transmit over RF to the gateway. The gateway — typically a GW1100 or similar — supports “customized” weather services, which is really just an HTTP POST to an arbitrary endpoint on an interval you choose. The exporter receives those posts, parses the Ecowitt protocol, applies unit conversions, and serves the results as Prometheus metrics. It can also forward the raw data to other HTTP endpoints — more on that below. Standard scrape-and-graph from there.</p>
<p>No polling. No API keys. No cloud dependency. The gateway pushes directly to your host.</p>
<h2 id="getting-the-module">Getting the module</h2>
<p>You have two options for pulling this into your NixOS configuration: directly from the flake, or through NUR.</p>
<h3 id="direct-flake-input">Direct flake input</h3>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
prometheus-ecowitt-exporter.url = "github:ijohanne/prometheus-ecowitt-exporter";
};
outputs = { nixpkgs, prometheus-ecowitt-exporter, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
modules = [
prometheus-ecowitt-exporter.nixosModules.default
./configuration.nix
];
};
};
}
</code></pre>
<p>This pins you to a specific revision of the exporter. You control when you update. Straightforward.</p>
<h3 id="via-nur">Via NUR</h3>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs.nur.url = "github:nix-community/NUR";
outputs = { self, nixpkgs, nur, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
modules = [
nur.modules.nixos.repos.ijohanne.prometheus-ecowitt-exporter
./configuration.nix
];
};
};
}
</code></pre>
<p>Same module, different delivery mechanism. If you already use NUR for other packages, this keeps your inputs list shorter.</p>
<h2 id="nixos-configuration">NixOS configuration</h2>
<p>Here’s a full configuration with all the knobs visible:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
services.prometheus-ecowitt-exporter = {
enable = true;
port = 8088;
temperatureUnit = "c";
pressureUnit = "hpa";
windUnit = "kmh";
rainUnit = "mm";
distanceUnit = "km";
irradianceUnit = "wm2";
aqiStandard = "epa";
outdoorLocation = "garden";
indoorLocation = "living-room";
forwardUrls = [
"http://homeassistant.local:8123/api/webhook/ecowitt"
];
enableLocalScraping = true;
enableGrafanaDashboard = true;
};
}
</code></pre>
<p>Two options deserve special attention. <code>enableLocalScraping</code> automatically adds a Prometheus scrape target for the exporter — no need to manually edit your <code>scrapeConfigs</code>. <code>enableGrafanaDashboard</code> provisions a Grafana dashboard that covers all the metric types the exporter produces. Both save you the tedium of wiring things together by hand.</p>
<p>The service runs as a dedicated <code>ecowitt-exporter</code> user by default. You can override <code>user</code> and <code>group</code> if you have opinions about that. The <code>listenAddress</code> defaults to <code>0.0.0.0</code> — change it if you want to bind to a specific interface.</p>
<h2 id="unit-conversions">Unit conversions</h2>
<p>The exporter handles all unit conversion server-side. You configure it once and every metric comes out in the units you actually want.</p>
<table><thead><tr><th>Measurement</th><th>Default</th><th>Options</th></tr></thead><tbody>
<tr><td>Temperature</td><td>c</td><td>c, f, k</td></tr>
<tr><td>Pressure</td><td>hpa</td><td>hpa, inhg, mmhg</td></tr>
<tr><td>Wind</td><td>kmh</td><td>kmh, mph, ms, knots, fps</td></tr>
<tr><td>Rain</td><td>mm</td><td>mm, in</td></tr>
<tr><td>Distance</td><td>km</td><td>km, mi</td></tr>
<tr><td>Irradiance</td><td>wm2</td><td>wm2, lx, fc</td></tr>
<tr><td>AQI Standard</td><td>epa</td><td>uk, epa, mep, nepm</td></tr>
</tbody></table>
<p>The AQI standard option is worth noting — it changes the algorithm used to calculate the air quality index from PM2.5 readings. UK, EPA, Chinese MEP, and Australian NEPM standards are all available. Pick whichever one your local regulatory body uses.</p>
<h2 id="multi-station-support">Multi-station support</h2>
<p>Each Ecowitt gateway posts to a URL path you configure, and that path segment becomes the <code>station</code> label on every metric:</p>
<pre><code>POST /report/garden → station="garden"
POST /report/rooftop → station="rooftop"
</code></pre>
<p>One exporter instance handles multiple stations. No duplicate deployments, no port juggling, no metric collisions. You just point each gateway at a different path.</p>
<h2 id="forwarding-to-other-receivers">Forwarding to other receivers</h2>
<p>Ecowitt gateways only support a single custom server destination. That’s a problem if you want data in both Prometheus and Home Assistant — or any other system that speaks the Ecowitt protocol. The exporter solves this by acting as a fan-out relay.</p>
<p>The <code>forwardUrls</code> option takes a list of URLs. Every time the exporter receives a POST from the gateway, it forwards the raw request body to each URL. The forwarding is fire-and-forget — the exporter doesn’t wait for responses and won’t block metric processing if a downstream receiver is slow or unreachable.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.prometheus-ecowitt-exporter = {
forwardUrls = [
"http://homeassistant.local:8123/api/webhook/ecowitt"
"https://backup-exporter.lan:8088/report/garden"
];
};
</code></pre>
<p>Self-signed certificates are accepted — TLS verification is disabled on the forwarding client, so you don’t need to mess with CA bundles for internal services.</p>
<p>On the CLI, pass <code>--forward-url</code> once per destination:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">prometheus-ecowitt-exporter \
--forward-url http://homeassistant.local:8123/api/webhook/ecowitt \
--forward-url https://other-server:8088/report/mystation
</code></pre>
<p>The exporter tracks forwarding health via two Prometheus counters: <code>ecowitt_forward_total</code> and <code>ecowitt_forward_errors</code>, both labeled by destination URL. The Grafana dashboard includes a “Forwarding” panel that shows the rate of each per URL, so you’ll know immediately if a downstream receiver starts failing.</p>
<h2 id="sensor-location-labels">Sensor location labels</h2>
<p>Multi-channel temperature and humidity sensors — the WN31, WH31, and similar — show up as numbered channels. Channel numbers are not particularly descriptive in a dashboard. The <code>tempLocations</code> option maps them to human-readable labels:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.prometheus-ecowitt-exporter = {
tempLocations = {
"1" = "greenhouse";
"2" = "garage";
"3" = "pool";
};
};
</code></pre>
<p>These strings become the <code>location</code> label on <code>ecowitt_temp</code> and <code>ecowitt_humidity</code> metrics. Your Grafana panels say “greenhouse” instead of “channel 1.”</p>
<h2 id="configuring-the-weather-station-gateway">Configuring the weather station gateway</h2>
<p>The gateway needs to know where to send its data. You configure this through the WSView Plus app on your phone:</p>
<ol>
<li>Open WSView Plus and go to Device List</li>
<li>Select your gateway</li>
<li>Navigate to Weather Services, then Customized</li>
<li>Set Enable to On</li>
<li>Set Protocol Type to Ecowitt</li>
<li>Enter your NixOS host’s IP address as the Server IP/Hostname</li>
<li>Set the Path to <code>/report/mystationname</code> — pick a name that makes sense as a Prometheus label</li>
<li>Set Port to <code>8088</code> (or whatever you configured)</li>
<li>Set Upload Interval to 60 seconds</li>
</ol>
<p>That last setting controls how often you get new data points. Sixty seconds is a reasonable default. You can go lower, but your weather doesn’t change that fast.</p>
<h2 id="what-metrics-you-get">What metrics you get</h2>
<p>The exporter produces a comprehensive set of metrics. Here’s what each family covers:</p>
<ul>
<li><code>ecowitt_temp</code> — temperature readings from outdoor, indoor, and up to eight multi-channel sensors, with <code>station</code>, <code>sensor</code>, <code>unit</code>, and <code>location</code> labels</li>
<li><code>ecowitt_humidity</code> — outdoor, indoor, and eight channels</li>
<li><code>ecowitt_pressure</code> — barometric pressure in both absolute and relative variants, plus vapor pressure deficit</li>
<li><code>ecowitt_windspeed</code> and <code>ecowitt_windspeed_beaufort</code> — current speed, gust, max daily gust, and wind direction</li>
<li><code>ecowitt_rain</code> — hourly, daily, weekly, monthly, yearly, and event totals for both standard and WS90 piezo rain sensors</li>
<li><code>ecowitt_solar_radiation</code> and <code>ecowitt_uv_index</code> — solar metrics</li>
<li><code>ecowitt_lightning_distance</code> and <code>ecowitt_lightning_count</code> — lightning detection data</li>
<li><code>ecowitt_pm25</code> and <code>ecowitt_aqi</code> — PM2.5 concentration with 24-hour averages, plus calculated AQI under your chosen standard</li>
<li><code>ecowitt_soil_moisture</code> — multiple channels with raw voltage readings</li>
<li><code>ecowitt_battery</code> — status, voltage, and level for every sensor type</li>
<li><code>ecowitt_forward_total</code> and <code>ecowitt_forward_errors</code> — counters tracking forwarded requests and failures per destination URL</li>
</ul>
<p>Every metric carries appropriate labels so you can filter and aggregate in PromQL without ambiguity.</p>
<h2 id="grafana-dashboard">Grafana dashboard</h2>
<p>When you set <code>enableGrafanaDashboard = true</code>, the module provisions a Grafana dashboard automatically through Grafana’s provisioning system. It covers all the metric families listed above with sensible panel layouts. You get graphs, gauges, and stat panels — the kind of thing that would take an afternoon to build by hand.</p>
<p>If you want to customize it, the provisioned dashboard is a starting point. Clone it in Grafana and edit from there.</p>
<h2 id="docker-alternative">Docker alternative</h2>
<p>Not on NixOS? The exporter builds as a standard Docker image:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">docker build -t prometheus-ecowitt-exporter .
docker run -p 8088:8088 prometheus-ecowitt-exporter \
--temperature-unit c --pressure-unit hpa
</code></pre>
<p>You lose the automatic Prometheus and Grafana integration, but the exporter itself works the same way. Point your gateway at the container’s IP and port, configure your Prometheus scrape target manually, and import a dashboard.</p>
<h2 id="testing-with-curl">Testing with curl</h2>
<p>You don’t need a weather station connected to verify the exporter is working. Fake a gateway POST:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">curl -X POST http://localhost:8088/report/test \
-d "tempf=72.5&humidity=45&baromrelin=29.92&windspeedmph=5.6&dailyrainin=0.02"
</code></pre>
<p>Then check the metrics endpoint:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">curl http://localhost:8088/metrics
</code></pre>
<p>You should see Prometheus-formatted metrics with the values you posted — converted to whatever units you configured. This is also useful for testing alerting rules before the next thunderstorm.</p>
<h2 id="supported-hardware">Supported hardware</h2>
<p>The exporter handles data from the full Ecowitt ecosystem:</p>
<ul>
<li><strong>WS2910</strong> — weather station</li>
<li><strong>GW1100</strong> — Wi-Fi gateway</li>
<li><strong>WS69 and WS90</strong> — sensor arrays</li>
<li><strong>WH41 and WH43</strong> — PM2.5 air quality sensors</li>
<li><strong>WH57</strong> — lightning detection sensor</li>
<li><strong>WN31, WH31, WN32, WH32</strong> — temperature and humidity sensors</li>
<li><strong>WN36</strong> — pool temperature sensor</li>
<li><strong>WH51</strong> — soil moisture meter</li>
</ul>
<p>If your sensor transmits to an Ecowitt gateway and the gateway can do a customized HTTP POST, the exporter will handle it.</p>
<h2 id="wrapping-up">Wrapping up</h2>
<p>The whole setup is a flake input, a module configuration, and a phone app setting. Your weather data stays local, updates every minute, and sits in the same Prometheus instance as the rest of your infrastructure metrics. You can alert on freezing temperatures the same way you alert on disk space. And if you need the data elsewhere — Home Assistant, a backup exporter, whatever — the forwarding feature turns the single-destination limitation of Ecowitt gateways into a non-issue. The <a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/prometheus-ecowitt-exporter">project is on GitHub</a> under MIT — contributions welcome.</p>
Crane: Nix Builds for Rust Without Losing cargo build2026-03-28T12:00:00+00:002026-03-28T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/crane-dual-workflow-nix-rust/<p>Half your team uses Nix. The other half just wants <code>cargo build</code>. Both are right.</p>
<p>The default way Nix builds Rust is painful. <code>buildRustPackage</code> hashes your dependency tree with <code>cargoHash</code>, which means every time you add a crate, you get a hash mismatch, go look up the new one, paste it in, and rebuild everything from scratch. It’s the kind of workflow that makes people mass-quit Nix.</p>
<p>Crane fixes this. It splits the build into two derivations — one for dependencies, one for your code — so changing a source file doesn’t re-download and re-compile 400 crates. And it doesn’t touch your <code>Cargo.toml</code> or project layout, so <code>cargo build</code> keeps working exactly as before.</p>
<h2 id="the-flake-skeleton">The flake skeleton</h2>
<p>Start with the inputs. Crane for the build, rust-overlay for toolchain control, pre-commit-hooks for formatting, flake-utils because you don’t want to write <code>system</code> four times.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
crane.url = "github:ipetkov/crane";
rust-overlay = {
url = "github:oxalica/rust-overlay";
inputs.nixpkgs.follows = "nixpkgs";
};
pre-commit-hooks = {
url = "github:cachix/pre-commit-hooks.nix";
inputs.nixpkgs.follows = "nixpkgs";
};
flake-utils.url = "github:numtide/flake-utils";
};
}
</code></pre>
<p>Nothing unusual here. The <code>follows</code> declarations keep the dependency tree shallow.</p>
<h2 id="toolchain-and-crane-setup">Toolchain and Crane setup</h2>
<p>rust-overlay gives you pinned toolchains without fighting nixpkgs versions. You pick stable, add the extensions you actually want, and hand it to Crane.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">rustToolchain = pkgs.rust-bin.stable.latest.default.override {
extensions = [ "rust-src" "clippy" "rustfmt" ];
};
craneLib = (crane.mkLib pkgs).overrideToolchain rustToolchain;
</code></pre>
<p><code>rust-src</code> is there for rust-analyzer. Without it, go-to-definition into std just gives you a wall of “source not available.”</p>
<h2 id="the-two-stage-build">The two-stage build</h2>
<p>This is the core pattern. Crane splits your build into a dependency layer and a source layer.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">src = craneLib.cleanCargoSource ./.;
commonArgs = {
inherit src;
strictDeps = true;
nativeBuildInputs = with pkgs; [ protobuf ];
PROTOC = "${pkgs.protobuf}/bin/protoc";
};
cargoArtifacts = craneLib.buildDepsOnly commonArgs;
my-service = craneLib.buildPackage (commonArgs // {
inherit cargoArtifacts;
});
</code></pre>
<p><code>buildDepsOnly</code> compiles every dependency in <code>Cargo.lock</code> and caches the result. <code>buildPackage</code> then takes those cached artifacts and only compiles your code. Change a line in <code>src/main.rs</code> and you skip the entire dependency build.</p>
<p><code>cleanCargoSource</code> strips non-Cargo files from the source — READMEs, CI configs, docs. This matters because Nix derivations rebuild when their inputs change. Without it, editing your README invalidates your build cache.</p>
<p>The <code>PROTOC</code> env var is there because protobuf codegen needs to find the compiler. If your project doesn’t use protobuf, drop both lines. The pattern is the same for any native dependency — declare it in <code>nativeBuildInputs</code>, set env vars if needed.</p>
<h2 id="private-git-dependencies">Private git dependencies</h2>
<p>Things get interesting when your <code>Cargo.toml</code> pulls crates from private repos. Crane calls <code>builtins.fetchGit</code> under the hood, which works fine on single-user Nix installs — your SSH keys are right there. On multi-user installs, including Determinate Nix, the nix daemon runs as its own user and has no access to your keys.</p>
<p>The fix is to pre-fetch private deps as flake inputs — the flake machinery handles authentication — and then tell Crane’s vendoring to use those pre-fetched sources instead of trying to fetch them itself.</p>
<p>First, add the dep as a <code>flake = false</code> input:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs = {
my-private-dep-src = {
url = "github:myorg/my-private-dep";
flake = false;
};
};
}
</code></pre>
<p>Then override the vendoring:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">cargoVendorDir = craneLib.vendorCargoDeps {
src = craneLib.cleanCargoSource ./.;
overrideVendorGitCheckout = lockMetadata: drv:
let
source = (builtins.head lockMetadata).source or "";
in
if builtins.match ".*my-private-dep.*" source != null then
drv.overrideAttrs (_: {
src = my-private-dep-src;
})
else
drv;
};
</code></pre>
<p>Then thread it into your build args:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">commonArgs = {
inherit src cargoVendorDir;
strictDeps = true;
# ...
};
</code></pre>
<h3 id="the-gotchas">The gotchas</h3>
<p>There are four things that will waste your afternoon if you don’t know them upfront.</p>
<p><strong><code>lockMetadata</code> is a list, not an attrset.</strong> The callback receives a list of lock entries. You need <code>(builtins.head lockMetadata).source</code> to get the git URL. Treating it as an attrset gives you an unhelpful type error deep in the Nix evaluator.</p>
<p><strong>Match on <code>source</code>, not on the input name.</strong> If you have multiple private git deps and write a blanket override, you’ll apply the wrong source to the wrong dep. The <code>source</code> field contains the git URL from <code>Cargo.lock</code> — match against it.</p>
<p><strong>The <code>else drv</code> fallback is essential.</strong> Without it, any git dependency that doesn’t match your condition returns <code>null</code>. Crane then silently produces a broken vendor directory. You’ll get a cargo error about missing crates with no indication that your Nix code is the problem.</p>
<p><strong>Use <code>drv.overrideAttrs</code>, not a fresh derivation.</strong> Crane’s vendor checkout derivation carries metadata that the rest of the pipeline depends on. Replacing it with <code>pkgs.runCommand</code> or similar drops that metadata and breaks downstream.</p>
<h2 id="the-devshell-cargo-build-just-works">The devShell — cargo build just works</h2>
<p>Crane provides <code>craneLib.devShell</code>, which drops you into a shell with the Rust toolchain and any extra packages you specify. Inside it, <code>cargo build</code>, <code>cargo test</code>, <code>cargo run</code> — all work exactly as they would outside Nix.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">devShells.default = craneLib.devShell {
checks = self.checks.${system};
packages = with pkgs; [ protobuf sqlite cargo-watch ];
PROTOC = "${pkgs.protobuf}/bin/protoc";
shellHook = ''
${self.checks.${system}.pre-commit.shellHook}
'';
};
</code></pre>
<p>The <code>checks</code> attribute wires up <code>nix flake check</code> outputs so they’re available in the shell. <code>packages</code> adds tools that aren’t Rust but that your project needs. The <code>shellHook</code> installs pre-commit hooks automatically — more on that below.</p>
<p>This is the key insight: the devShell gives Nix developers the standard Cargo workflow. They don’t run <code>nix build</code> during development. They run <code>cargo build</code>. Nix just sets up the environment.</p>
<h2 id="direnv-makes-it-transparent">direnv makes it transparent</h2>
<p>The entire <code>.envrc</code>:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">use flake
</code></pre>
<p>That’s it. Walk into the project directory and your shell has the right Rust version, protobuf, sqlite, whatever else is declared in the devShell. No <code>nix develop</code>, no manual activation. It just happens.</p>
<h2 id="the-non-nix-escape-hatch">The non-Nix escape hatch</h2>
<p>For developers who don’t have Nix installed, you need an alternative path. The approach that works: document the environment variables your devShell sets and provide a setup script that installs the same tools via rustup, brew, apt, whatever your team uses.</p>
<p>The shared interface is environment variables. <code>PROTOC</code> points at the protobuf compiler. <code>DATABASE_URL</code> points at the dev database. Nix sets these in the devShell; non-Nix developers set them manually or via a <code>setup.sh</code>. The Rust code doesn’t care where they came from.</p>
<p>This is what “dual workflow” actually means. Not two build systems — one codebase that works with or without Nix, with the build system being orthogonal to the development experience.</p>
<h2 id="local-path-overrides-and-the-trap">Local path overrides and the trap</h2>
<p>When you’re iterating on a private dependency locally, you don’t want to push, update the flake input, and rebuild. Cargo has path overrides for this:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">cargo build --config \
'patch."https://github.com/myorg/my-dep.git".my-dep.path="../my-dep"'
</code></pre>
<p>This tells Cargo to use your local checkout of <code>my-dep</code> instead of the git version. Works great for development.</p>
<p>The trap: never persist this in <code>.cargo/config.toml</code>. It works on your machine because <code>../my-dep</code> exists. Inside the Nix sandbox, that path doesn’t exist. Your <code>nix build</code> will fail with a confusing “path not found” error that has nothing to do with Nix — it’s Cargo looking for a directory that isn’t there.</p>
<p>Keep it on the command line. Or use a <code>.cargo/config.toml</code> that’s in <code>.gitignore</code>. Either way, don’t commit it.</p>
<h2 id="pre-commit-hooks">Pre-commit hooks</h2>
<p>Formatting arguments are a waste of everyone’s time. Automate them.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">pre-commit = pre-commit-hooks.lib.${system}.run {
src = ./.;
hooks = {
nixfmt.enable = true;
rustfmt.enable = true;
};
};
</code></pre>
<p>This runs <code>nixfmt</code> on <code>.nix</code> files and <code>rustfmt</code> on Rust files before every commit. The <code>shellHook</code> in the devShell installs the git hooks automatically, so Nix developers get them without thinking about it.</p>
<p>Non-Nix developers can install the same hooks via <code>pre-commit install</code> with a <code>.pre-commit-config.yaml</code> — same tools, different entry point.</p>
<h2 id="flake-checks-everything-in-one-command">Flake checks — everything in one command</h2>
<p>Wire up your checks so <code>nix flake check</code> runs the build, Clippy, and format verification in one shot.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">checks = {
inherit my-service;
inherit pre-commit;
my-service-clippy = craneLib.cargoClippy (commonArgs // {
inherit cargoArtifacts;
cargoClippyExtraArgs = "--all-targets -- -D warnings";
});
my-service-fmt = craneLib.cargoFmt { inherit src; };
};
</code></pre>
<p><code>inherit my-service</code> means the build itself is a check — if it doesn’t compile, the check fails. <code>cargoClippy</code> runs lints with <code>-D warnings</code> so any warning is a hard failure. <code>cargoFmt</code> verifies formatting without modifying files.</p>
<p>In CI, your entire pipeline is <code>nix flake check</code>. One command, fully hermetic, same result on every machine.</p>
<h2 id="clippy-pedantic">Clippy pedantic</h2>
<p>While you’re setting up lints, turn on Clippy’s pedantic preset. It catches things the default lints miss — redundant clones, needless borrows, missing docs on public items.</p>
<p>In your <code>Cargo.toml</code>:</p>
<pre data-lang="toml" class="language-toml "><code class="language-toml" data-lang="toml">[lints.clippy]
pedantic = { level = "warn", priority = -1 }
</code></pre>
<p>The <code>priority = -1</code> means pedantic lints are warnings, but you can still override individual lints at higher priority. Some pedantic lints are genuinely annoying — <code>module_name_repetitions</code> comes to mind — and you’ll want to <code>#[allow]</code> those. But the majority catch real issues.</p>
<h2 id="the-result">The result</h2>
<p>You end up with a Rust project that builds with <code>cargo build</code> for daily development and <code>nix build</code> for CI and deployment. The Cargo workflow is untouched — no wrapper scripts, no custom build commands, no “run this Nix thing instead.” Developers who don’t use Nix never need to know it’s there.</p>
<p>The Nix side gives you hermetic CI, binary caching, and NixOS deployment modules. Crane’s two-stage build means your CI isn’t re-compiling 400 crates on every push. And when something breaks, it breaks the same way on every machine, which — if you’ve spent enough time debugging “works on my laptop” — is actually a feature.</p>
Running a Private OCI Registry on NixOS with Zot2026-03-27T12:00:00+00:002026-03-27T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/private-oci-registry-nixos-zot/<p>Every container you push to Docker Hub is one <code>docker pull</code> rate limit away from ruining your Friday night deployment. You could pay for a hosted registry, but you already have perfectly good hardware sitting in a rack. What you need is something lightweight, OCI-native, and not beholden to anyone’s pricing page.</p>
<p>Zot fits the bill. It’s a single-binary, vendor-neutral OCI registry that speaks the distribution spec natively. No wrapping Docker’s registry in duct tape. No Java. It ships with a built-in UI, search, vulnerability scanning, and Prometheus metrics. And there’s a NixOS module that makes the whole thing declarative.</p>
<h2 id="adding-the-module">Adding the module</h2>
<p>The Zot NixOS module lives in <code>ijohanne/nur-packages</code>. You can pull it in two ways.</p>
<p>Through the NUR overlay:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs.nur.url = "github:nix-community/NUR";
outputs = { self, nixpkgs, nur, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
modules = [
nur.modules.nixos.repos.ijohanne.zot
];
};
};
}
</code></pre>
<p>Or as a direct flake input — useful if you don’t want the full NUR:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs.nur-packages.url = "github:ijohanne/nur-packages";
outputs = { self, nixpkgs, nur-packages, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
modules = [
nur-packages.nixosModules.zot
];
};
};
}
</code></pre>
<p>Either way, you get the same module. Pick whichever matches how you already manage inputs.</p>
<h2 id="basic-setup">Basic setup</h2>
<p>The minimum viable registry is surprisingly short:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.zot = {
enable = true;
dataDir = "/var/lib/zot";
user = "zot";
group = "zot";
};
</code></pre>
<p>That gets you Zot running on port 5000 with sane defaults — distribution spec 1.1.1, garbage collection on a 1-hour delay with a 6-hour interval, search, UI, scrub, metrics, and lint extensions all enabled. CVE database updates every 2 hours, scrub runs every 24 hours.</p>
<p>But a registry on localhost port 5000 isn’t useful to anyone. You want TLS, a real domain, and authentication. The module handles all of that.</p>
<h2 id="nginx-reverse-proxy">Nginx reverse proxy</h2>
<p>The module includes an nginx integration that does the right things:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.zot.nginx = {
enable = true;
domain = "registry.example.com";
forceSSL = true;
acme = true;
acmeDns01 = false;
};
</code></pre>
<p>This generates an nginx virtual host with Let’s Encrypt certificates, sets <code>client_max_body_size</code> to 0 — because you don’t want nginx rejecting your 2 GB container images — disables proxy buffering, and enables chunked transfer encoding. It also blocks the <code>/metrics</code> endpoint with a 403, which matters later when you set up monitoring.</p>
<h2 id="users-and-authentication">Users and authentication</h2>
<p>Zot uses htpasswd for authentication. The module manages it declaratively, regenerating the htpasswd file on every service start. Passwords come from files, which makes this sops-nix friendly:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.zot.auth.users = {
admin = {
passwordFile = config.sops.secrets.zot_admin_password.path;
admin = true;
};
ci-user = {
passwordFile = config.sops.secrets.zot_ci_password.path;
};
};
</code></pre>
<p>No plaintext passwords in your Nix config. The <code>passwordFile</code> attribute points to a file containing the raw password — sops-nix decrypts it at activation time, and the module reads it at service start. The <code>admin = true</code> flag gives the user elevated privileges in the access control system.</p>
<p>One thing to note: if <code>metrics.enable</code> is true — which it is by default — the module automatically creates a Prometheus scraping user. You don’t need to add one manually.</p>
<h2 id="access-control">Access control</h2>
<p>Authentication tells you <em>who</em> someone is. Access control tells you <em>what they can do</em>. The module supports fine-grained, per-repository policies using glob patterns:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.zot.accessControl = {
adminActions = [ "read" "create" "update" "delete" ];
defaultPolicy = [];
anonymousPolicy = [];
repositories."myorg/**" = {
policies = [{
users = [ "ci-user" ];
actions = [ "read" "create" "update" "delete" ];
}];
defaultPolicy = [];
anonymousPolicy = [ "read" ];
};
};
</code></pre>
<p>This configuration locks things down. The <code>defaultPolicy</code> and <code>anonymousPolicy</code> at the top level are empty — no access unless explicitly granted. Admin users get full read/create/update/delete. Under <code>myorg/**</code>, <code>ci-user</code> gets full access and anonymous users can pull.</p>
<p>That glob pattern is doing the heavy lifting. Everything under <code>myorg/</code> — <code>myorg/frontend</code>, <code>myorg/api</code>, <code>myorg/worker</code> — matches the same policy. You can add as many repository blocks as you need, each with its own set of users and permissions.</p>
<h2 id="retention-policies">Retention policies</h2>
<p>Registries accumulate images. Without retention policies, your disk fills up. The module makes garbage collection declarative:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.zot.retention = {
dryRun = false;
delay = "24h";
policies = [
{
repositories = [ "myorg/**" ];
deleteReferrers = true;
deleteUntagged = true;
keepTags = [
{ patterns = [ "latest" ]; }
{ pushedWithin = "168h"; }
{ mostRecentlyPushedCount = 10; }
{ patterns = [ "v.*" ]; pulledWithin = "720h"; }
];
}
];
defaultPolicy = {
deleteReferrers = false;
deleteUntagged = true;
keepTags = [
{ patterns = [ ".*" ]; }
];
};
};
</code></pre>
<p>Read the <code>keepTags</code> list as a set of survival conditions. An image in <code>myorg/**</code> stays if it’s tagged <code>latest</code>, was pushed in the last 7 days, is one of the 10 most recently pushed, or matches a semver pattern and was pulled in the last 30 days. Everything else gets cleaned up after 24 hours.</p>
<p>The <code>defaultPolicy</code> at the bottom is the fallback — it keeps everything tagged and deletes untagged images. Conservative, but it stops the obvious leak.</p>
<p>Set <code>dryRun = true</code> first and check the logs before you let it actually delete anything.</p>
<h2 id="monitoring-on-a-single-server">Monitoring on a single server</h2>
<p>If Prometheus and Grafana run on the same machine as your registry, the module does everything for you:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.zot = {
metrics.enable = true;
metrics.user = "prometheus";
metrics.password = "prometheus";
enableLocalScraping = true;
grafanaDashboard = true;
};
</code></pre>
<p><code>enableLocalScraping</code> adds a Prometheus scrape config pointed at Zot’s metrics endpoint on localhost. <code>grafanaDashboard</code> provisions a Grafana dashboard automatically. Metrics are enabled by default, so you technically only need the last two lines.</p>
<p>The auto-created Prometheus user authenticates against Zot’s htpasswd, so scraping goes through the same auth layer as everything else.</p>
<h2 id="monitoring-from-an-external-host">Monitoring from an external host</h2>
<p>When Prometheus runs on a different machine, you configure the scrape config yourself:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.prometheus.scrapeConfigs = [
{
job_name = "zot";
honor_labels = true;
metrics_path = "/metrics";
basic_auth = {
username = "prometheus";
password = "prometheus";
};
static_configs = [
{
targets = [ "10.255.101.200:5000" ];
labels = { instance = "registry"; };
}
];
}
];
</code></pre>
<p>The critical detail here — point your scrape target at Zot’s port directly, not at the nginx domain. The nginx config blocks <code>/metrics</code> with a 403. You want <code>10.255.101.200:5000</code>, not <code>registry.example.com:443</code>. If your monitoring traffic crosses untrusted networks, put it behind a VPN or a separate authenticated tunnel.</p>
<h2 id="the-grafana-dashboard">The Grafana dashboard</h2>
<p>The bundled dashboard is more thorough than you’d expect from a “batteries included” module. It covers download and upload counts with rates, per-repository metrics, per-prefix breakdowns, HTTP latency by method with heatmaps, scheduler queue length, and storage lock latency.</p>
<p>You get it for free with <code>grafanaDashboard = true</code>. No JSON to download, no manual import. It provisions itself through Grafana’s provisioning system.</p>
<h2 id="using-the-registry">Using the registry</h2>
<p>Once everything is deployed, the workflow is standard Docker:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">docker login registry.example.com -u ci-user
docker tag my-app:latest registry.example.com/myorg/my-app:latest
docker push registry.example.com/myorg/my-app:latest
</code></pre>
<p>Pulling works the same way:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">docker pull registry.example.com/myorg/my-app:latest
</code></pre>
<p>Zot’s built-in web UI is available at <code>https://registry.example.com</code>. It lets you browse repositories, inspect tags, and view vulnerability scan results. Nothing to install — it ships with the binary.</p>
<h2 id="full-example">Full example</h2>
<p>Here’s a complete single-server configuration tying everything together — sops secrets, users, access control, retention, nginx with ACME, and local monitoring:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{ config, ... }:
{
sops.secrets = {
zot_admin_password = { };
zot_ci_password = { };
};
services.zot = {
enable = true;
dataDir = "/var/lib/zot";
# Nginx reverse proxy with Let's Encrypt
nginx = {
enable = true;
domain = "registry.example.com";
forceSSL = true;
acme = true;
};
# Declarative users — passwords from sops
auth.users = {
admin = {
passwordFile = config.sops.secrets.zot_admin_password.path;
admin = true;
};
ci-user = {
passwordFile = config.sops.secrets.zot_ci_password.path;
};
};
# Access control
accessControl = {
adminActions = [ "read" "create" "update" "delete" ];
defaultPolicy = [ ];
anonymousPolicy = [ ];
repositories."myorg/**" = {
policies = [
{
users = [ "ci-user" ];
actions = [ "read" "create" "update" "delete" ];
}
];
defaultPolicy = [ ];
anonymousPolicy = [ "read" ];
};
};
# Retention — keep things tidy
retention = {
dryRun = false;
delay = "24h";
policies = [
{
repositories = [ "myorg/**" ];
deleteReferrers = true;
deleteUntagged = true;
keepTags = [
{ patterns = [ "latest" ]; }
{ pushedWithin = "168h"; }
{ mostRecentlyPushedCount = 10; }
{ patterns = [ "v.*" ]; pulledWithin = "720h"; }
];
}
];
defaultPolicy = {
deleteReferrers = false;
deleteUntagged = true;
keepTags = [
{ patterns = [ ".*" ]; }
];
};
};
# Monitoring — local Prometheus and Grafana
metrics.enable = true;
enableLocalScraping = true;
grafanaDashboard = true;
};
}
</code></pre>
<p>The systemd service runs as a simple service type with <code>PrivateTmp</code>, <code>ProtectHome</code>, <code>NoNewPrivileges</code>, and a file descriptor limit of 500,000. The module handles all of that — you don’t need to think about it.</p>
<p>That’s a private OCI registry with authentication, per-repo access control, automatic image cleanup, TLS, and full observability. One file, no imperative setup, and <code>nixos-rebuild switch</code> gets you there.</p>
Everything You Can Set on macOS with nix-darwin2026-03-26T12:00:00+00:002026-03-26T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/everything-nix-darwin-macos/<p>Most people discover Nix on macOS through packages. <code>nix-env -iA nixpkgs.ripgrep</code>, a few shell tools, maybe a dev environment. Useful, but it leaves the rest of your system untouched — a sprawl of <code>defaults write</code> commands, System Settings clicks you can’t remember, and a Dock that resets itself after every migration.</p>
<p><a rel="noopener noreferrer" target="_blank" href="https://github.com/LnL7/nix-darwin">nix-darwin</a> changes the scope. It’s a NixOS-style module system for macOS: you describe your system in Nix, run <code>darwin-rebuild switch</code>, and it converges. Packages, preferences, services, keyboard remapping, Homebrew casks, Dock layout — all from one configuration. Wipe the machine, rebuild, and everything is back exactly as it was.</p>
<p>This isn’t a setup tutorial. It’s a tour of what’s actually available — with real config snippets you can drop into your own <code>darwin-configuration.nix</code> and adapt.</p>
<h2 id="system-defaults-the-big-one">System defaults — the big one</h2>
<p>The <code>system.defaults</code> attrset is where nix-darwin really flexes. Every <code>defaults write</code> you’ve ever cargo-culted from a dotfiles repo has a typed, declarative equivalent here.</p>
<p><strong>NSGlobalDomain</strong> covers the settings that apply everywhere — dark mode, key repeat, scroll direction, that infuriating beep:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">system.defaults.NSGlobalDomain = {
AppleInterfaceStyle = "Dark";
ApplePressAndHoldEnabled = false;
KeyRepeat = 2;
InitialKeyRepeat = 15;
"com.apple.swipescrolldirection" = false;
"com.apple.sound.beep.volume" = 0.0;
"com.apple.sound.beep.feedback" = 0;
};
</code></pre>
<p><code>ApplePressAndHoldEnabled = false</code> is the one that gives you key repeat instead of the accent character picker. If you’ve ever held down <code>j</code> in Vim and watched nothing happen, this is why.</p>
<p><strong>Dock</strong> — hide it, size it, kill recents, disable space rearranging, set hot corners:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">system.defaults.dock = {
autohide = false;
tilesize = 48;
show-recents = false;
mru-spaces = false;
minimize-to-application = true;
expose-animation-duration = 0.2;
# Hot corners: 4=Desktop, 5=Screensaver, 12=Notification Center, 14=Quick Note
wvous-bl-corner = 4;
wvous-br-corner = 14;
wvous-tl-corner = 5;
wvous-tr-corner = 12;
};
</code></pre>
<p><code>mru-spaces = false</code> stops macOS from silently reordering your spaces based on usage. If you use numbered spaces and expect Space 3 to stay in position 3, you need this.</p>
<p><strong>Finder</strong> — show hidden files, default to list view, add a quit menu item:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">system.defaults.finder = {
AppleShowAllExtensions = true;
AppleShowAllFiles = true;
FXEnableExtensionChangeWarning = false;
FXPreferredViewStyle = "Nlsv"; # list view
FXRemoveOldTrashItems = true;
QuitMenuItem = true;
ShowPathbar = true;
ShowStatusBar = true;
};
</code></pre>
<p><code>QuitMenuItem = true</code> adds Cmd+Q to Finder. Sounds minor until you realize there’s no other way to fully quit it without <code>killall Finder</code>.</p>
<p><strong>Trackpad</strong> — tap to click and three-finger drag:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">system.defaults.trackpad = {
Clicking = true;
TrackpadRightClick = true;
TrackpadThreeFingerDrag = true;
};
</code></pre>
<p><strong>Screen capture</strong> — change the save location, format, and kill that thumbnail preview:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">system.defaults.screencapture = {
location = "~/Downloads";
type = "png";
disable-shadow = true;
show-thumbnail = false;
};
</code></pre>
<p><strong>Menu bar clock</strong>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">system.defaults.menuExtraClock = {
Show24Hour = true;
ShowSeconds = false;
ShowDate = 0;
};
</code></pre>
<p><strong>Login window</strong>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">system.defaults.loginwindow = {
GuestEnabled = false;
DisableConsoleAccess = true;
};
</code></pre>
<p><strong>Spaces across displays</strong>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">system.defaults.spaces.spans-displays = true;
</code></pre>
<p>That’s a lot of <code>defaults write</code> commands you no longer need to remember.</p>
<h2 id="customuserpreferences-the-escape-hatch">CustomUserPreferences — the escape hatch</h2>
<p>Not everything has a typed option in nix-darwin. <code>CustomUserPreferences</code> is a freeform attrset that maps directly to <code>defaults write</code> — any domain, any key. Think of it as the declarative version of your post-install shell script:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">system.defaults.CustomUserPreferences = {
"com.apple.finder" = {
ShowExternalHardDrivesOnDesktop = false;
ShowRemovableMediaOnDesktop = false;
_FXSortFoldersFirst = true;
FXDefaultSearchScope = "SCcf"; # search current folder
};
"com.apple.desktopservices" = {
DSDontWriteNetworkStores = true;
DSDontWriteUSBStores = true;
};
"com.apple.screensaver" = {
askForPassword = 1;
askForPasswordDelay = 0;
};
"com.apple.AdLib".allowApplePersonalizedAdvertising = false;
"com.apple.SoftwareUpdate" = {
AutomaticCheckEnabled = true;
ScheduleFrequency = 1;
AutomaticDownload = 1;
CriticalUpdateInstall = 1;
};
"com.apple.TimeMachine".DoNotOfferNewDisksForBackup = true;
"com.apple.WindowManager" = {
HideDesktop = true;
StandardHideDesktopIcons = true;
};
};
</code></pre>
<p><code>DSDontWriteNetworkStores</code> and <code>DSDontWriteUSBStores</code> stop macOS from scattering <code>.DS_Store</code> files across every network share and USB drive you mount. If you’ve ever committed a <code>.DS_Store</code> to a repo, you know why this matters.</p>
<p>You can also disable keyboard shortcuts like Ctrl+Space (input source switching) through <code>com.apple.symbolichotkeys</code> if you need that binding for something else.</p>
<h2 id="keyboard-remapping">Keyboard remapping</h2>
<p>Caps Lock as Control. Two lines, no Karabiner:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">system.keyboard = {
enableKeyMapping = true;
remapCapsLockToControl = true;
};
</code></pre>
<p>This uses the system-level key remapping — it works everywhere, including the login screen.</p>
<h2 id="touch-id-for-sudo">Touch ID for sudo</h2>
<p>One line. No more editing <code>/etc/pam.d/sudo_local</code> by hand and hoping a macOS update doesn’t overwrite it:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">security.pam.services.sudo_local.touchIdAuth = true;
</code></pre>
<p>Every <code>sudo</code> prompt becomes a fingerprint tap. This is the single most satisfying line in my entire nix-darwin config.</p>
<h2 id="declarative-homebrew">Declarative Homebrew</h2>
<p>Nix handles most packages, but some macOS GUI apps only exist as Homebrew casks, and some things only live on the Mac App Store. nix-darwin can manage Homebrew itself — casks, taps, MAS apps, and automatic cleanup of anything not declared:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">homebrew = {
enable = true;
caskArgs.no_quarantine = true;
onActivation = {
autoUpdate = true;
cleanup = "uninstall"; # remove anything not declared
upgrade = true;
};
casks = [ "firefox" "notion" "slack" "discord" ];
masApps = {
"WhatsApp" = 310633997;
"Xcode" = 497799835;
};
};
</code></pre>
<p><code>cleanup = "uninstall"</code> is the key line. Anything installed via Homebrew that isn’t in your <code>casks</code> or <code>masApps</code> list gets removed on rebuild. Your Mac only has what you’ve declared — nothing more.</p>
<p>Those numeric IDs come from the Mac App Store. Look them up with <code>mas</code>:</p>
<pre data-lang="sh" class="language-sh "><code class="language-sh" data-lang="sh">mas search WhatsApp
# 310633997 WhatsApp Messenger (24.9.80)
mas search Xcode
# 497799835 Xcode (16.3)
</code></pre>
<p>The number on the left is what goes in your <code>masApps</code> attrset. You can also grab it from any App Store URL — it’s the number after <code>/id</code> in <code>https://apps.apple.com/app/whatsapp-messenger/id310633997</code>.</p>
<h2 id="nix-settings-binary-caches-and-remote-builders">Nix settings — binary caches and remote builders</h2>
<p>Flakes, substituters, distributed builds to a Linux box:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">nix.settings = {
experimental-features = [ "nix-command" "flakes" ];
trusted-users = [ "root" "@admin" "youruser" ];
substituters = [ "https://cache.garnix.io" ];
trusted-public-keys = [
"cache.garnix.io:CTFPyKSLcx5RMJKfLo5EEPUObbA78b0YQ2DTCJXqr9g="
];
};
nix.distributedBuilds = true;
nix.buildMachines = [{
hostName = "10.0.0.50";
systems = [ "x86_64-linux" "aarch64-linux" ];
protocol = "ssh-ng";
maxJobs = 64;
supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
}];
</code></pre>
<p>That <code>trusted-users</code> line is worth a closer look. Nix builds run as the <code>nixbld</code> daemon user, not as you — so by default the builder can’t see your SSH keys or GitHub tokens. If your flake inputs point at private repos, the build will fail with authentication errors.</p>
<p>Adding yourself (or <code>@admin</code>) to <code>trusted-users</code> lets the daemon inherit your user-level access tokens. Pair it with <code>nix.extraOptions</code> to tell the daemon where to find your Git credentials:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">nix.extraOptions = ''
!include /etc/nix/access-tokens.conf
'';
</code></pre>
<p>Then drop a file at <code>/etc/nix/access-tokens.conf</code> — outside the store, so it stays out of world-readable <code>/nix/store</code> — with:</p>
<pre><code>access-tokens = github.com=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
</code></pre>
<p>Now <code>nix build</code> and <code>darwin-rebuild switch</code> can pull private flake inputs without manual <code>git clone</code>. The token never lands in the Nix store and you can rotate it in one place.</p>
<h2 id="services">Services</h2>
<p>nix-darwin can manage launchd services the same way NixOS manages systemd units. OpenSSH is the most common:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.openssh = {
enable = true;
extraConfig = ''
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no
'';
};
</code></pre>
<p>But the service catalog goes well beyond SSH — yabai, skhd, sketchybar, lorri, and the nix-daemon itself are all managed this way.</p>
<h2 id="activation-scripts">Activation scripts</h2>
<p>Need to run arbitrary shell commands on every <code>darwin-rebuild switch</code>? Activation scripts:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">system.activationScripts.postActivation.text = ''
chsh -s /run/current-system/sw/bin/fish youruser
'';
</code></pre>
<p>This is the escape hatch for anything nix-darwin doesn’t have a module for. It runs as root during activation, so you can set ownership, create directories, modify system files — whatever the rebuild needs to converge.</p>
<h2 id="dock-entries">Dock entries</h2>
<p>nix-darwin manages Dock contents natively through <code>system.defaults.dock.persistent-apps</code> — no third-party tools like <code>dockutil</code> needed. Just list your apps:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">system.defaults.dock.persistent-apps = [
"/Applications/Safari.app"
"/Applications/Slack.app"
"/System/Applications/Music.app"
];
</code></pre>
<p>Strings auto-coerce to app entries. For folders or spacers, use the tagged form: <code>{ folder = "/path"; }</code> or <code>{ spacer = { small = true; }; }</code>.</p>
<p>No more dragging icons around after a fresh install. The Dock shows exactly what you declared — nothing more, nothing less.</p>
<h2 id="everything-else">Everything else</h2>
<p>There’s more than fits in one post. A quick survey of other nix-darwin options worth exploring:</p>
<ul>
<li><code>fonts.packages</code> — declarative font installation, no more dragging <code>.ttf</code> files into Font Book</li>
<li><code>services.yabai</code> / <code>services.skhd</code> — tiling window management</li>
<li><code>services.sketchybar</code> — custom menu bar replacement</li>
<li><code>services.jankyborders</code> — window borders for tiling setups</li>
<li><code>launchd.user.agents</code> / <code>launchd.daemons</code> — custom launchd services for anything</li>
<li><code>programs.fish</code> / <code>programs.zsh</code> — shell configuration</li>
<li><code>power</code> — sleep and wake settings</li>
<li><code>system.defaults.universalaccess</code> — accessibility settings</li>
<li><code>system.defaults.WindowManager</code> — Stage Manager configuration</li>
<li><code>system.defaults.controlcenter</code> — control center visibility</li>
<li><code>networking.dns</code> / <code>networking.search</code> — DNS configuration</li>
<li><code>system.defaults.smb</code> — SMB/network settings</li>
</ul>
<h2 id="the-full-picture">The full picture</h2>
<p>The point isn’t any single option. It’s that your Mac — the preferences, the services, the packages, the Dock, the keyboard, the shell — is described in one place. You can diff it, review it, roll it back, and hand it to a new machine.</p>
<p>The best way to discover what’s available is the <a rel="noopener noreferrer" target="_blank" href="https://searchix.alanpearce.eu/options/darwin/search">nix-darwin option search</a>. Type a keyword, see what’s declarative. You’ll be surprised how much of System Settings has a Nix equivalent.</p>
<p><code>darwin-rebuild switch</code> and your Mac is exactly the machine you described.</p>
nix run and nix develop — Try Anything Without Installing It2026-03-25T12:00:00+00:002026-03-25T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/nix-run-develop-try-anything/<p>Every package manager works the same way. You hear about a tool, you install it, you try it, and then you either keep it or forget to uninstall it. Six months later you’re staring at <code>brew list</code> wondering what half of these things are and whether removing them will break something.</p>
<p>Nix inverts this. The default is impermanence. You run a program, it runs, and then it’s gone from your PATH. No install step, no uninstall step, no residue. If you want to keep it, that’s a separate, deliberate decision.</p>
<p>This is one of those features that sounds like a minor convenience until you use it for a week and can’t go back.</p>
<h2 id="tier-1-one-shot-commands-from-nixpkgs">Tier 1: one-shot commands from nixpkgs</h2>
<p>The simplest form. You want to run a program — once, right now, without thinking about it:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># Try btop without installing it
nix run nixpkgs#btop
# Try the Helix editor
nix run nixpkgs#helix
# Need jq for one pipeline?
nix run nixpkgs#jq -- '.[] | .name' < data.json
# Check disk usage with dust instead of du
nix run nixpkgs#dust
# Try ripgrep before deciding if it replaces grep
nix run nixpkgs#ripgrep -- "pattern" ./src
# Quick look at a CSV file
nix run nixpkgs#visidata -- data.csv
# Need to convert an image, once
nix run nixpkgs#imagemagick -- convert input.png -resize 50% output.png
# Quick HTTP server for the current directory
nix run nixpkgs#python3 -- -m http.server 8080
</code></pre>
<p>The <code>--</code> separates Nix’s flags from the program’s flags. Everything after <code>--</code> gets passed through to the program itself.</p>
<p>First run fetches and caches the package. Second run is instant — the binary is already in the Nix store. But it never appears in <code>which</code>, never shows up in your package list, never conflicts with anything. It’s there when you ask for it and invisible when you don’t.</p>
<p>The <code>nixpkgs#</code> prefix means “from the nixpkgs flake.” That’s the same package set behind <code>apt</code>, <code>brew</code>, and every NixOS system — about 100,000 packages. If it’s packaged for Linux, it’s probably in nixpkgs.</p>
<h2 id="tier-2-running-directly-from-github">Tier 2: running directly from GitHub</h2>
<p>Any project that ships a <code>flake.nix</code> with a <code>packages</code> or <code>apps</code> output can be run straight from its repository. No clone, no build setup, no README scavenger hunt:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># Ghostty — GPU-accelerated terminal emulator
nix run github:ghostty-org/ghostty
# OpenCode — terminal-based AI coding assistant
nix run github:anomalyco/opencode
# NixVim — full Neovim distribution configured with Nix
nix run github:nix-community/nixvim
</code></pre>
<p>You can pin to a branch, tag, or commit:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># Specific tag
nix run github:ghostty-org/ghostty/v1.0.0
# Specific branch
nix run github:ghostty-org/ghostty/main
# Specific app from a multi-output flake
nix run github:org/repo#specific-app
</code></pre>
<p>The URL anatomy, for reference:</p>
<pre><code>nix run github:<owner>/<repo>/<ref>#<app>
│ │ │ │ │
│ │ │ │ └── flake app output (optional, defaults to "default")
│ │ │ └── branch, tag, or commit (optional)
│ │ └── repository name
│ └── GitHub user or org
└── flake URL scheme
</code></pre>
<p>This is genuinely powerful. Someone posts a link to a project, you <code>nix run</code> it, and thirty seconds later you’re using it. No installation instructions, no dependency conflicts, no “works on my machine.” The flake defines exactly what gets built and how.</p>
<h2 id="tier-3-full-dev-environments-with-nix-develop">Tier 3: full dev environments with nix develop</h2>
<p><code>nix run</code> executes a single program. <code>nix develop</code> drops you into the project’s entire development environment — compilers, libraries, tools, environment variables, the lot.</p>
<p>You can do this without even cloning the repo:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># Enter the dev shell of a GitHub project directly
nix develop github:ghostty-org/ghostty
# Specific dev shell from a multi-shell flake
nix develop github:org/repo#shell-name
</code></pre>
<p>Or from a local checkout, which is the more common case:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">git clone https://github.com/user/my-project && cd my-project
nix develop
</code></pre>
<p>Inside the shell, everything the project’s <code>flake.nix</code> declares is available. The right compiler version, the right formatter, the right linter, any helper scripts — all in PATH. Leave the shell and they vanish. No global state touched.</p>
<p>This is what makes onboarding to a Nix-based project feel like cheating. No “install these seven things first” document. No version mismatches. <code>nix develop</code>, and you’re ready. If it built on CI, it builds on your machine, because it’s the same environment.</p>
<h2 id="direnv-the-invisible-version">direnv: the invisible version</h2>
<p>Typing <code>nix develop</code> every time you <code>cd</code> into a project gets old. direnv automates it. Add a <code>.envrc</code> to the project root:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">use flake
</code></pre>
<p>Now the dev shell activates when you enter the directory and deactivates when you leave. No manual shell entry, no remembering which project needs what. Your terminal just has the right tools available, always.</p>
<p>This combines especially well with <a rel="noopener noreferrer" target="_blank" href="https://github.com/nix-community/nix-direnv">nix-direnv</a>, which caches the dev shell so re-entering the directory doesn’t trigger a rebuild. First entry takes a few seconds. Every entry after that is instant.</p>
<h2 id="nix-shell-vs-nix-run">nix shell vs nix run</h2>
<p>These two look similar but serve different purposes:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># nix run: executes the program's default command, then you're done
nix run nixpkgs#htop
# nix shell: adds packages to PATH, then drops you into a shell
nix shell nixpkgs#imagemagick
convert input.png output.jpg # 'convert' is now in PATH
mogrify -resize 50% *.png # so is every other imagemagick command
exit # leave the shell, they're gone
</code></pre>
<p><code>nix run</code> is for “I want to use this program right now.” <code>nix shell</code> is for “I need these tools available for a while.” You can also combine packages in a single shell:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix shell nixpkgs#ffmpeg nixpkgs#imagemagick nixpkgs#jq
</code></pre>
<p>Three packages, one ephemeral environment. Useful when you’re doing some ad-hoc media processing and need a few tools together without setting up a flake.</p>
<h2 id="comma-the-laziest-possible-workflow">Comma: the laziest possible workflow</h2>
<p>If even <code>nix run nixpkgs#</code> is too much typing, <a rel="noopener noreferrer" target="_blank" href="https://github.com/nix-community/comma">comma</a> exists:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># Install comma once
nix profile install nixpkgs#comma
# Then just prefix any command with a comma
, btop
, dust ./
, cowsay "hello from nix"
</code></pre>
<p>Comma looks up the command in a nix-index database, finds the package that provides it, and runs it. No <code>nixpkgs#</code> prefix, no knowing the package name — just the command you want.</p>
<p>The trade-off is that you need to build the nix-index database first (<code>nix run nixpkgs#nix-index</code>), and it can return the wrong package if multiple packages provide the same command name. But for casual use — “I want to try that tool I saw on Hacker News” — it’s hard to beat.</p>
<h2 id="the-try-before-you-install-workflow">The try-before-you-install workflow</h2>
<p>Put it all together and you get a fundamentally different relationship with your tools:</p>
<ol>
<li><strong>Discover</strong> — hear about a tool, <code>nix run nixpkgs#tool</code> to try it</li>
<li><strong>Evaluate</strong> — use it for a few days via <code>nix shell</code> or comma</li>
<li><strong>Commit</strong> — if it’s worth keeping, add it to your NixOS configuration or home-manager</li>
<li><strong>Or don’t</strong> — do nothing, the cached build gets garbage-collected eventually</li>
</ol>
<p>The default is impermanence. You only permanently install things you’ve already decided are worth it. There’s no “I installed this six months ago and forgot” because you never installed it in the first place.</p>
<p>This also changes how you think about tool recommendations. Someone says “try hyperfine for benchmarking”? That’s a ten-second experiment, not a commitment. <code>nix run nixpkgs#hyperfine -- --warmup 3 'my-command'</code> — either it’s useful or it isn’t, and either way your system is unchanged.</p>
<h2 id="when-it-won-t-work">When it won’t work</h2>
<p>Not everything is <code>nix run</code>-able:</p>
<ul>
<li><strong>Libraries, fonts, and packages without executables</strong> — there’s nothing to run. Use <code>nix shell</code> or <code>nix build</code> instead.</li>
<li><strong>Programs that need system integration</strong> — a window manager, a display server, things that need to register with D-Bus or set up system services. These need proper installation.</li>
<li><strong>Projects without a flake.nix</strong> — the <code>github:</code> URL scheme requires a flake. An increasing number of projects have one, but plenty don’t. For non-flake Nix projects, you can sometimes use <code>--impure</code> or <code>nix-shell -p</code>, but that’s a different workflow.</li>
<li><strong>Large builds from source</strong> — <code>nix run github:some/project</code> might need to compile the entire thing if there’s no binary cache. That’s not a quick experiment, that’s a coffee break. Projects that publish to <a rel="noopener noreferrer" target="_blank" href="https://www.cachix.org/">cachix</a> or the NixOS binary cache avoid this.</li>
</ul>
<p>These are real limitations, but they’re the edges. For the vast majority of CLI tools, utilities, and development environments, the workflow just works. And once you’re used to it, going back to <code>brew install</code> / <code>brew uninstall</code> / <code>brew cleanup</code> feels like paperwork.</p>
Syncing qBittorrent Ports with ProtonVPN NAT-PMP on NixOS2026-03-24T12:00:00+00:002026-03-24T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/qbittorrent-protonvpn-natpmp-nixos/<p>Everything works until the port changes. You’re running qBittorrent behind ProtonVPN, port forwarding is enabled, peers are connecting — and then the VPN gateway silently reassigns your external port. qBittorrent doesn’t know. Peers try the old port, get nothing, and your swarm participation drops to zero. You might not notice for hours.</p>
<p>ProtonVPN handles port forwarding through NAT-PMP (RFC 6886). The gateway assigns an external port dynamically — it changes on reconnect and can change mid-session. There’s no static port option. If you want incoming connections, something needs to continuously track the assigned port and tell qBittorrent about it.</p>
<p>Manual configuration doesn’t survive a single VPN reconnect. You need a daemon.</p>
<p>I wrote one — <a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/proton-port-sync">proton-port-sync</a>. If you just want to use it, the README has everything you need. This post is about how it works and why it’s built the way it is.</p>
<h2 id="why-not-the-natpmp-crate">Why not the natpmp crate</h2>
<p>The Rust <code>natpmp</code> crate exists and implements the protocol. It has one critical problem with policy-based routing.</p>
<p>When you run WireGuard with policy routing — say, routing table 51820 matching traffic from <code>10.2.0.2</code> — the <code>natpmp</code> crate binds its UDP socket to <code>0.0.0.0:0</code>. The NAT-PMP request goes out over the VPN tunnel correctly, but the response from the gateway doesn’t route back. The kernel sees a packet for an unbound socket with no interface affinity and has no reason to route it through the VPN’s routing table. The request times out and you’re debugging network issues that don’t exist.</p>
<p>The fix: bind the UDP socket directly to the WireGuard interface IP. That’s the whole trick — the socket is now associated with the VPN interface, responses traverse the correct routing table, and everything works. It’s obvious in hindsight and confusing for about fifteen minutes.</p>
<p>There’s a second issue. RFC 6886 says internal port 0 means “delete all mappings.” ProtonVPN expects internal port 1 — the actual value is irrelevant since ProtonVPN assigns the port server-side, but it must be non-zero. The <code>natpmp</code> crate sends 0 by default.</p>
<p>Two bugs, both trivial individually, both invisible until you’ve wasted an evening on packet captures.</p>
<h2 id="the-protocol">The protocol</h2>
<p>NAT-PMP is refreshingly simple. Twelve bytes out, sixteen bytes back, over UDP to port 5351:</p>
<pre data-lang="rust" class="language-rust "><code class="language-rust" data-lang="rust">fn request_protocol_mapping(&self, opcode: u8, lifetime_secs: u32) -> Result<u16> {
let socket = UdpSocket::bind(SocketAddr::new(self.bind_address, 0))?;
socket.connect(SocketAddr::new(self.gateway, NATPMP_PORT))?;
let mut request = [0u8; 12];
request[1] = opcode;
request[4..6].copy_from_slice(&1u16.to_be_bytes()); // internal port = 1
request[8..12].copy_from_slice(&lifetime_secs.to_be_bytes());
// Exponential backoff per RFC 6886: 250ms initial, doubling, up to 9 attempts
let mut timeout = Duration::from_millis(250);
for attempt in 0..9 {
socket.set_read_timeout(Some(timeout))?;
socket.send(&request)?;
let mut buf = [0u8; 16];
match socket.recv(&mut buf) {
Ok(16) => {
let result_code = u16::from_be_bytes([buf[2], buf[3]]);
if result_code != 0 {
anyhow::bail!("NAT-PMP error: result code {result_code}");
}
let external_port = u16::from_be_bytes([buf[10], buf[11]]);
return Ok(external_port);
}
Ok(n) => anyhow::bail!("Unexpected response size: {n}"),
Err(_) if attempt < 8 => {
timeout *= 2;
continue;
}
Err(e) => return Err(e.into()),
}
}
anyhow::bail!("NAT-PMP request timed out after 9 attempts")
}
</code></pre>
<p>The request format is version (1 byte), opcode (1 byte), reserved (2 bytes), internal port (2 bytes), external port (2 bytes), and lifetime (4 bytes). The response mirrors it with a result code and the actual assigned external port. The RFC specifies exponential backoff starting at 250ms, doubling each attempt — nine attempts covers about two minutes of retries before giving up.</p>
<p>The daemon requests both UDP and TCP mappings. If they differ — which shouldn’t happen with ProtonVPN but can with other NAT-PMP gateways — it logs the discrepancy and uses the TCP port, since that’s what qBittorrent primarily uses for peer connections.</p>
<h2 id="the-main-loop">The main loop</h2>
<p>The core logic is a loop that renews the NAT-PMP mapping, detects port changes, and pushes updates to qBittorrent:</p>
<pre data-lang="rust" class="language-rust "><code class="language-rust" data-lang="rust">loop {
match natpmp_client.request_mapping(60) {
Ok(port) => {
fail_count = 0;
if current_port != Some(port) {
info!(%port, "Port changed, updating qBittorrent");
qbt.set_listen_port(port).await?;
current_port = Some(port);
// Update Prometheus metrics
}
}
Err(e) => {
warn!(?e, "NAT-PMP renewal failed");
fail_count += 1;
if fail_count >= max_failures {
warn!("Too many failures, restarting WireGuard");
Command::new("systemctl")
.args(["restart", &wg_unit])
.status()?;
fail_count = 0;
current_port = None;
sleep(Duration::from_secs(10)).await;
continue;
}
sleep(Duration::from_secs(15)).await;
continue;
}
}
sleep(Duration::from_secs(renew_interval)).await;
}
</code></pre>
<p>Three design choices worth noting:</p>
<p><strong>45-second renewal interval.</strong> NAT-PMP mappings are requested with a 60-second lifetime and renewed at 45 seconds. That’s a 15-second buffer — enough to absorb a slow response without the mapping expiring.</p>
<p><strong>3-failure threshold.</strong> After three consecutive NAT-PMP failures, the daemon restarts the WireGuard unit. This sounds aggressive, but in practice NAT-PMP failures almost always mean the tunnel is in a bad state. A stale gateway, a half-torn-down connection, a routing table that’s out of sync — restarting WireGuard clears all of it. Three failures at 15 seconds each means you wait 45 seconds before pulling the trigger.</p>
<p><strong>10-second post-restart cooldown.</strong> After restarting WireGuard, the daemon sleeps for 10 seconds before retrying. The tunnel needs time to re-establish — handshake, key exchange, routing table update. Hammering NAT-PMP requests during that window just adds noise to the logs.</p>
<h2 id="talking-to-qbittorrent">Talking to qBittorrent</h2>
<p>qBittorrent exposes a WebUI API. The daemon uses two endpoints — login and set preferences:</p>
<pre data-lang="rust" class="language-rust "><code class="language-rust" data-lang="rust">pub struct QbtClient {
client: reqwest::Client, // with cookie store
base_url: String,
username: String,
password: String,
}
impl QbtClient {
pub async fn set_listen_port(&self, port: u16) -> Result<()> {
self.login().await?;
self.client
.post(format!("{}/api/v2/app/setPreferences", self.base_url))
.form(&[("json", format!(r#"{{"listen_port":{port}}}"#))])
.send().await?
.error_for_status()?;
Ok(())
}
async fn login(&self) -> Result<()> {
self.client
.post(format!("{}/api/v2/auth/login", self.base_url))
.form(&[("username", &self.username), ("password", &self.password)])
.send().await?
.error_for_status()?;
Ok(())
}
}
</code></pre>
<p>The reqwest client is configured with a cookie store, so the session cookie from <code>login()</code> persists across requests. The daemon re-authenticates on every port change — qBittorrent sessions can expire, and re-logging in is cheaper than tracking session state.</p>
<p>The password comes from a file (<code>--qbt-password-file</code>), loaded once at startup. No passwords in CLI args, no passwords in environment variables. The file path is the only thing that appears in process listings or systemd unit files.</p>
<h2 id="prometheus-metrics">Prometheus metrics</h2>
<p>Six metrics on an optional <code>/metrics</code> endpoint:</p>
<pre><code>proton_port_sync_current_port — currently mapped port (gauge)
proton_port_sync_port_changes_total — total port changes (counter)
proton_port_sync_last_change_timestamp — unix timestamp of last change (gauge)
proton_port_sync_renewals_total — successful NAT-PMP renewals (counter)
proton_port_sync_failures_total — NAT-PMP request failures (counter)
proton_port_sync_wg_restarts_total — WireGuard restarts triggered (counter)
</code></pre>
<p>The useful alerts: <code>port_changes_total</code> increasing rapidly means the gateway is reassigning ports faster than expected — possibly a VPN issue. <code>failures_total</code> spiking means the tunnel is unstable. <code>wg_restarts_total</code> going up means the daemon is repeatedly cycling WireGuard, which warrants investigation.</p>
<p>The metrics endpoint is served via axum on a configurable address and port. It’s optional — skip <code>--metrics-addr</code> and the daemon runs without it. But if you’re already running Prometheus, the five minutes to wire it up will save you the next time something goes silently wrong at 3 AM.</p>
<h2 id="the-nixos-module">The NixOS module</h2>
<p>The module wraps all of this in a declarative systemd service:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{ config, lib, pkgs, ... }:
let
cfg = config.services.proton-port-sync;
in {
options.services.proton-port-sync = {
enable = lib.mkEnableOption "proton-port-sync";
gateway = lib.mkOption { type = lib.types.str; default = "10.2.0.1"; };
bindAddress = lib.mkOption { type = lib.types.str; default = "10.2.0.2"; };
qbtUrl = lib.mkOption { type = lib.types.str; default = "http://127.0.0.1:8080"; };
qbtUser = lib.mkOption { type = lib.types.str; default = "admin"; };
qbtPasswordFile = lib.mkOption { type = lib.types.path; };
renewInterval = lib.mkOption { type = lib.types.int; default = 45; };
maxFailures = lib.mkOption { type = lib.types.int; default = 3; };
wgUnit = lib.mkOption { type = lib.types.str; default = "wireguard-wg0.service"; };
metrics = {
enable = lib.mkEnableOption "Prometheus metrics";
address = lib.mkOption { type = lib.types.str; default = "127.0.0.1"; };
port = lib.mkOption { type = lib.types.port; default = 9834; };
};
};
config = lib.mkIf cfg.enable {
systemd.services.proton-port-sync = {
description = "Proton VPN NAT-PMP port sync for qBittorrent";
after = [ "network-online.target" cfg.wgUnit ];
bindsTo = [ cfg.wgUnit ];
wants = [ "qbittorrent.service" ];
wantedBy = [ "multi-user.target" ];
serviceConfig = {
ExecStart = "${pkgs.proton-port-sync}/bin/proton-port-sync"
+ " --gateway ${cfg.gateway}"
+ " --bind-address ${cfg.bindAddress}"
+ " --qbt-url ${cfg.qbtUrl}"
+ " --qbt-user ${cfg.qbtUser}"
+ " --qbt-password-file \${CREDENTIALS_DIRECTORY}/qbt-password"
+ lib.optionalString cfg.metrics.enable
" --metrics-addr ${cfg.metrics.address}:${toString cfg.metrics.port}";
LoadCredential = "qbt-password:${cfg.qbtPasswordFile}";
Restart = "on-failure";
RestartSec = "5s";
# Security hardening
ProtectSystem = "strict";
ProtectHome = true;
NoNewPrivileges = true;
PrivateTmp = true;
};
};
};
}
</code></pre>
<p>The systemd wiring matters more than it looks:</p>
<p><strong><code>bindsTo</code></strong> ties the service lifecycle to WireGuard. If WireGuard stops — whether from a manual stop, a crash, or a restart — this service stops too. No orphaned daemon sending NAT-PMP requests into the void.</p>
<p><strong><code>wants</code></strong> qBittorrent as a soft dependency. The daemon starts regardless of whether qBittorrent is running — it’ll just fail to update the port until qBittorrent comes up. A hard dependency (<code>requires</code>) would be wrong here because the daemon should survive a qBittorrent restart without being killed.</p>
<p><strong><code>LoadCredential</code></strong> handles the password file. systemd copies the file into a private credentials directory, accessible only to the service. The original file path never appears in the process’s environment or arguments — only the path under <code>$CREDENTIALS_DIRECTORY</code>. This is systemd’s credential passing mechanism, and it’s strictly better than <code>EnvironmentFile</code> or passing secrets via CLI args.</p>
<p>The hardening options — <code>ProtectSystem=strict</code>, <code>ProtectHome=true</code>, <code>NoNewPrivileges=true</code> — are standard for a daemon that only needs network access and one credential file. The service can’t write to the filesystem, can’t read home directories, and can’t escalate privileges. If the binary is compromised, the blast radius is minimal.</p>
<h2 id="wiring-it-up">Wiring it up</h2>
<h3 id="flake-input">Flake input</h3>
<p>Add the flake to your inputs with <code>nixpkgs.follows</code> to avoid pulling a second copy of nixpkgs:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">inputs = {
proton-port-sync.url = "github:ijohanne/proton-port-sync";
proton-port-sync.inputs.nixpkgs.follows = "nixpkgs";
};
</code></pre>
<h3 id="host-configuration">Host configuration</h3>
<p>Import the module and configure it:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">imports = [ proton-port-sync.nixosModules.default ];
services.proton-port-sync = {
enable = true;
gateway = "10.2.0.1";
qbtUser = "admin";
qbtPasswordFile = config.sops.secrets."qbittorrent/webui_password".path;
metrics = {
enable = true;
address = "10.100.0.10"; # private backhaul IP
port = 9834;
};
};
</code></pre>
<p>The <code>qbtPasswordFile</code> points to a sops-nix secret. On activation, sops-nix decrypts it to <code>/run/secrets/qbittorrent/webui_password</code>, and systemd’s <code>LoadCredential</code> picks it up from there. The password never appears in your Nix configuration, never lands in the Nix store, and only exists decrypted in a tmpfs mount.</p>
<h3 id="prometheus-scraping">Prometheus scraping</h3>
<p>On your monitoring host, add a scrape target:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.prometheus.scrapeConfigs = [
{
job_name = "proton-port-sync";
honor_labels = true;
static_configs = [
{
targets = [ "10.100.0.10:9834" ];
labels = { instance = "myhost"; };
}
];
}
];
</code></pre>
<p>That’s the full stack — NAT-PMP renewals, automatic qBittorrent updates, WireGuard failure recovery, Prometheus metrics, and sops-nix secrets management, all declared in a few dozen lines of Nix. The daemon itself is about 300 lines of Rust. The interesting part wasn’t the protocol or the API integration — it was the one-line socket bind fix that makes it actually work behind policy routing.</p>
Using Private GitHub Repositories with Nix Flakes2026-03-23T12:00:00+00:002026-03-23T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/private-github-repos-nix-flakes/<p>Private GitHub repos and Nix flakes have a failure mode that wastes exactly the right amount of time to be infuriating. A flake input like this:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">inputs = {
my-tool.url = "github:myorg/private-tool";
my-tool.inputs.nixpkgs.follows = "nixpkgs";
};
</code></pre>
<p>produces this on <code>nix flake update</code>:</p>
<pre><code>error: unable to download 'https://api.github.com/repos/myorg/private-tool/tarball/...': HTTP error 404
</code></pre>
<p>A 404. Not a 401. Not “authentication required.” Just “not found” — as if the repo doesn’t exist. GitHub returns 404 for unauthenticated requests to private repos, specifically to avoid confirming that the repo exists. It’s a deliberate security choice that sends people down exactly the wrong debugging path — double-checking URLs, triple-checking org names, wondering if someone deleted the repo.</p>
<p>The fix is to tell Nix how to authenticate.</p>
<h2 id="how-access-tokens-work">How access-tokens work</h2>
<p>Nix has a built-in <code>access-tokens</code> setting for exactly this:</p>
<pre><code>access-tokens = github.com=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxx
</code></pre>
<p>When Nix fetches from <code>github.com</code>, it includes this token in the request headers. Simple key-value — the host on the left, the token on the right. One line handles every private repo on that host.</p>
<p>This setting can live in:</p>
<ul>
<li><code>~/.config/nix/nix.conf</code> — user-level</li>
<li><code>/etc/nix/nix.conf</code> — system-level (for the Nix daemon)</li>
<li>A separate file pulled in via <code>!include</code></li>
</ul>
<p>That third option is the important one. The <code>!include</code> directive lets you keep the token in its own file, managed and rotated independently from <code>nix.conf</code>. You don’t want a long-lived secret sitting in a config file that might be world-readable, committed to a dotfiles repo, or copied around between machines.</p>
<pre><code>!include /path/to/access-tokens.conf
</code></pre>
<p>The included file contains the <code>access-tokens = ...</code> line. Nix reads it at evaluation time. The token never touches <code>nix.conf</code> itself.</p>
<h2 id="creating-the-right-token">Creating the right token</h2>
<p>Before wiring up configuration, you need a token with the right scope.</p>
<p><strong>Fine-grained tokens</strong> (the newer option):</p>
<ol>
<li>GitHub → Settings → Developer settings → Personal access tokens → Fine-grained tokens</li>
<li>Repository access: select the specific private repos your flake needs</li>
<li>Permissions: Contents → Read-only</li>
</ol>
<p>That’s the minimum. Nix fetches tarballs from the GitHub API — it needs to read repository contents, nothing else. No write access, no admin, no workflow permissions.</p>
<p><strong>Classic tokens</strong> with <code>repo</code> scope also work but grant far more access than Nix needs. If you’re already using a classic token for other purposes, it’ll work. If you’re creating one specifically for Nix, fine-grained is the better choice.</p>
<p>One token, all repos. If you give it access to <code>myorg/tool-a</code>, <code>myorg/tool-b</code>, and <code>myorg/tool-c</code>, a single <code>access-tokens</code> line authenticates fetches for all of them:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">inputs = {
tool-a.url = "github:myorg/tool-a";
tool-b.url = "github:myorg/tool-b";
tool-c.url = "github:myorg/tool-c";
};
</code></pre>
<p>No per-input configuration needed.</p>
<h2 id="pattern-1-nixos-with-sops-nix">Pattern 1: NixOS with sops-nix</h2>
<p>On NixOS, the Nix daemon runs as root and reads system-level configuration. sops-nix handles decryption — encrypted secrets in your repo, decrypted into <code>/run/secrets/</code> during system activation.</p>
<p>Declare the secret and tell Nix to include it:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">sops.secrets.nix_builder_access_tokens = { };
nix.extraOptions = ''
!include ${config.sops.secrets.nix_builder_access_tokens.path}
'';
</code></pre>
<p>The encrypted secrets file (<code>secrets/myhost.yaml</code>) contains:</p>
<pre data-lang="yaml" class="language-yaml "><code class="language-yaml" data-lang="yaml">nix_builder_access_tokens: ENC[AES256_GCM,data:...,type:str]
</code></pre>
<p>The plaintext value is just:</p>
<pre><code>access-tokens = github.com=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxx
</code></pre>
<p>On activation, sops-nix decrypts the YAML, writes the plaintext to <code>/run/secrets/nix_builder_access_tokens</code>, and Nix picks it up via the <code>!include</code>. The token is never in your nix.conf, never in your repo, and only exists decrypted in a tmpfs mount.</p>
<h3 id="setting-up-sops-nix">Setting up sops-nix</h3>
<p>If you haven’t set up sops-nix yet, here’s the short version. <code>.sops.yaml</code> at your repo root maps secrets to age keys:</p>
<pre data-lang="yaml" class="language-yaml "><code class="language-yaml" data-lang="yaml">keys:
- &myhost age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
- &personal age1yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
creation_rules:
- path_regex: secrets/myhost\.(yaml|json|env|ini)$
key_groups:
- age:
- *personal
- *myhost
</code></pre>
<p>Each host’s age key comes from its SSH host key:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">ssh-keyscan myhost.example.com 2>/dev/null | ssh-to-age
</code></pre>
<p>Create or edit secrets with:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">sops secrets/myhost.yaml
</code></pre>
<p>sops encrypts the file with every key listed in the matching creation rule. The host can decrypt with its own key, and you can decrypt with your personal key.</p>
<h2 id="pattern-2-macos-darwin-with-activation-scripts">Pattern 2: macOS/Darwin with activation scripts</h2>
<p>macOS doesn’t have NixOS’s activation system, but nix-darwin provides activation scripts that run during <code>darwin-rebuild switch</code>. The approach: decrypt the token with sops-nix, then copy it into the user’s Nix config directory.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">system.activationScripts.postActivation.text = ''
if [ -f /run/secrets/nix_builder_access_tokens ]; then
USER_NIX_DIR="/Users/''${user.username}/.config/nix"
mkdir -p "$USER_NIX_DIR"
cp /run/secrets/nix_builder_access_tokens "$USER_NIX_DIR/access-tokens.conf"
chmod 600 "$USER_NIX_DIR/access-tokens.conf"
chown ''${user.username}:staff "$USER_NIX_DIR/access-tokens.conf"
grep -q '!include' "$USER_NIX_DIR/nix.conf" 2>/dev/null || \
echo '!include access-tokens.conf' >> "$USER_NIX_DIR/nix.conf"
chown ''${user.username}:staff "$USER_NIX_DIR/nix.conf"
fi
'';
</code></pre>
<p>This copies the decrypted token to <code>~/.config/nix/access-tokens.conf</code> with mode 600 (owner-read-only), then ensures <code>nix.conf</code> has the <code>!include</code> line. Idempotent — safe to run repeatedly.</p>
<p>On macOS, Nix typically runs in single-user mode or the daemon reads from the user’s config. Either way, the token ends up where Nix can find it.</p>
<h2 id="remote-builders">Remote builders</h2>
<p>If you use remote builders, the builder machine needs the access tokens too. It’s the machine doing the fetching — your local Nix sends it a build job, and the builder pulls the flake inputs.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">nix.buildMachines = [
{
hostName = "builder.example.com";
systems = [ "x86_64-linux" "aarch64-linux" ];
protocol = "ssh-ng";
sshUser = "root";
sshKey = "/etc/nix/builder_ed25519";
maxJobs = 64;
speedFactor = 2;
supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
}
];
</code></pre>
<p>Same <code>!include</code> pattern on the builder host. If the builder doesn’t have the token, it hits the same 404 when it tries to fetch your private inputs. There’s nothing special about how remote builders authenticate — they just need the same <code>access-tokens</code> configuration as any other Nix installation that fetches from private repos.</p>
<p>The builder’s SSH key can also be managed via sops-nix on the machine that initiates the connection.</p>
<h2 id="the-bootstrap-problem">The bootstrap problem</h2>
<p>There’s a chicken-and-egg issue with fresh hosts. A newly installed NixOS machine needs access tokens to build its own configuration (because the flake has private inputs), but the tokens are in sops-nix secrets that only get deployed <em>after</em> a successful build.</p>
<p>The solution: build on an existing host that already has tokens, deploy the result to the new host.</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run nixpkgs#nixos-rebuild -- switch \
--flake .#myhost \
--target-host root@myhost.example.com \
--build-host root@builder.example.com
</code></pre>
<p>The builder fetches the private inputs with its own tokens, builds the system closure, and ships it to the target. After activation, sops-nix decrypts the tokens on the new host, and from that point forward it can build its own config.</p>
<p>This is a one-time problem per host. I wrote a <a href="/blog/sops-bootstrap-problem/">full post on the bootstrap dance</a> if you want the details.</p>
<h2 id="common-mistakes">Common mistakes</h2>
<p><strong>Token in nix.conf directly.</strong> Tempting for a quick test, dangerous as a habit. <code>/etc/nix/nix.conf</code> is often world-readable. Use <code>!include</code> and a separate file with restrictive permissions.</p>
<p><strong>Using netrc instead of access-tokens.</strong> Nix supports <code>~/.config/nix/netrc</code> for HTTP authentication, and it works. But <code>access-tokens</code> is purpose-built for this — it’s simpler, specific to code forges, and doesn’t conflict with other tools that read netrc files.</p>
<p><strong>Forgetting the Nix daemon.</strong> On multi-user Nix installs (the default on NixOS and recommended on macOS), the Nix daemon does the fetching. If you put the token in your user config but the daemon reads from <code>/etc/nix/nix.conf</code>, the daemon never sees it. On NixOS, <code>nix.extraOptions</code> writes to the system-level config. On macOS, make sure the daemon’s config includes the token, not just your user config.</p>
<p><strong>Fine-grained token without Contents permission.</strong> The GitHub fine-grained token UI has a lot of permission knobs. If you skip Contents read or leave it at “No access,” Nix gets the same 404. It’s not a different error — you just don’t have permission to see the tarball.</p>
<p><strong>Token expired or revoked.</strong> Fine-grained tokens have expiration dates. If your flake worked last week and doesn’t today, check the token. GitHub doesn’t send you a reminder email before it expires.</p>
<h2 id="the-minimal-setup">The minimal setup</h2>
<p>If you just want it working and you’ll worry about secrets management later, the fastest path:</p>
<ol>
<li>Create a fine-grained token with Contents read on your private repos</li>
<li>Add one line to <code>~/.config/nix/nix.conf</code>:</li>
</ol>
<pre><code>access-tokens = github.com=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxx
</code></pre>
<ol start="3">
<li>Run <code>nix flake update</code></li>
</ol>
<p>That gets you unblocked. Then move the token into <code>!include</code> and sops-nix at your own pace. The sops-nix setup is the right long-term answer — encrypted at rest, automatically deployed, rotatable — but it’s not a prerequisite for fetching private flake inputs. One line in your Nix config is all that’s strictly required.</p>
writeShellScriptBin — Why Every Nix Flake Script Needs an Explicit Shell2026-03-22T02:00:00+00:002026-03-22T02:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/writeshellscriptbin-explicit-shell/<p>The devShell looks clean. A <code>shellHook</code> that defines a <code>build</code> function, a <code>serve</code> helper, maybe some environment setup. It works perfectly — on your machine. A colleague pulls, runs <code>nix develop</code>, types <code>build</code>, and gets:</p>
<pre><code>fish: Unknown command 'build()'
</code></pre>
<p>They’re on fish. You’re on bash. Your shellHook is bash syntax. And <code>nix develop</code> just handed it to their shell verbatim.</p>
<h2 id="the-inheritance-problem">The inheritance problem</h2>
<p>When you enter a devShell with <code>nix develop</code>, Nix doesn’t spawn bash. It spawns <em>your</em> shell — whatever <code>$SHELL</code> points to. The <code>shellHook</code> runs inside that shell. If your hook uses bash syntax and the user runs fish, zsh, or nushell, anything beyond trivial <code>export</code> statements will break.</p>
<p>This isn’t a bug. It’s how <code>mkShell</code> works. The shellHook is shell code with no shebang — it runs in whatever interpreter shows up.</p>
<h2 id="the-ways-it-breaks">The ways it breaks</h2>
<p>The list of bash-isms that fail in fish is longer than you’d expect:</p>
<ul>
<li><code>set -euo pipefail</code> — fish error: <code>set: Unknown option '-e'</code></li>
<li><code>[[ ... ]]</code> double-bracket tests — fish error: <code>Unknown command '[['</code></li>
<li><code>if [ -z "$VAR" ]; then ... fi</code> — fish uses <code>if test -z "$VAR"; ... end</code></li>
<li><code>for f in *.txt; do ... done</code> — fish uses <code>for f in *.txt; ... end</code></li>
<li>Array syntax <code>arr=(a b c)</code> — fish uses <code>set arr a b c</code></li>
<li>Process substitution <code><(cmd)</code> — doesn’t exist in fish</li>
<li><code>source script.sh</code> — fish tries to interpret the bash syntax and fails</li>
<li>Function definitions <code>build() { ... }</code> — fish uses <code>function build; ... end</code></li>
</ul>
<p>Basically everything structural. Variables and <code>export</code> work. Control flow, functions, and error handling don’t.</p>
<p>Here’s what a typical broken shellHook looks like:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">devShells.default = pkgs.mkShell {
shellHook = ''
build() {
set -euo pipefail
echo "Building..."
${pkgs.zola}/bin/zola build
}
'';
};
</code></pre>
<p>Bash user gets a working <code>build</code> function. Fish user gets a wall of syntax errors on shell entry. They’ll either switch to bash, delete the shellHook, or stop using your devShell — none of which are what you wanted.</p>
<p>And it’s not just shellHook. If you put a <code>scripts/build.sh</code> in the repo, add it to PATH, and forget the shebang, the user’s shell interprets it directly. Same problem, less obvious cause.</p>
<h2 id="the-fix-writeshellscriptbin">The fix: writeShellScriptBin</h2>
<p><code>writeShellScriptBin</code> creates a standalone executable in the Nix store with an explicit bash shebang:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">serve = pkgs.writeShellScriptBin "serve" ''
echo "Serving at http://localhost:1111"
${pkgs.zola}/bin/zola serve --port 1111
'';
build = pkgs.writeShellScriptBin "build" ''
set -euo pipefail
echo "Building site..."
${pkgs.zola}/bin/zola build
echo "Done. Output in public/"
'';
devShells.default = pkgs.mkShell {
packages = [ serve build ];
};
</code></pre>
<p>Each script becomes a real executable at <code>/nix/store/...-serve/bin/serve</code>. The generated wrapper starts with <code>#!/nix/store/.../bin/bash</code> — not <code>#!/bin/bash</code>, which doesn’t exist on NixOS. The only thing in <code>/bin</code> is <code>sh</code>, for POSIX compatibility. You could use <code>#!/usr/bin/env bash</code>, but that depends on <code>bash</code> being in PATH at execution time. The Nix store path sidesteps both problems — it’s a direct, absolute reference to a specific bash derivation. When the user types <code>serve</code>, their shell finds the executable in PATH, sees the shebang, and hands it off to bash. Doesn’t matter if they’re running fish, zsh, nushell, or something they wrote themselves last weekend. The script always executes under bash.</p>
<h2 id="full-path-dependencies">Full-path dependencies</h2>
<p>Notice the <code>${pkgs.zola}/bin/zola</code> in the examples above. That Nix interpolation expands at build time to something like <code>/nix/store/abc123-zola-0.22.0/bin/zola</code>. The script doesn’t rely on <code>zola</code> being in PATH — it contains the absolute store path to the exact binary.</p>
<p>This matters. A script that calls bare <code>zola</code> works inside <code>nix develop</code> (where the devShell puts zola in PATH) but fails if someone runs it outside the shell, or if the PATH gets modified. Full store paths make the script self-contained. It works anywhere, any time, regardless of environment.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">build = pkgs.writeShellScriptBin "build" ''
set -euo pipefail
TYPST_FONT_PATHS="${pkgs.inter}/share/fonts" \
${pkgs.typst}/bin/typst compile cv-template.typ static/cv.pdf
${pkgs.zola}/bin/zola build
'';
</code></pre>
<p>Every dependency is pinned to a specific Nix store path. The script’s closure captures exactly what it needs. This is the same property that makes NixOS system configurations reproducible — applied to your little helper scripts.</p>
<h2 id="what-shellhook-should-actually-do">What shellHook should actually do</h2>
<p>Keep shellHook minimal. It should only do things that are genuinely shell-agnostic:</p>
<ul>
<li><strong>Setting environment variables</strong> — <code>export</code> works in bash, zsh, and modern fish</li>
<li><strong>Printing a welcome message</strong> — <code>echo</code> is universal</li>
<li><strong>Setting a prompt hint</strong> — if you must</li>
</ul>
<p>That’s about it. Anything with control flow, function definitions, or bash-specific syntax belongs in a <code>writeShellScriptBin</code> package. The boundary is clear: if it would break in fish, it doesn’t belong in shellHook.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">devShells.default = pkgs.mkShell {
packages = [ serve build build-pdf ];
shellHook = ''
export PROJECT_ROOT="$(pwd)"
echo "Commands: serve, build, build-pdf"
'';
};
</code></pre>
<p>Environment setup in the hook, real logic in wrapped scripts. Your fish users stop filing issues, your bash users notice no difference.</p>
<h2 id="writeshellapplication-the-stricter-variant">writeShellApplication — the stricter variant</h2>
<p>If you’re already wrapping scripts with <code>writeShellScriptBin</code>, consider <code>writeShellApplication</code> instead. It does three things on top:</p>
<ol>
<li><strong>Adds <code>set -euo pipefail</code> automatically</strong> — you don’t need to remember it</li>
<li><strong>Runs shellcheck at build time</strong> — catches bugs before the script ever executes</li>
<li><strong>Puts dependencies in <code>runtimeInputs</code></strong> — added to PATH, so you can use bare command names</li>
</ol>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">build = pkgs.writeShellApplication {
name = "build";
runtimeInputs = [ pkgs.zola pkgs.typst ];
text = ''
echo "Building site..."
TYPST_FONT_PATHS="${pkgs.inter}/share/fonts" \
typst compile cv-template.typ static/cv.pdf
zola build
'';
};
</code></pre>
<p>The trade-off: <code>runtimeInputs</code> adds dependencies to PATH at runtime rather than using full store paths. This is slightly less hermetic — the script depends on PATH being set up correctly — but it’s more readable and shellcheck can actually lint the command names. For project devShell scripts where readability matters more than absolute hermeticity, this is usually the right call.</p>
<p><code>writeShellApplication</code> also produces a slightly different output structure — the result is a derivation with <code>bin/<name></code> like <code>writeShellScriptBin</code>, but the build process includes the shellcheck step. If shellcheck finds issues, the build fails. You’ll discover that your <code>$variable</code> should be <code>"$variable"</code> at <code>nix build</code> time instead of at 2 AM in production.</p>
<h2 id="when-to-use-which">When to use which</h2>
<p><strong><code>writeShellScriptBin</code></strong> — when you want full control. You manage <code>set -euo pipefail</code> yourself, use full Nix store paths for dependencies, and don’t want shellcheck opinions about your quoting.</p>
<p><strong><code>writeShellApplication</code></strong> — when you want guardrails. Automatic strict mode, shellcheck enforcement, cleaner dependency declaration. This is the default choice for most devShell scripts.</p>
<p><strong>shellHook</strong> — only for environment variables and welcome messages. Nothing else.</p>
<p>The underlying principle is the same for both: an explicit shebang pointing at a Nix store bash, guaranteeing your script runs under the shell you wrote it for. Everything else is ergonomics.</p>
Distributing a Private CLI via Homebrew with Nix Cross-Compilation2026-03-21T12:00:00+00:002026-03-21T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/private-homebrew-tap-nix/<p>You have a private CLI tool. It’s built with Nix, produces <a href="/blog/portable-rust-binaries-nix/">static binaries for four platforms</a>, and works beautifully — on your machine. Now your teammates want it, and they don’t have Nix.</p>
<p>What you want is <code>brew install my-cli</code>. One command, any Mac or Linux box, no Nix required. But the source repo is private. Homebrew can’t authenticate against it. You can’t just point a formula at your GitHub Releases and call it a day.</p>
<p>This post covers the pattern: a private Homebrew tap repo, Nix cross-compilation, and a release script that builds, uploads, and generates the formula in one shot.</p>
<h2 id="the-auth-problem">The auth problem</h2>
<p>Homebrew formulas download tarballs via HTTPS. When you <code>brew install something</code>, Homebrew fetches the URL in the formula’s <code>url</code> field — no authentication, no SSH keys, just a plain HTTP GET. If your source repo is private, that GET returns a 404.</p>
<p>This is the fundamental constraint. The formula needs to download binaries from somewhere Homebrew can reach.</p>
<h2 id="the-tap-repo-trick">The tap repo trick</h2>
<p>The solution is a separate repo — a <em>tap</em> — that’s also private but becomes accessible once a user runs <code>brew tap</code>. Here’s how it works:</p>
<ol>
<li>You create a private repo named <code>homebrew-tap</code> (or <code>homebrew-tools</code>, <code>homebrew-internal</code> — the <code>homebrew-</code> prefix is what matters)</li>
<li>Your teammates run <code>brew tap myorg/tap</code>, which clones <code>github.com/myorg/homebrew-tap</code> via SSH</li>
<li>Once tapped, Homebrew can access that repo’s GitHub Releases via authenticated HTTPS</li>
<li>Your formula points to release assets on the <em>tap</em> repo, not the source repo</li>
</ol>
<p>The naming convention is load-bearing. Homebrew automatically prepends <code>homebrew-</code> to the short name and assumes GitHub:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">brew tap myorg/tap
# → clones github.com/myorg/homebrew-tap via SSH
</code></pre>
<p>No full URL needed. If your teammates have SSH keys configured for GitHub (they do — they’re developers), this just works. For HTTPS-only setups, <code>HOMEBREW_GITHUB_API_TOKEN</code> handles auth.</p>
<h2 id="tap-repo-structure">Tap repo structure</h2>
<p>Minimal. One directory, one file:</p>
<pre><code>homebrew-tap/
├── Formula/
│ └── my-cli.rb
</code></pre>
<p>That’s it. No Brewfile, no Casks, no scripts. The release assets live in GitHub Releases on this same repo.</p>
<h2 id="the-formula">The formula</h2>
<p>A Homebrew formula is a Ruby class that tells <code>brew</code> where to download binaries and how to install them. For a multi-platform CLI:</p>
<pre data-lang="ruby" class="language-ruby "><code class="language-ruby" data-lang="ruby">class MyCli < Formula
desc "Description of your CLI tool"
homepage "https://github.com/myorg/homebrew-tap"
version "0.1.0"
license "Proprietary"
on_macos do
if Hardware::CPU.arm?
url "https://github.com/myorg/homebrew-tap/releases/download/my-cli-v0.1.0/my-cli-0.1.0-darwin-arm64.tar.gz"
sha256 "DARWIN_ARM64_SHA256"
else
url "https://github.com/myorg/homebrew-tap/releases/download/my-cli-v0.1.0/my-cli-0.1.0-darwin-amd64.tar.gz"
sha256 "DARWIN_AMD64_SHA256"
end
end
on_linux do
if Hardware::CPU.arm?
url "https://github.com/myorg/homebrew-tap/releases/download/my-cli-v0.1.0/my-cli-0.1.0-linux-arm64.tar.gz"
sha256 "LINUX_ARM64_SHA256"
else
url "https://github.com/myorg/homebrew-tap/releases/download/my-cli-v0.1.0/my-cli-0.1.0-linux-amd64.tar.gz"
sha256 "LINUX_AMD64_SHA256"
end
end
def install
bin.install "my-cli"
end
test do
assert_match "my-cli", shell_output("#{bin}/my-cli --version")
end
end
</code></pre>
<p>Four platform blocks, four sha256 hashes, one <code>bin.install</code>. The URLs all point to the tap repo’s releases — not the source repo.</p>
<p>The <code>test</code> block is optional but good practice. <code>brew test my-cli</code> will run it after install.</p>
<h2 id="the-release-script">The release script</h2>
<p>You don’t want to build four tarballs, compute four hashes, upload them, edit the formula, commit, and push — by hand. That’s the kind of process you do correctly twice and then fumble on the third.</p>
<p>Instead, add a <code>homebrew-release</code> package to your flake. It’s a shell script with Nix-provided dependencies:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">homebrew-release = let
runtimeDeps = with pkgs; [ gh coreutils gnutar gzip git openssh jq ];
in pkgs.writeShellScriptBin "homebrew-release" ''
export PATH="${pkgs.lib.makeBinPath runtimeDeps}:$PATH"
set -euo pipefail
GH_OWNER="myorg"
GH_REPO="my-cli"
GH_TAP_REPO="homebrew-tap"
PACKAGE="my-cli"
TAP_REPO="git@github.com:$GH_OWNER/$GH_TAP_REPO.git"
if ! gh auth status &>/dev/null; then
echo "Error: gh CLI is not authenticated. Run: gh auth login" >&2
exit 1
fi
VERSION=$(grep '^version' cli/Cargo.toml | head -1 | sed 's/.*"\(.*\)"/\1/')
TAG="$PACKAGE-v$VERSION"
echo "Version: $VERSION (tag: $TAG)"
WORK=$(mktemp -d)
trap 'rm -rf "$WORK"' EXIT
# Clone tap repo for existing sha256 values
TAP_DIR="$WORK/tap"
git clone "$TAP_REPO" "$TAP_DIR" 2>&1 | grep -v "^warning:" || true
mkdir -p "$TAP_DIR/Formula"
FORMULA="$TAP_DIR/Formula/my-cli.rb"
existing_sha() {
local plat=$1
if [ -f "$FORMULA" ]; then
sed -n "/$plat\.tar\.gz/{n;s/.*sha256 \"\([^\"]*\)\".*/\1/p;}" "$FORMULA"
fi
}
# Create or reuse GitHub release on the tap repo
if gh release view "$TAG" --repo "$GH_OWNER/$GH_TAP_REPO" &>/dev/null; then
echo "Release $TAG already exists."
else
gh release create "$TAG" \
--repo "$GH_OWNER/$GH_TAP_REPO" \
--title "$PACKAGE $TAG" \
--notes "$PACKAGE release $VERSION"
fi
EXISTING_ASSETS=$(gh release view "$TAG" \
--repo "$GH_OWNER/$GH_TAP_REPO" \
--json assets -q '.assets[].name' 2>/dev/null || true)
DARWIN_ARM64_SHA="PLACEHOLDER"
DARWIN_AMD64_SHA="PLACEHOLDER"
LINUX_ARM64_SHA="PLACEHOLDER"
LINUX_AMD64_SHA="PLACEHOLDER"
PLATFORMS="darwin-arm64:aarch64-darwin darwin-amd64:x86_64-darwin linux-arm64:aarch64-linux linux-amd64:x86_64-linux"
for ENTRY in $PLATFORMS; do
PLAT="''${ENTRY%%:*}"
NIX_SYS="''${ENTRY##*:}"
FILENAME="$PACKAGE-$VERSION-$PLAT.tar.gz"
if echo "$EXISTING_ASSETS" | grep -qF "$FILENAME"; then
SHA=$(existing_sha "$PLAT")
if [ -n "$SHA" ] && [ "$SHA" != "PLACEHOLDER" ]; then
echo "[$PLAT] Already published (sha256: ''${SHA:0:16}...)"
else
echo "[$PLAT] Already published, fetching sha256..."
gh release download "$TAG" \
--repo "$GH_OWNER/$GH_TAP_REPO" \
--pattern "$FILENAME" \
--dir "$WORK"
SHA=$(sha256sum "$WORK/$FILENAME" | cut -d' ' -f1)
fi
else
echo "[$PLAT] Building .#packages.$NIX_SYS.my-cli-static..."
OUT=$(nix build ".#packages.$NIX_SYS.my-cli-static" --no-link --print-out-paths)
BDIR="$WORK/build-$PLAT"
mkdir -p "$BDIR"
cp "$OUT/bin/my-cli" "$BDIR/"
TARBALL="$WORK/$FILENAME"
tar czf "$TARBALL" -C "$BDIR" my-cli
SHA=$(sha256sum "$TARBALL" | cut -d' ' -f1)
echo "[$PLAT] sha256: ''${SHA:0:16}..."
echo "[$PLAT] Uploading to release..."
gh release upload "$TAG" "$TARBALL" \
--repo "$GH_OWNER/$GH_TAP_REPO" \
--clobber
fi
case "$PLAT" in
darwin-arm64) DARWIN_ARM64_SHA="$SHA" ;;
darwin-amd64) DARWIN_AMD64_SHA="$SHA" ;;
linux-arm64) LINUX_ARM64_SHA="$SHA" ;;
linux-amd64) LINUX_AMD64_SHA="$SHA" ;;
esac
done
# Generate the formula with real sha256 values
# (template omitted for brevity — same structure as above,
# with $DARWIN_ARM64_SHA etc. interpolated)
cd "$TAP_DIR"
git add Formula/my-cli.rb
if git diff --cached --quiet; then
echo "Formula already up to date."
else
git commit -m "Update my-cli to $VERSION"
git push -u origin HEAD
echo "Tap updated."
fi
'';
</code></pre>
<p>Then expose it in your flake outputs:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">packages.homebrew-release = homebrew-release;
</code></pre>
<p>A few things worth calling out:</p>
<ul>
<li><strong>Incremental by design.</strong> If a platform’s tarball is already uploaded, the script reads its sha256 from the existing formula and skips the build. You can resume after a partial failure — say, if the linux-amd64 remote builder was down — without re-uploading what’s already done.</li>
<li><strong><code>gh</code> for everything.</strong> Creates releases, uploads assets, downloads for sha256 recovery. Must be authenticated via <code>gh auth login</code>.</li>
<li><strong>Clones the tap to a temp dir.</strong> Reads existing sha256 values, writes the updated formula, commits, pushes. The temp dir gets cleaned up on exit.</li>
</ul>
<h2 id="the-release-workflow">The release workflow</h2>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># 1. Bump version in cli/Cargo.toml
# 2. Run the release script
nix run .#homebrew-release
</code></pre>
<p>That’s it. The script:</p>
<ol>
<li>Reads the version from <code>Cargo.toml</code></li>
<li>Creates a GitHub Release tagged <code>my-cli-v0.1.0</code> on the tap repo</li>
<li>Builds static binaries for all four platforms via <code>nix build .#packages.$system.my-cli-static</code></li>
<li>Tars them up, computes sha256 hashes, uploads as release assets</li>
<li>Generates the formula with real hashes, commits, and pushes to the tap repo</li>
</ol>
<p>Your teammates run <code>brew upgrade my-cli</code> and get the new version. No Nix, no Docker, no “download this tarball and put it in your PATH.”</p>
<h2 id="user-installation">User installation</h2>
<p>From the user’s perspective — the person who just wants the CLI:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># One-time setup
brew tap myorg/tap
# Install
brew install my-cli
# Later
brew upgrade my-cli
</code></pre>
<p>Three commands total, one of which they do once. If they have SSH access to the tap repo, it just works. That’s the whole point.</p>
<h2 id="public-vs-private-source-repos">Public vs. private source repos</h2>
<p>If your source repo is public, you don’t need the tap repo trick at all. Point the formula directly at the source repo’s GitHub Releases:</p>
<pre data-lang="ruby" class="language-ruby "><code class="language-ruby" data-lang="ruby">url "https://github.com/myorg/my-cli/releases/download/v0.1.0/my-cli-0.1.0-darwin-arm64.tar.gz"
</code></pre>
<p>Homebrew can fetch from public repos without auth. You still need a tap repo for the formula itself (unless you’re submitting to homebrew-core, which requires open source), but the binaries can live on the source repo.</p>
<p>The whole re-hosting dance — uploading binaries to the tap repo’s releases — only matters when the source repo is private and Homebrew can’t reach it. If that constraint goes away, simplify.</p>
<h2 id="prerequisites">Prerequisites</h2>
<ul>
<li>A Nix flake that produces static binaries for four platforms — see <a href="/blog/portable-rust-binaries-nix/">Building Portable Rust CLI Binaries with Nix</a> for the full pattern</li>
<li><code>gh</code> CLI installed and authenticated</li>
<li>Remote builders configured for cross-platform builds (Linux from Mac, or vice versa)</li>
<li>A <code>homebrew-tap</code> repo created on GitHub (private, with the <code>homebrew-</code> prefix)</li>
<li>SSH access to the tap repo (or <code>HOMEBREW_GITHUB_API_TOKEN</code> for HTTPS)</li>
</ul>
<p>The Nix cross-compilation and Homebrew tap are independent problems that happen to compose well. Nix gives you reproducible, hermetic builds across four targets. Homebrew gives you a distribution channel that meets developers where they already are. The release script is the glue.</p>
Monitoring PostgreSQL on NixOS with pg_exporter2026-03-20T02:00:00+00:002026-03-20T02:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/pg-exporter-nixos/<p>You set up <code>services.prometheus.exporters.postgres</code> on your NixOS box. It works. You get a handful of metrics — connection counts, database sizes, a few transaction stats. You wire up a basic Grafana dashboard and call it done.</p>
<p>Then something goes sideways. Autovacuum is running hot, WAL generation is spiking, your cache hit ratio tanked overnight, and the built-in exporter doesn’t have an opinion on any of it. You find yourself writing custom SQL queries, bolting them onto the exporter’s <code>--extend.query-path</code>, managing a YAML file by hand, and wondering if someone has already solved this.</p>
<p>Someone has.</p>
<h2 id="pg-exporter-vs-postgres-exporter">pg_exporter vs postgres_exporter</h2>
<p>The <a rel="noopener noreferrer" target="_blank" href="https://github.com/prometheus-community/postgres_exporter">prometheus-postgres-exporter</a> in nixpkgs is the community standard. It does the job for basic monitoring. But if you want deep PostgreSQL observability — the kind where you catch problems before they page you — it starts to feel thin.</p>
<p><a rel="noopener noreferrer" target="_blank" href="https://github.com/pgsty/pg_exporter">pg_exporter</a>, from the <a rel="noopener noreferrer" target="_blank" href="https://pigsty.io">Pigsty</a> project, ships over 50 collector definitions out of the box. WAL generation rates, checkpoint durations, lock breakdowns by mode, replication lag, sequential scan ratios, dead tuple counts, temp file writes — all pre-configured. Each collector is a standalone YAML file that can be individually enabled or disabled. It supports auto-discovery of all databases on the server. And it handles pgbouncer metrics natively if you run a connection pooler.</p>
<p>The tradeoff is that pg_exporter isn’t in nixpkgs. You need to package it yourself and write a NixOS module. Which turns out to be a good exercise in how NixOS module design makes this kind of thing surprisingly pleasant.</p>
<h2 id="the-package">The package</h2>
<p>pg_exporter is a Go binary. Packaging it is straightforward with <code>buildGoModule</code>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">pkgs.buildGoModule {
pname = "pg-exporter";
version = "1.2.0";
src = sources.pg-exporter; # github:pgsty/pg_exporter/v1.2.0
vendorHash = "sha256-j5RwjJPUDh9AUYqqfvnvcDnGrykRh+6ydbaRsTutO+U=";
doCheck = false;
postInstall = ''
mkdir -p $out/share/pg_exporter
cp config/*.yml $out/share/pg_exporter/
'';
meta = {
description = "Advanced PostgreSQL & Pgbouncer metrics exporter for Prometheus";
homepage = "https://github.com/pgsty/pg_exporter";
license = pkgs.lib.licenses.asl20;
};
}
</code></pre>
<p>The <code>postInstall</code> is the important bit. pg_exporter’s repo has a <code>config/</code> directory with all 50+ collector YAML files. Each defines queries, metric types, and metadata for a specific area of PostgreSQL internals. The binary reads a config directory at startup and loads every YAML it finds. By copying those into <code>$out/share/pg_exporter/</code>, the module can reference them at build time — no runtime fetching, no mutable state.</p>
<h2 id="getting-it-into-your-flake">Getting it into your flake</h2>
<p>You have two paths depending on how you like to consume third-party NixOS packages.</p>
<h3 id="as-a-nur-package">As a NUR package</h3>
<p>The package and module live in my <a rel="noopener noreferrer" target="_blank" href="https://github.com/nix-community/NUR">NUR</a> repository. Add NUR as a flake input and import the module from <code>nur.repos.ijohanne</code>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
nur.url = "github:nix-community/NUR";
};
outputs = { nixpkgs, nur, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
modules = [
nur.repos.ijohanne.modules.pg-exporter
{
services.pg-exporter.enable = true;
}
];
};
};
}
</code></pre>
<p>NUR handles overlay registration. The NixOS module and the package are both provided by the repo — you just import the module and use it.</p>
<h3 id="as-a-direct-flake-input">As a direct flake input</h3>
<p>If you prefer to skip NUR entirely, add the repo as a direct flake input:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
ijohanne-nur.url = "github:ijohanne/nur-packages";
};
outputs = { nixpkgs, ijohanne-nur, ... }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
modules = [
ijohanne-nur.nixosModules.pg-exporter
{
services.pg-exporter.enable = true;
}
];
};
};
}
</code></pre>
<p>Note the lack of <code>inputs.nixpkgs.follows</code> — the NUR repo pins its own nixpkgs-unstable because pg_exporter requires Go 1.26, which isn’t available on the stable channel yet. This means it pulls in a separate nixpkgs eval, but that only affects the pg_exporter build closure, not the rest of your system.</p>
<p>Either way, once the module is imported the configuration is identical. The rest of this post assumes the module is available — how it got there is up to you.</p>
<h2 id="the-nixos-module">The NixOS module</h2>
<p>Here’s where it gets interesting. The module needs to do three things: manage collector configuration, run the service, and optionally wire up Grafana and Prometheus.</p>
<h3 id="options">Options</h3>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">options.services.pg-exporter = {
enable = mkEnableOption "pg_exporter PostgreSQL metrics exporter";
port = mkOption { type = types.port; default = 9630; };
listenAddress = mkOption { type = types.str; default = "0.0.0.0"; };
environmentFile = mkOption { type = types.nullOr types.path; default = null; };
defaultCollectors = mkOption { type = types.bool; default = true; };
disabledCollectors = mkOption { type = types.listOf types.str; default = [ ]; };
settings = mkOption { type = settingsFormat.type; default = { }; };
autoDiscovery = mkOption { type = types.bool; default = false; };
extraFlags = mkOption { type = types.listOf types.str; default = [ ]; };
user = mkOption { type = types.nullOr types.str; default = null; };
enableLocalScraping = mkEnableOption "scraping by local prometheus";
grafanaDashboard = mkEnableOption "Grafana dashboard provisioning";
};
</code></pre>
<p><code>defaultCollectors</code> controls whether the bundled YAML files ship. <code>disabledCollectors</code> lets you turn off specific ones by name. <code>settings</code> is a freeform Nix attrset that renders to YAML — for custom queries. <code>user</code> lets you run as <code>postgres</code> for socket auth instead of the default <code>DynamicUser</code>.</p>
<h3 id="config-directory-assembly">Config directory assembly</h3>
<p>This is the core mechanism. pg_exporter reads all YAML files in its config directory alphabetically and merges them. The module builds this directory at Nix evaluation time:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">settingsFormat = pkgs.formats.yaml { };
disabledFile = pkgs.writeText "pg-exporter-disabled.yml" (builtins.toJSON
(builtins.listToAttrs (map (name: {
inherit name;
value = { skip = true; };
}) cfg.disabledCollectors)));
settingsFile = settingsFormat.generate "pg-exporter-custom.yml" cfg.settings;
configDir = pkgs.runCommand "pg-exporter-config" { } (''
mkdir -p $out
'' + optionalString cfg.defaultCollectors ''
cp ${package}/share/pg_exporter/*.yml $out/
'' + optionalString (cfg.disabledCollectors != [ ]) ''
cp ${disabledFile} $out/9999-disabled.yml
'' + optionalString (cfg.settings != { }) ''
cp ${settingsFile} $out/9999-custom.yml
'');
</code></pre>
<p>Three layers:</p>
<ol>
<li><strong>Default collectors</strong> — copies the 50+ bundled YAMLs from the package</li>
<li><strong>Disabled collectors</strong> — generates a <code>9999-disabled.yml</code> with <code>skip: true</code> entries</li>
<li><strong>Custom settings</strong> — renders your Nix attrset to <code>9999-custom.yml</code> via <code>pkgs.formats.yaml</code></li>
</ol>
<p>The <code>9999-</code> prefix ensures overrides load last. pg_exporter processes files alphabetically, so a later file wins. This is the same pattern you’d use with systemd drop-in directories — convention over configuration.</p>
<h3 id="the-systemd-service">The systemd service</h3>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">systemd.services.pg-exporter = {
description = "pg_exporter PostgreSQL metrics exporter";
wantedBy = [ "multi-user.target" ];
after = [ "network.target" "postgresql.service" ];
serviceConfig = {
Restart = "always";
ProtectHome = true;
PrivateTmp = true;
NoNewPrivileges = true;
} // (if cfg.user != null then {
User = cfg.user;
} else {
DynamicUser = true;
ProtectSystem = "strict";
}) // {
ExecStart = concatStringsSep " " ([
"\${getBin package}/bin/pg_exporter"
"--config=\${configDir}"
"--web.listen-address=\${cfg.listenAddress}:\${toString cfg.port}"
]
++ optional cfg.autoDiscovery "--auto-discovery"
++ cfg.extraFlags);
} // optionalAttrs (cfg.environmentFile != null) {
EnvironmentFile = cfg.environmentFile;
};
};
</code></pre>
<p>When <code>user</code> is null, the service uses <code>DynamicUser</code> with strict system protection — no writable paths, no privilege escalation. If you set <code>user = "postgres"</code>, it drops <code>DynamicUser</code> and <code>ProtectSystem</code> so it can use the postgres Unix socket for authentication. This avoids needing a password in the environment file entirely.</p>
<h3 id="one-toggle-grafana-and-prometheus">One-toggle Grafana and Prometheus</h3>
<p>The last piece is integration wiring:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix"># Grafana dashboard provisioning
services.grafana.provision.dashboards.settings.providers =
mkIf cfg.grafanaDashboard [{
name = "pg-exporter";
options.path = pkgs.runCommand "pg-exporter-dashboards" { } ''
mkdir -p $out
cp ${./dashboard.json} $out/postgres.json
'';
}];
# Prometheus scrape config
services.prometheus.scrapeConfigs = mkIf cfg.enableLocalScraping [{
job_name = "postgres";
honor_labels = true;
static_configs = [{
targets = [ "127.0.0.1:\${toString cfg.port}" ];
}];
}];
</code></pre>
<p><code>grafanaDashboard = true</code> provisions a dashboard JSON into Grafana’s filesystem provisioning. <code>enableLocalScraping = true</code> adds a Prometheus scrape target pointing at localhost. Both are <code>mkIf</code>-guarded — if you don’t flip the toggle, they produce no configuration.</p>
<h2 id="nix-to-yaml-custom-collectors">Nix-to-YAML custom collectors</h2>
<p>The <code>settings</code> option is where <code>pkgs.formats.yaml</code> shines. You define custom queries in Nix and they render to valid YAML without you ever touching a YAML file:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.pg-exporter.settings = {
my_custom_query = {
query = "SELECT count(*) as total_users FROM users";
metrics = [
{
total_users = {
usage = "GAUGE";
description = "Total number of users";
};
}
];
ttl = 60;
tags = [ "custom" ];
};
};
</code></pre>
<p>Full Nix type checking. No YAML quoting issues. No hand-managing a sidecar file. The Nix module system validates the structure at evaluation time — if you fat-finger a field name, you get an error before the config ever reaches the exporter.</p>
<h2 id="the-dashboard">The dashboard</h2>
<p>The bundled Grafana dashboard covers the metrics that actually matter for day-to-day PostgreSQL operations.</p>
<p>Transactions per second — commits and rollbacks, total across all databases. Zero rollbacks is what you want to see:</p>
<p><img src="https://perlpimp.net/blog/pg-exporter-nixos/transactions-per-sec.png" alt="Transactions per second — commits and rollbacks across all databases" /></p>
<p>Tuple operations give you the read/write breakdown. The “returned” line is what PostgreSQL scanned; “fetched” is what it actually sent back to the client. A large gap between the two means sequential scans are doing a lot of wasted work:</p>
<p><img src="https://perlpimp.net/blog/pg-exporter-nixos/tuple-operations.png" alt="Tuple operations per second — fetched, returned, and inserted" /></p>
<p>Sequential scans vs index scans. You want the yellow line (index scans) to dominate. When sequential scans spike, either a query plan changed or you’re missing an index:</p>
<p><img src="https://perlpimp.net/blog/pg-exporter-nixos/seq-vs-index-scans.png" alt="Sequential scans vs index scans per second" /></p>
<p>Connections by state shows your connection pool saturation at a glance. The idle connections (the filled area) sitting around 50–60 is typical for a pooled setup:</p>
<p><img src="https://perlpimp.net/blog/pg-exporter-nixos/connections-by-state.png" alt="Connections by state — active, disabled, and idle" /></p>
<p>Sessions per second tracks session creation rate. Flat and low is healthy — spikes mean something is churning connections:</p>
<p><img src="https://perlpimp.net/blog/pg-exporter-nixos/sessions-per-sec.png" alt="Sessions per second — total, abandoned, fatal, and killed" /></p>
<p>All of these panels — plus cache hit ratios, WAL generation rates, checkpoint durations, lock breakdowns, autovacuum activity, database sizes, and temp file writes — ship with a single toggle.</p>
<h2 id="putting-it-together">Putting it together</h2>
<p>A minimal production config:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.pg-exporter = {
enable = true;
environmentFile = config.sops.secrets.pg-exporter-env.path;
grafanaDashboard = true;
enableLocalScraping = true;
};
</code></pre>
<p>The environment file — managed by sops-nix or agenix or however you handle secrets — contains the connection string:</p>
<pre><code>PG_EXPORTER_URL=postgresql://pg_exporter:password@localhost:5432/postgres?sslmode=disable
</code></pre>
<p>If you want socket auth and fine-grained control over collectors:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.pg-exporter = {
enable = true;
autoDiscovery = true;
user = "postgres";
environmentFile = config.sops.secrets.pg-exporter-env.path;
disabledCollectors = [
"pgbouncer_list"
"pgbouncer_database"
"pgbouncer_stat"
"pgbouncer_pool"
"pg_tsdb_hypertable"
"pg_citus"
"pg_recv"
"pg_sub"
"pg_origin"
"pg_pubrel"
"pg_subrel"
"pg_sync_standby"
"pg_downstream"
"pg_heartbeat"
];
grafanaDashboard = true;
enableLocalScraping = true;
};
</code></pre>
<p>The <code>disabledCollectors</code> list is where you prune what you don’t need. No pgbouncer? Disable those four. No TimescaleDB or Citus? Gone. No replication? Drop the replication collectors. The exporter won’t waste time querying for metrics that don’t apply to your setup.</p>
<h2 id="the-nixos-module-pattern">The NixOS module pattern</h2>
<p>This is the same pattern every time: a Go binary gets <code>buildGoModule</code>, its config files land in <code>$out/share/</code>, the module assembles a config directory from Nix expressions, and integration options (<code>grafanaDashboard</code>, <code>enableLocalScraping</code>) wire into other NixOS services with <code>mkIf</code>. The systemd service uses <code>DynamicUser</code> for least-privilege. Secrets stay in environment files, never in the Nix store.</p>
<p>If you’re packaging any Prometheus exporter for NixOS, this is the template. The specific queries and YAML files change. The module structure doesn’t.</p>
PostgreSQL 14→18 on NixOS — A Flake-Native Migration2026-03-19T02:00:00+00:002026-03-19T02:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/postgresql-14-18-nixos-migration/<p>You check your NixOS server and PostgreSQL is on 14.22. The server’s been running <code>system.stateVersion = "22.05"</code> since it was provisioned, and NixOS dutifully kept the default PostgreSQL version for that era. NixOS 25.11 ships PostgreSQL 17 as the default for new installs, and PG 18 is in nixpkgs. PG 14 reaches end-of-life in November 2026. Thirteen databases across multiple application instances need to move.</p>
<p>PostgreSQL does not support in-place major version upgrades. The on-disk format changes between major versions, so you either use <code>pg_upgrade</code> — which rewrites catalog metadata while preserving data files — or do a full <code>pg_dumpall</code>/restore cycle. <code>pg_upgrade</code> is dramatically faster because it links or copies existing data files rather than re-inserting every row through SQL. That’s what we’ll use.</p>
<h2 id="why-flake-scripts-instead-of-ad-hoc-shell-commands">Why flake scripts instead of ad-hoc shell commands</h2>
<p>The conventional approach involves SSHing into the server and running a series of commands copied from a wiki page. This has problems:</p>
<ol>
<li><strong>Reproducibility</strong> — ad-hoc commands are easy to mistype, especially the <code>pg_upgrade</code> invocation which requires exact paths to both old and new PostgreSQL binaries.</li>
<li><strong>Binary path management</strong> — on a flake-based NixOS system, there’s no <code><nixpkgs></code> channel. You can’t run <code>nix-build '<nixpkgs>' -A postgresql_14</code> to get a package path. The packages live in the Nix store, and their paths are determined at evaluation time by the flake lock file.</li>
<li><strong>Atomicity</strong> — by encoding the procedure in the flake, the exact PG 14 and PG 18 store paths are baked into the scripts at build time. The scripts and the NixOS configuration reference the same nixpkgs revision, eliminating version skew.</li>
</ol>
<h2 id="flake-structure">Flake structure</h2>
<h3 id="pinning-the-nixpkgs-revision">Pinning the nixpkgs revision</h3>
<p>The flake uses a <code>nixpkgs-stable</code> input pinned to <code>nixos-25.11</code>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">inputs = {
nixpkgs-stable = {
url = "github:NixOS/nixpkgs/nixos-25.11";
};
};
</code></pre>
<p>In the per-system block:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">let
pkgs = nixpkgs.legacyPackages.${system};
pkgs-stable = nixpkgs-stable.legacyPackages.${system};
in
</code></pre>
<p>This matters: the server uses <code>nixpkgs-stable</code> (NixOS 25.11), not <code>nixpkgs</code> (unstable). The PostgreSQL packages baked into the scripts must come from the same nixpkgs revision that the server’s NixOS configuration uses. The PG 18 binary in the upgrade script will be byte-for-byte identical to the one NixOS runs as a service after deployment.</p>
<h3 id="script-definitions">Script definitions</h3>
<p>The scripts are defined inside the flake’s <code>eachDefaultSystem</code> block, guarded by <code>optionalAttrs</code> so they only exist on x86_64-linux:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">pgUpgradeScripts = pkgs.lib.optionalAttrs (system == "x86_64-linux") (
let
pg14 = pkgs-stable.postgresql_14;
pg18 = pkgs-stable.postgresql_18;
oldDir = "/var/lib/postgresql/14";
newDir = "/var/lib/postgresql/18";
in
{
# step1, step2, step3 defined here
}
);
</code></pre>
<p>When Nix evaluates <code>${pg14}</code>, it becomes something like <code>/nix/store/abc123-postgresql-14.22</code>. When it evaluates <code>${pg18}</code>, it becomes <code>/nix/store/xyz789-postgresql-18.3</code>. The resulting shell scripts contain hardcoded, fully-qualified store paths. No runtime resolution, no PATH dependency, no risk of picking up the wrong binary. Both PostgreSQL versions are pulled into each script’s closure, meaning <code>nix run .#postgresql-upgrade-14-18-step1</code> on the server automatically fetches both PG 14 and PG 18 binaries from the cache before executing.</p>
<h3 id="exposing-as-packages-and-apps">Exposing as packages and apps</h3>
<p>The scripts are merged into the flake’s <code>packages</code> output:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">packages = {
default = myPackage;
} // pgUpgradeScripts;
</code></pre>
<p>And automatically mapped to <code>apps</code>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">apps = builtins.mapAttrs (name: pkg: {
type = "app";
program = "${pkg}/bin/${name}";
}) pgUpgradeScripts;
</code></pre>
<p>This enables <code>nix run .#postgresql-upgrade-14-18-step1</code> syntax without repeating the app definition for each script.</p>
<h2 id="why-three-steps-with-human-checkpoints">Why three steps with human checkpoints</h2>
<p>The migration is deliberately split into three scripts:</p>
<ul>
<li><strong>Step 1</strong> runs while PG 14 is live. If anything fails, abort with zero consequences.</li>
<li><strong>Step 2</strong> performs the irreversible <code>pg_upgrade</code>. Separated so the operator reviews step 1’s output before committing.</li>
<li><strong>Step 3</strong> runs after deploying the new NixOS configuration with PG 18. A <code>nixos-rebuild switch</code> must happen between step 2 and step 3.</li>
</ul>
<p>You could automate all three into one script. You probably shouldn’t. The gap between “preflight passed” and “actually rewrite my production database catalogs” is exactly where you want a human reading terminal output and deciding whether to proceed.</p>
<h2 id="step-1-backup-and-preflight">Step 1: backup and preflight</h2>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">set -euo pipefail
echo "=== Step 1: Backup and preflight check ==="
[[ $EUID -eq 0 ]] || { echo "Run as root"; exit 1; }
echo "Checking current checksum status..."
sudo -u postgres ${pg14}/bin/pg_controldata ${oldDir} | grep -i checksum
echo ""
echo "Checking for MD5 passwords..."
sudo -u postgres ${pg14}/bin/psql -c \
"SELECT rolname,
CASE WHEN rolpassword LIKE 'md5%'
THEN 'MD5 (migrate to SCRAM!)'
ELSE 'OK'
END AS auth
FROM pg_authid WHERE rolpassword IS NOT NULL;"
echo ""
echo "Checking for expression indexes..."
sudo -u postgres ${pg14}/bin/psql -At -c \
"SELECT schemaname || '.' || indexname || ': ' || indexdef
FROM pg_indexes
WHERE indexdef ~ '\\(.*\\('" 2>/dev/null || true
echo ""
echo "Checking for FTS indexes..."
sudo -u postgres ${pg14}/bin/psql -At -c \
"SELECT schemaname || '.' || indexname
FROM pg_indexes
WHERE indexdef LIKE '%tsvector%'
OR indexdef LIKE '%gin%'
OR indexdef LIKE '%gist%'" 2>/dev/null || true
echo ""
echo "Taking pg_dumpall backup..."
mkdir -p /var/backup
sudo -u postgres ${pg14}/bin/pg_dumpall \
> "/var/backup/postgresql-14-pre-upgrade-$(date +%Y%m%d).sql"
echo "Backup saved to /var/backup/"
echo ""
echo "Listing databases for reference..."
sudo -u postgres ${pg14}/bin/psql -l
echo ""
echo "Stopping PostgreSQL..."
systemctl stop postgresql.service
echo "Creating socket directory..."
mkdir -p /var/run/postgresql
chown postgres:postgres /var/run/postgresql
checksum_status=$(sudo -u postgres ${pg14}/bin/pg_controldata ${oldDir} \
| grep "Data page checksum" | awk '{print $NF}')
initdb_flags=""
if [ "$checksum_status" = "0" ] \
|| echo "$checksum_status" | grep -qi "off\|disabled"; then
echo "Old cluster has checksums DISABLED — passing --no-data-checksums to initdb"
initdb_flags="--no-data-checksums"
fi
echo "Initializing new data directory..."
sudo -u postgres ${pg18}/bin/initdb $initdb_flags -D ${newDir}
echo "Running pg_upgrade --check (dry run)..."
cd /var/lib/postgresql
sudo -u postgres ${pg18}/bin/pg_upgrade \
--socketdir=/var/run/postgresql \
--old-bindir=${pg14}/bin \
--new-bindir=${pg18}/bin \
--old-datadir=${oldDir} \
--new-datadir=${newDir} \
--check
echo ""
echo "=== Preflight passed. Proceed to step 2. ==="
echo "NOTE: PostgreSQL is stopped. If aborting, run:"
echo " rm -rf ${newDir} && systemctl start postgresql.service"
</code></pre>
<h3 id="what-each-check-does">What each check does</h3>
<p><strong>Checksum status</strong> — this is the single most critical check for a 14→18 upgrade. PG 18 changed the <code>initdb</code> default to enable data page checksums. If the old cluster has checksums disabled (<code>Data page checksum version: 0</code>), the new cluster must also be initialized without them. Miss this and <code>pg_upgrade</code> refuses to proceed because the checksum settings don’t match.</p>
<p><strong>MD5 password audit</strong> — PG 18 deprecates MD5 password hashing in favor of SCRAM-SHA-256. The query against <code>pg_authid</code> identifies roles still on MD5 so you can plan migration before or after the upgrade, rather than discovering deprecation warnings flooding your logs at 2 AM.</p>
<p><strong>Expression indexes</strong> — PG 17 tightened security around <code>search_path</code> in functions used by expression indexes and materialized views. The check identifies expression indexes so you can verify they’ll survive the upgrade.</p>
<p><strong>FTS/GIN/GiST indexes</strong> — PG 18 changed the full-text search collation provider. After <code>pg_upgrade</code>, FTS and trigram indexes need reindexing because the collation metadata embedded in the index may no longer match. Identifying them upfront tells you what step 3’s reindex will cover.</p>
<p><strong>pg_dumpall</strong> — a full logical backup as a safety net. If something goes catastrophically wrong, this SQL dump restores the entire cluster to a fresh PG 14 installation. The old data directory is also preserved until explicitly deleted in cleanup, providing a second recovery path.</p>
<p><strong>Conditional initdb</strong> — creates the PG 18 data directory at <code>/var/lib/postgresql/18</code>, passing <code>--no-data-checksums</code> only if the old cluster has them disabled. NixOS convention uses <code>/var/lib/postgresql/<major-version></code> as the data directory.</p>
<p><strong>pg_upgrade –check</strong> — the <code>--check</code> flag performs a complete dry run. It verifies binary compatibility, checks for data types that changed representation, validates that required shared libraries exist, and confirms the clusters are upgrade-compatible — all without modifying any data. If this passes, the real upgrade will pass.</p>
<h2 id="step-2-the-actual-migration">Step 2: the actual migration</h2>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">set -euo pipefail
echo "=== Step 2: Run pg_upgrade ==="
[[ $EUID -eq 0 ]] || { echo "Run as root"; exit 1; }
echo "Ensuring PostgreSQL is stopped..."
systemctl stop postgresql.service 2>/dev/null || true
mkdir -p /var/run/postgresql
chown postgres:postgres /var/run/postgresql
echo "Running pg_upgrade..."
cd /var/lib/postgresql
sudo -u postgres ${pg18}/bin/pg_upgrade \
--socketdir=/var/run/postgresql \
--old-bindir=${pg14}/bin \
--new-bindir=${pg18}/bin \
--old-datadir=${oldDir} \
--new-datadir=${newDir}
echo ""
echo "=== pg_upgrade completed. ==="
echo ""
echo "Next steps:"
echo " 1. Update postgresql config to set package = pkgs.postgresql_18"
echo " 2. Commit, push, deploy"
echo " 3. Run step 3 for post-upgrade verification"
</code></pre>
<p>This is the short one. It does exactly one thing: run <code>pg_upgrade</code> for real. The operation proceeds in phases internally:</p>
<ol>
<li><strong>Global objects</strong> — roles, tablespaces, and other cluster-wide objects are dumped from the old cluster and restored into the new one.</li>
<li><strong>Schema dump/restore</strong> — each database’s schema is dumped as SQL from PG 14 and replayed into PG 18. This is where catalog format differences are resolved.</li>
<li><strong>Data file copy</strong> — the actual table and index data files are copied from old to new. Since we’re on the same filesystem, this is a straightforward copy. The data files are forward-compatible at the page level.</li>
<li><strong>Transaction metadata</strong> — XID counters, multixact state, and WAL position are carried forward for transactional continuity.</li>
<li><strong>Extension update script</strong> — <code>pg_upgrade</code> generates <code>update_extensions.sql</code> for extensions that need <code>ALTER EXTENSION ... UPDATE</code>. Step 3 applies this.</li>
</ol>
<p>The entire operation completes in seconds for moderately-sized databases because <code>pg_upgrade</code> doesn’t re-process row data. It just rewrites the catalog and links the data files.</p>
<h2 id="the-deploy">The deploy</h2>
<p>Between step 2 and step 3, deploy the new NixOS configuration. The PostgreSQL module change is minimal:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix"># services/postgresql.nix
{ pkgs, ... }:
{
services.postgresql = {
enable = true;
package = pkgs.postgresql_18;
};
}
</code></pre>
<p><code>nixos-rebuild switch</code> activates this. NixOS builds a new system closure referencing <code>postgresql-18.3</code> instead of <code>postgresql-14.22</code>, generates new systemd unit files pointing to the PG 18 binary, starts <code>postgresql.service</code> against the <code>/var/lib/postgresql/18</code> data directory that <code>pg_upgrade</code> prepared, and starts all dependent services via systemd dependency ordering. One atomic transition.</p>
<h2 id="step-3-post-upgrade-verification">Step 3: post-upgrade verification</h2>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">set -euo pipefail
echo "=== Step 3: Post-upgrade verification ==="
[[ $EUID -eq 0 ]] || { echo "Run as root"; exit 1; }
echo "Checking PostgreSQL service status..."
systemctl status postgresql.service --no-pager
echo ""
echo "Checking PostgreSQL version..."
sudo -u postgres ${pg18}/bin/psql -c "SELECT version();"
echo ""
echo "Listing databases..."
sudo -u postgres ${pg18}/bin/psql -l
if [ -f /var/lib/postgresql/update_extensions.sql ]; then
echo ""
echo "Applying extension updates..."
sudo -u postgres ${pg18}/bin/psql \
-f /var/lib/postgresql/update_extensions.sql
fi
echo ""
echo "Reindexing all databases (required for FTS collation changes)..."
for db in $(sudo -u postgres ${pg18}/bin/psql -At -c \
"SELECT datname FROM pg_database WHERE datistemplate = false;"); do
echo " Reindexing $db..."
sudo -u postgres ${pg18}/bin/reindexdb "$db" \
|| echo " WARNING: reindex of $db failed"
done
echo ""
echo "Checking for MD5 passwords (deprecated in PG18)..."
sudo -u postgres ${pg18}/bin/psql -c \
"SELECT rolname,
CASE WHEN rolpassword LIKE 'md5%'
THEN 'MD5 — MIGRATE TO SCRAM!'
ELSE 'SCRAM (ok)'
END AS auth
FROM pg_authid WHERE rolpassword IS NOT NULL;"
echo ""
echo "Running ANALYZE on all databases..."
sudo -u postgres ${pg18}/bin/vacuumdb --all --analyze-only
echo ""
echo "Checking dependent services..."
for svc in service-a service-b service-c analytics webmail; do
status=$(systemctl is-active "$svc" 2>/dev/null || echo "not found")
printf " %-25s %s\n" "$svc" "$status"
done
echo ""
echo "=== Verification complete ==="
echo ""
echo "If everything looks good:"
echo " 1. Remove old data directory: rm -rf ${oldDir}"
echo " 2. Delete pg_upgrade logs/scripts in /var/lib/postgresql/"
echo " 3. If MD5 passwords were found, migrate them to SCRAM-SHA-256"
</code></pre>
<p><strong>Version confirmation</strong> — <code>SELECT version()</code> should return “PostgreSQL 18.x”, confirming the new binary is serving the migrated data.</p>
<p><strong>Extension updates</strong> — applies <code>update_extensions.sql</code> generated by <code>pg_upgrade</code>. This runs <code>ALTER EXTENSION ... UPDATE</code> for extensions whose catalog entries need updating (e.g., citext).</p>
<p><strong>Full reindex</strong> — every database is reindexed with <code>reindexdb</code>. Essential after a major upgrade to PG 18 because of the FTS collation provider change — text search indexes built under PG 14’s collation assumptions may produce incorrect results under PG 18. Rebuilds all indexes from source data, guaranteeing correctness.</p>
<p><strong>ANALYZE</strong> — <code>vacuumdb --all --analyze-only</code> refreshes the query planner’s statistics for every table in every database. <code>pg_upgrade</code> preserves data files but not planner statistics, so without this the planner defaults to sequential scans until autovacuum eventually collects fresh stats. You don’t want to discover this via a production latency spike.</p>
<p><strong>Service health check</strong> — confirms all dependent services reconnected and are running after the PG version change.</p>
<h2 id="what-makes-this-nixos-specific">What makes this NixOS-specific</h2>
<p><strong>No package manager state</strong> — on Debian you’d <code>apt install postgresql-18</code>, manage <code>pg_lsclusters</code>, deal with the <code>postgresql-common</code> wrapper, and worry about APT restarting services. On NixOS the package is a store path. Installing PG 18 doesn’t affect the running PG 14 until you activate a new system configuration.</p>
<p><strong>stateVersion controls defaults</strong> — the server was still on PG 14 despite running NixOS 25.11 which ships PG 17 for new installs. This is by design — NixOS won’t silently change your database version. The explicit <code>package = pkgs.postgresql_18;</code> pin overrides the stateVersion default and documents the intentional upgrade.</p>
<p><strong>Atomic system activation</strong> — <code>nixos-rebuild switch</code> doesn’t upgrade PostgreSQL in isolation. It atomically transitions the entire system — PostgreSQL, dependent services, systemd units, environment — in a single operation. If the activation fails, NixOS rolls back to the previous generation with PG 14 still configured.</p>
<p><strong>Flake-pinned binaries</strong> — the upgrade scripts and the NixOS configuration derive from the same <code>flake.lock</code>. The PG 18 binary used for <code>pg_upgrade</code> is byte-for-byte identical to the one NixOS runs as a service. On a traditional system, there’s always a risk that the PG 18 you installed for the upgrade is a slightly different build than what the package manager configures as the service.</p>
<p><strong>Garbage collection awareness</strong> — after the upgrade, <code>nix-collect-garbage</code> will eventually remove the PG 14 store path since nothing references it. The upgrade scripts still reference it (PG 14 is in their closure), so as long as the scripts exist in the flake, the PG 14 binary remains available for rollback. Remove the scripts from the flake and PG 14 becomes eligible for garbage collection. This is a nice property — your safety net exists exactly as long as you keep the scripts around, and it disappears automatically when you clean up.</p>
<h2 id="cleanup">Cleanup</h2>
<p>After verification confirms everything is healthy:</p>
<ul>
<li>Remove <code>/var/lib/postgresql/14</code> — the old data directory</li>
<li>Delete <code>delete_old_cluster.sh</code> — <code>pg_upgrade</code>’s generated cleanup script</li>
<li>Delete <code>update_extensions.sql</code> — already applied</li>
</ul>
<p>The final state of <code>/var/lib/postgresql/</code> should contain only the <code>18</code> directory. Remove the upgrade scripts from your flake when you’re confident you won’t need to roll back. Until then, they serve as both documentation and a safety net — keeping PG 14 in the Nix store costs a few hundred megabytes and buys you the option to investigate if something subtle surfaces later.</p>
Building Portable Rust CLI Binaries with Nix — Static Linux, Portable macOS2026-03-18T14:00:00+00:002026-03-18T14:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/portable-rust-binaries-nix/<p>You build a Rust binary with Nix. It works on your machine. Ship it to a colleague’s Linux box and it segfaults or complains about missing <code>glibc</code>. Ship the macOS binary to someone without Nix and it fails with <code>dyld: Library not loaded: /nix/store/.../libiconv.2.dylib</code>. The binary has Nix store paths burned into its dynamic linker references.</p>
<p>What you actually want:</p>
<ul>
<li><strong>Linux</strong>: a truly static binary. Zero runtime deps. Runs on any distro, any container, bare metal.</li>
<li><strong>macOS</strong>: a portable binary that only depends on system libraries (<code>libSystem</code>, <code>libiconv</code>) at their standard paths, not Nix store paths.</li>
</ul>
<p>This post covers how to get both from a single flake, plus cross-compilation between platforms.</p>
<h2 id="the-baseline-package">The baseline package</h2>
<p>Start with a standard <code>rustPlatform.buildRustPackage</code>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">packages.my-cli = pkgs.rustPlatform.buildRustPackage {
pname = "my-cli";
version = "0.1.0";
src = ./.;
cargoLock.lockFile = ./Cargo.lock;
};
</code></pre>
<p>If your CLI lives in a subdirectory of a monorepo, use <code>buildAndTestSubdir</code> and copy the lockfile up:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">packages.my-cli = pkgs.rustPlatform.buildRustPackage {
pname = "my-cli";
version = "0.1.0";
src = ./.;
buildAndTestSubdir = "cli";
cargoLock.lockFile = ./cli/Cargo.lock;
postUnpack = ''
cp $sourceRoot/cli/Cargo.lock $sourceRoot/Cargo.lock
'';
};
</code></pre>
<p>This builds a working binary per platform via <code>eachDefaultSystem</code>, but it’s dynamically linked and not portable.</p>
<h2 id="static-builds-platform-conditional-logic">Static builds: platform-conditional logic</h2>
<p>Linux and macOS need fundamentally different approaches. Use <code>stdenv.hostPlatform.isLinux</code> to branch:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">packages.my-cli-static =
if pkgs.stdenv.hostPlatform.isLinux then
# Fully static musl binary
pkgs.pkgsStatic.rustPlatform.buildRustPackage {
pname = "my-cli";
version = "0.1.0";
src = ./.;
cargoLock.lockFile = ./Cargo.lock;
doCheck = false;
}
else
# macOS: static Rust runtime, rewrite dylib paths for portability
pkgs.rustPlatform.buildRustPackage {
pname = "my-cli";
version = "0.1.0";
src = ./.;
cargoLock.lockFile = ./Cargo.lock;
RUSTFLAGS = "-C target-feature=+crt-static";
nativeBuildInputs = [ pkgs.darwin.cctools ];
doCheck = false;
postInstall = ''
install_name_tool -change \
${pkgs.libiconv}/lib/libiconv.2.dylib \
/usr/lib/libiconv.2.dylib \
$out/bin/my-cli
'';
};
</code></pre>
<p>Two completely different strategies behind the same attribute name. Let’s break each one down.</p>
<h2 id="linux-pkgsstatic-and-musl">Linux: pkgsStatic and musl</h2>
<p><code>pkgs.pkgsStatic</code> is a nixpkgs overlay that swaps glibc for musl and forces static linking across all packages. When you use <code>pkgs.pkgsStatic.rustPlatform.buildRustPackage</code>, the Rust toolchain links against musl libc statically. The result is a single binary with zero dynamic dependencies:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">$ file result/bin/my-cli
result/bin/my-cli: ELF 64-bit LSB executable, x86-64, statically linked, stripped
$ ldd result/bin/my-cli
not a dynamic executable
</code></pre>
<p>This binary runs on any Linux — Alpine, Ubuntu, Debian, RHEL, bare containers, you name it.</p>
<p>Disable tests (<code>doCheck = false</code>) for the static variant — test suites sometimes rely on dynamic features or network access that break under musl static linking. Run your tests in the dynamic build instead.</p>
<h2 id="macos-why-you-can-t-go-fully-static">macOS: why you can’t go fully static</h2>
<p>macOS doesn’t support fully static binaries. Apple requires dynamic linking against <code>libSystem.B.dylib</code> (the kernel interface). There’s no musl equivalent for Darwin.</p>
<p>What you <em>can</em> do:</p>
<ol>
<li>Statically link the Rust/C runtime with <code>RUSTFLAGS = "-C target-feature=+crt-static"</code></li>
<li>Rewrite any remaining Nix store dylib references to system paths</li>
</ol>
<p><code>install_name_tool</code> (from <code>pkgs.darwin.cctools</code>) rewrites the Mach-O load commands. After rewriting, <code>otool -L</code> should show only system paths:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">$ otool -L result/bin/my-cli
result/bin/my-cli:
/usr/lib/libiconv.2.dylib (...)
/usr/lib/libSystem.B.dylib (...)
</code></pre>
<p>No <code>/nix/store/...</code> references. The binary works on any macOS system.</p>
<p>If your binary links against additional C libraries, you’ll need one <code>install_name_tool -change</code> per library. Check <code>otool -L</code> on the binary before rewriting to see what needs fixing.</p>
<h2 id="cross-compilation">Cross-compilation</h2>
<h3 id="building-linux-from-macos">Building Linux from macOS</h3>
<p>With <code>eachDefaultSystem</code>, each system gets its own package set. To build a Linux binary from macOS:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix build .#packages.x86_64-linux.my-cli-static
</code></pre>
<p>This works because nixpkgs has cross-compilation toolchains built in. Nix fetches the appropriate cross-compiler and musl target libraries. The result is a static Linux ELF binary built on your Mac.</p>
<p>For aarch64 Linux (ARM servers, Raspberry Pi, Graviton):</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix build .#packages.aarch64-linux.my-cli-static
</code></pre>
<h3 id="building-macos-from-linux">Building macOS from Linux</h3>
<p>This also works in the other direction:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix build .#packages.aarch64-darwin.my-cli-static
nix build .#packages.x86_64-darwin.my-cli-static
</code></pre>
<p>Though in practice, macOS cross-compilation from Linux is less common and can hit edge cases with Apple framework headers. If you have a Mac available, prefer native builds for Darwin targets.</p>
<h3 id="all-four-static-targets">All four static targets</h3>
<p>With <code>eachDefaultSystem</code> + the platform-conditional static build, one flake produces:</p>
<table><thead><tr><th>Target</th><th>Linking</th><th>Technique</th></tr></thead><tbody>
<tr><td><code>x86_64-linux</code></td><td>Fully static (musl)</td><td><code>pkgsStatic</code></td></tr>
<tr><td><code>aarch64-linux</code></td><td>Fully static (musl)</td><td><code>pkgsStatic</code></td></tr>
<tr><td><code>x86_64-darwin</code></td><td>Partial static + rewrite</td><td><code>crt-static</code> + <code>install_name_tool</code></td></tr>
<tr><td><code>aarch64-darwin</code></td><td>Partial static + rewrite</td><td><code>crt-static</code> + <code>install_name_tool</code></td></tr>
</tbody></table>
<h2 id="cargo-toml-optimize-for-release-size">Cargo.toml: optimize for release size</h2>
<p>For CLI tools, optimize the release profile for binary size:</p>
<pre data-lang="toml" class="language-toml "><code class="language-toml" data-lang="toml">[profile.release]
lto = true # Link-time optimization — slower build, smaller binary
strip = true # Strip debug symbols
codegen-units = 1 # Single codegen unit — better optimization, slower build
opt-level = "z" # Optimize aggressively for size
</code></pre>
<p>This typically cuts binary size by 50–70% compared to default release settings.</p>
<h2 id="avoid-openssl-use-rustls">Avoid OpenSSL — use rustls</h2>
<p>If your CLI makes HTTPS requests, use <code>rustls</code> (pure Rust TLS) instead of <code>openssl-sys</code>. OpenSSL is a C dependency that’s painful to cross-compile and link statically. Most Rust HTTP/gRPC crates support a <code>rustls</code> feature flag:</p>
<pre data-lang="toml" class="language-toml "><code class="language-toml" data-lang="toml">[dependencies]
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls"] }
# or for gRPC:
tonic = { version = "0.12", features = ["tls-rustls"] }
</code></pre>
<p>Zero C TLS dependencies = cross-compilation just works.</p>
<h2 id="protobuf-codegen">Protobuf codegen</h2>
<p>If your CLI uses Protocol Buffers (e.g., gRPC), provide <code>protoc</code> via Nix:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">nativeBuildInputs = [ pkgs.protobuf ];
PROTOC = "${pkgs.protobuf}/bin/protoc";
</code></pre>
<p>With a <code>build.rs</code> that runs <code>tonic_build</code> or <code>prost_build</code>:</p>
<pre data-lang="rust" class="language-rust "><code class="language-rust" data-lang="rust">fn main() -> Result<(), Box<dyn std::error::Error>> {
tonic_build::configure()
.build_server(false)
.compile_protos(&["proto/service.proto"], &["proto"])?;
Ok(())
}
</code></pre>
<p>Protobuf codegen is text generation — it’s platform-independent and needs no special cross-compilation handling. Just make sure <code>PROTOC</code> is set so it finds the compiler.</p>
<h2 id="making-it-runnable-via-nix-run">Making it runnable via <code>nix run</code></h2>
<p>Register the package as an app:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">apps.my-cli = {
type = "app";
program = "${self.packages.${system}.my-cli}/bin/my-cli";
};
</code></pre>
<p>Then: <code>nix run .#my-cli -- --help</code></p>
<h2 id="why-not-crane-or-naersk">Why not crane or naersk?</h2>
<p><code>rustPlatform.buildRustPackage</code> is built into nixpkgs and works natively with <code>pkgsStatic</code>. External tools like crane or naersk add incremental build caching (useful for CI), but for a single CLI binary the added complexity isn’t worth it. <code>cargoLock.lockFile</code> gives you reproducibility, <code>pkgsStatic</code> gives you musl — that’s all you need.</p>
<h2 id="putting-it-together">Putting it together</h2>
<p>The full pattern in one flake: a dynamic build for development and testing, a static/portable build for distribution, cross-compilation across all four targets, and <code>nix run</code> support. The platform-conditional logic handles the Linux/macOS split transparently — consumers of the flake don’t need to think about it. They <code>nix build .#my-cli-static</code> and get a binary that works anywhere their target OS runs.</p>
Deploying Elixir/Phoenix on NixOS — What Actually Works2026-03-17T14:00:00+00:002026-03-17T14:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/elixir-phoenix-nixos/<p>So this started, as these things usually do, with wanting to run staging and production on the same box. One Phoenix app, two instances, different ports, different databases, different secrets. NixOS should make this declarative and clean. And it does — eventually. But there is a surprising amount of ceremony involved in getting an Elixir release to build under Nix, and the multi-instance NixOS module pattern has enough moving parts that you will get at least three of them wrong on the first try.</p>
<p>This post covers the full path: building the Elixir release in Nix, structuring the NixOS module for multiple instances, wiring up PostgreSQL and secrets without shell script hacks, and the various traps along the way.</p>
<h2 id="building-the-release-in-nix">Building the release in Nix</h2>
<p>Elixir releases under Nix use <code>mixRelease</code> from the BEAM package set. Pin your Erlang version explicitly — don’t let it float with nixpkgs:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">beamPackages = pkgs.beam.packages.erlang_27;
</code></pre>
<p>Fetch mix dependencies offline with <code>fetchMixDeps</code>. This is Nix’s equivalent of <code>mix deps.get</code>, but reproducible and sandboxed:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">mixFodDeps = beamPackages.fetchMixDeps {
pname = "my-app-mix-deps";
version = "0.1.0";
src = ./.;
sha256 = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
};
</code></pre>
<p>The <code>sha256</code> is a fixed-output derivation hash. Get the real value by running the build once with a fake hash and letting Nix tell you the correct one.</p>
<p>The Nix sandbox has no network access, which means esbuild and tailwind cannot self-install the way they normally do via mix. You have to hand them the binaries from nixpkgs:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">packages.default = beamPackages.mixRelease {
pname = "my-app";
version = "0.1.0";
src = ./.;
inherit mixFodDeps;
MIX_ESBUILD_PATH = "${pkgs.esbuild}/bin/esbuild";
MIX_TAILWIND_PATH = "${pkgs.tailwindcss}/bin/tailwindcss";
postBuild = ''
mix assets.deploy --no-deps-check
'';
fixupPhase = ''
echo "my_app_cookie" > $out/releases/COOKIE
'';
};
</code></pre>
<p>The <code>--no-deps-check</code> on <code>mix assets.deploy</code> is important — deps are already compiled at that point and the check would fail in the sandbox. The cookie in <code>fixupPhase</code> is for Erlang distribution; set it to something deterministic so nodes can cluster.</p>
<p>One more thing in <code>mix.exs</code> — disable compile-time environment validation in your release config:</p>
<pre data-lang="elixir" class="language-elixir "><code class="language-elixir" data-lang="elixir">releases: [my_app: [validate_compile_env: false]]
</code></pre>
<p>The build environment and runtime environment are different machines with different env vars. Without this, the release will refuse to start because it detects a mismatch.</p>
<h2 id="heroicons-skip-mix-wire-it-manually">Heroicons: skip mix, wire it manually</h2>
<p>If your Phoenix app uses heroicons, you have likely already discovered that it tries to do a git clone during compilation. That does not work in a Nix sandbox. The fix is to fetch heroicons separately and symlink it into <code>deps/</code>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">heroiconsSrc = pkgs.fetchFromGitHub {
owner = "tailwindlabs";
repo = "heroicons";
rev = "v2.2.0";
hash = "sha256-...";
};
</code></pre>
<p>In the package’s <code>preBuild</code>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">preBuild = ''
mkdir -p deps
ln -sf ${heroiconsSrc} deps/heroicons
'';
</code></pre>
<p>Mirror this in your devShell so local development sees the same thing:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">shellHook = ''
mkdir -p deps
ln -sfn ${heroiconsSrc} deps/heroicons
'';
</code></pre>
<p>Both environments resolve heroicons identically. No hex install, no git clone, no network.</p>
<h2 id="the-multi-instance-module">The multi-instance module</h2>
<p>The core idea is an <code>instances</code> attrset where each key is an instance name and each value is a submodule with its own ports, database, secrets, and system user. One NixOS module produces N systemd services, N PostgreSQL databases, and N nginx vhosts from a single declaration.</p>
<p>Start with the root options:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">options.services.myApp = {
instances = mkOption {
type = types.attrsOf (types.submodule instanceModule);
default = {};
};
package = mkOption { type = types.package; };
};
</code></pre>
<p>The instance submodule defines everything an instance needs to run:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">instanceModule = { name, config, ... }: {
options = {
enable = mkOption { type = types.bool; default = true; };
package = mkOption { type = types.package; default = cfg.package; };
user = mkOption {
type = types.str;
default = "my_app_${sanitizeName name}";
};
group = mkOption { type = types.str; default = config.user; };
listenAddress = mkOption { type = types.str; default = "127.0.0.1"; };
port = mkOption { type = types.port; default = 4000; };
metricsPort = mkOption { type = types.nullOr types.port; default = null; };
host = mkOption { type = types.str; };
scheme = mkOption {
type = types.enum [ "http" "https" ];
default = "https";
};
secretKeyBaseFile = mkOption { type = types.path; };
database = {
host = mkOption { type = types.str; default = "/run/postgresql"; };
name = mkOption { type = types.str; default = config.user; };
passwordFile = mkOption {
type = types.nullOr types.path;
default = null;
};
};
autoMigrate = mkOption { type = types.bool; default = false; };
nginxHelper = {
enable = mkOption { type = types.bool; default = false; };
domain = mkOption { type = types.str; default = config.host; };
enableACME = mkOption { type = types.bool; default = false; };
};
};
};
</code></pre>
<p>Note the <code>sanitizeName</code> helper — NixOS system users cannot have dashes:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">sanitizeName = name: builtins.replaceStrings [ "-" ] [ "_" ] name;
</code></pre>
<p>An instance named <code>"production"</code> gets user <code>my_app_production</code>. An instance named <code>"us-east"</code> gets user <code>my_app_us_east</code>. Without this, NixOS will reject the user creation and the error message will not point you anywhere useful.</p>
<h2 id="filtering-instances">Filtering instances</h2>
<p>Not every piece of config applies to every instance. An instance using an external database should not trigger local PostgreSQL provisioning. An instance without nginx should not generate a vhost. Filter the instances into categories early:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">enabledInstances = filterAttrs (_: icfg: icfg.enable) cfg.instances;
localPgInstances = filterAttrs (_: icfg:
icfg.database.passwordFile == null
) enabledInstances;
nginxInstances = filterAttrs (_: icfg:
icfg.nginxHelper.enable
) enabledInstances;
</code></pre>
<p>Then gate each config section on the relevant subset. This prevents one instance’s settings from dragging in services that another instance does not need.</p>
<h2 id="one-user-per-instance-no-exceptions">One user per instance, no exceptions</h2>
<p>Each instance gets its own system user and group. This is not optional. If staging and production share a user, a misconfigured secret path in staging can expose production credentials. systemd’s <code>LoadCredential</code> binds files to the service’s user context — sharing users breaks that isolation.</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">users.users = mapAttrs' (_: icfg:
nameValuePair icfg.user {
isSystemUser = true;
group = icfg.group;
home = "/var/lib/my-app";
}
) enabledInstances;
users.groups = mapAttrs' (_: icfg:
nameValuePair icfg.group {}
) enabledInstances;
</code></pre>
<h2 id="assertions-catch-mistakes-at-eval-time">Assertions catch mistakes at eval time</h2>
<p>When you have multiple instances, the most common misconfiguration is duplicate values — two instances claiming the same port, database name, or domain. These should fail at <code>nixos-rebuild</code> time, not at 3am when the second service fails to bind:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">assertions = let
allPorts = allValues (icfg: icfg.port);
allDomains = mapAttrsToList (_: icfg:
icfg.nginxHelper.domain
) nginxInstances;
allDbNames = allValues (icfg: icfg.database.name);
in [
{
assertion = length allPorts == length (unique allPorts);
message = "myApp: all port values must be unique across instances.";
}
{
assertion = length allDomains == length (unique allDomains);
message = "myApp: all nginx domains must be unique across instances.";
}
{
assertion = length allDbNames == length (unique allDbNames);
message = "myApp: all database names must be unique across instances.";
}
];
</code></pre>
<p>This turns a runtime mystery into a build-time error with a clear message. Add assertions for every value that must be unique — ports, gRPC ports, metrics ports, domains, database names.</p>
<h2 id="generating-systemd-services">Generating systemd services</h2>
<p>One systemd service per instance, generated with <code>mapAttrs'</code>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">systemd.services = mapAttrs' (name: icfg:
let
stateDir = "my-app/${name}";
runtimeDir = "my-app-${name}";
pkg = icfg.package;
in nameValuePair "my-app-${name}" {
description = "My App (${name})";
wantedBy = [ "multi-user.target" ];
after = [ "network.target" "postgresql.service" ];
wants = [ "postgresql.service" ];
environment = {
PHX_SERVER = "true";
PORT = toString icfg.port;
LISTEN_ADDRESS = icfg.listenAddress;
PHX_HOST = icfg.host;
RELEASE_NODE = "my_app_${sanitizeName name}";
RELEASE_TMP = "/tmp/my-app-${name}";
DATABASE_HOST = icfg.database.host;
DATABASE_NAME = icfg.database.name;
DATABASE_USER = icfg.user;
LANG = "en_US.UTF-8";
};
serviceConfig = {
Type = "exec";
User = icfg.user;
Group = icfg.group;
Restart = "on-failure";
RestartSec = 5;
WorkingDirectory = "/var/lib/${stateDir}";
StateDirectory = stateDir;
RuntimeDirectory = runtimeDir;
NoNewPrivileges = true;
ProtectSystem = "strict";
ProtectHome = true;
PrivateTmp = true;
PrivateDevices = true;
ProtectKernelTunables = true;
ProtectKernelModules = true;
ProtectControlGroups = true;
RestrictSUIDSGID = true;
RemoveIPC = true;
LoadCredential = lib.filter (x: x != null) [
"secret_key_base:${icfg.secretKeyBaseFile}"
(if icfg.database.passwordFile != null
then "db_password:${icfg.database.passwordFile}"
else null)
];
};
script = ''
export SECRET_KEY_BASE="$(< $CREDENTIALS_DIRECTORY/secret_key_base)"
${lib.optionalString (icfg.database.passwordFile != null) ''
export DATABASE_PASSWORD="$(< $CREDENTIALS_DIRECTORY/db_password)"
''}
${lib.optionalString icfg.autoMigrate ''
${pkg}/bin/my_app eval 'MyApp.Release.migrate()'
''}
exec ${pkg}/bin/my_app start
'';
}
) enabledInstances;
</code></pre>
<p>A few things worth calling out:</p>
<p><strong><code>RELEASE_NODE</code> must be unique per instance.</strong> Erlang uses this to identify nodes for clustering and remote shells. Two instances with the same node name on the same host will collide, and the second one will silently fail to start distributed Erlang — or worse, connect to the first one’s runtime.</p>
<p><strong><code>StateDirectory</code> and <code>RuntimeDirectory</code></strong> are namespaced per instance. systemd creates these automatically with the correct ownership. No <code>mkdir -p</code> in shell scripts, no <code>chown</code> hacks.</p>
<p><strong>Security hardening</strong> is applied uniformly. <code>ProtectSystem = "strict"</code> makes the filesystem read-only except for the state and runtime directories. <code>PrivateTmp</code> gives each service its own <code>/tmp</code>. This is free isolation — there is no reason not to use it.</p>
<h2 id="frontload-everything-in-the-start-script">Frontload everything in the start script</h2>
<p>Elixir releases read config at boot via <code>runtime.exs</code>. You cannot inject environment variables after the BEAM starts — there is no hot-reload of system env in a running release.</p>
<p>Do not use systemd <code>Environment=</code> for secrets. Those values end up in the unit file and are visible in <code>/proc</code>. Use <code>LoadCredential</code> instead — systemd places the file contents in <code>$CREDENTIALS_DIRECTORY</code>, and the start script reads them into env vars before exec.</p>
<p>The pattern is: non-secret config goes in <code>environment</code>, secrets go through <code>LoadCredential</code>, and the <code>script</code> block ties them together. Migrations run in the same script block because they need the same env vars. Do not try to run migrations in a separate <code>ExecStartPre</code> — it will not have your database credentials.</p>
<p>All env, all secrets, all pre-start tasks — one script block. One context. Nothing leaking between phases.</p>
<h2 id="postgresql-without-shell-script-hacks">PostgreSQL without shell script hacks</h2>
<p>The temptation is to use <code>ExecStartPre</code> to run <code>createuser</code>, <code>createdb</code>, and <code>psql -c "GRANT ..."</code>. This works until it does not. You end up fighting race conditions with the postgresql service, handling “already exists” errors with <code>|| true</code>, and accumulating shell that breaks on the next NixOS upgrade.</p>
<p>The right approach: the system user the service runs under has the same name as the database. NixOS wires this correctly through <code>ensureDatabases</code> and <code>ensureUsers</code>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.postgresql = mkIf (localPgInstances != {}) {
ensureDatabases = mapAttrsToList (_: icfg:
icfg.database.name
) localPgInstances;
ensureUsers = let
dbUsers = unique (mapAttrsToList (_: icfg:
icfg.user
) localPgInstances);
in map (u: {
name = u;
ensureDBOwnership = true;
}) dbUsers;
};
</code></pre>
<p>With <code>ensureDBOwnership = true</code>, the PostgreSQL role automatically owns its database. The app connects over a Unix socket at <code>/run/postgresql</code> — peer authentication, no password needed for local connections, no <code>pg_hba.conf</code> hacks.</p>
<p>In <code>runtime.exs</code>:</p>
<pre data-lang="elixir" class="language-elixir "><code class="language-elixir" data-lang="elixir">db_host = System.get_env("DATABASE_HOST", "/run/postgresql")
host_opts =
if String.starts_with?(db_host, "/"),
do: [socket_dir: db_host],
else: [hostname: db_host]
</code></pre>
<p>This handles both the NixOS case (Unix socket) and external databases (TCP hostname) from the same code path. The gating on <code>localPgInstances</code> ensures that instances using a remote database URL do not trigger local PostgreSQL provisioning.</p>
<h2 id="nginx-with-content-type-routing">Nginx with content-type routing</h2>
<p>When an instance enables <code>nginxHelper</code>, the module auto-generates an nginx vhost. If your app serves both HTTP and gRPC on different ports, you can route based on content type:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.nginx = mkIf (nginxInstances != {}) {
enable = true;
virtualHosts = mapAttrs' (_: icfg:
nameValuePair icfg.nginxHelper.domain ({
forceSSL = icfg.nginxHelper.enableACME;
enableACME = icfg.nginxHelper.enableACME;
locations."~ ^/" = {
proxyPass = "http://${icfg.listenAddress}:${toString icfg.port}";
proxyWebsockets = true;
};
})
) nginxInstances;
};
</code></pre>
<p><code>proxyWebsockets = true</code> is essential for Phoenix LiveView. Without it, the WebSocket upgrade fails and LiveView falls back to long-polling — or just breaks, depending on your configuration.</p>
<p>ACME is optional per instance. Staging might use a test CA or no TLS at all. Production uses Let’s Encrypt. The module does not assume.</p>
<h2 id="putting-it-together">Putting it together</h2>
<p>Here is what a two-instance deployment looks like from the consumer’s side:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
services.myApp = {
package = my-app.packages.${pkgs.system}.default;
instances = {
production = {
host = "app.example.com";
port = 4000;
secretKeyBaseFile = /run/secrets/prod-secret-key;
database.name = "my_app_prod";
autoMigrate = true;
nginxHelper.enable = true;
nginxHelper.enableACME = true;
};
staging = {
host = "app-staging.example.com";
port = 4001;
secretKeyBaseFile = /run/secrets/staging-secret-key;
database.name = "my_app_staging";
autoMigrate = true;
nginxHelper.enable = true;
nginxHelper.enableACME = true;
};
};
};
}
</code></pre>
<p>From this declaration, <code>nixos-rebuild</code> produces:</p>
<ul>
<li>Two systemd services: <code>my-app-production</code>, <code>my-app-staging</code></li>
<li>Two system users: <code>my_app_production</code>, <code>my_app_staging</code></li>
<li>Two state directories under <code>/var/lib/my-app/</code></li>
<li>Two PostgreSQL databases with matching owners</li>
<li>Two nginx vhosts with separate ACME certificates</li>
<li>Assertions that prevent port, domain, or database collisions</li>
</ul>
<p>Add a third instance and you get a third of everything. Remove one and NixOS cleans up the service and vhost. The database and user persist — NixOS does not drop databases on removal, which is the correct default.</p>
<h2 id="what-i-would-do-differently">What I would do differently</h2>
<p>If I were starting over, I would use <code>writeShellApplication</code> instead of bare <code>script</code> blocks for the pre-start logic. <code>writeShellApplication</code> runs shellcheck and adds <code>set -euo pipefail</code> automatically — it catches the kind of bugs that only surface in production when a credential file is missing and the script silently continues with an empty variable.</p>
<p>I would also add health checks from day one. systemd supports <code>Type = "notify"</code> with a watchdog, and there are Elixir libraries that integrate with it. Without health checks, a service that starts but stops accepting connections will sit there in “active (running)” state until someone notices manually.</p>
<p>But those are refinements. The patterns above — typed instance submodules, <code>mapAttrs'</code> for service generation, <code>LoadCredential</code> for secrets, <code>ensureDatabases</code> for PostgreSQL, assertions for uniqueness — these are the load-bearing walls. Get them right and everything else is furniture.</p>
Solving the NixOS SOPS Bootstrap Problem2026-03-16T20:00:00+00:002026-03-16T20:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/sops-bootstrap-problem/<p>You’ve just finished a <a href="/blog/nixos-anywhere-ovh-kimsufi/">nixos-anywhere</a> deploy. The server rebooted into NixOS. You SSH in, feeling good about yourself, and run <code>nixos-rebuild switch --flake .#myhost</code>. Nix starts evaluating your flake, hits a private GitHub input, and dies:</p>
<pre><code>error: unable to download 'https://api.github.com/repos/yourorg/private-flake/tarball/...': HTTP error 404
</code></pre>
<p>Your flake has private inputs. The access tokens that authenticate those fetches are managed by sops-nix. sops-nix decrypts secrets during NixOS activation. But you haven’t activated the new configuration yet — that’s what you’re trying to do. The host can’t build its own config because building requires tokens that only exist after a successful build.</p>
<p>Welcome to the bootstrap problem.</p>
<h2 id="how-sops-nix-secrets-work-the-short-version">How sops-nix secrets work (the short version)</h2>
<p>If you’re already familiar with sops-nix, skip ahead. For everyone else, here’s the minimum context.</p>
<p>Secrets live in encrypted YAML files in your repo — <code>secrets/myhost.yaml</code>, for instance. A <code>.sops.yaml</code> file at the repo root maps path patterns to the age public keys that can decrypt them. Each NixOS host has an age key derived from its SSH host key:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">ssh-keyscan myhost.example.com 2>/dev/null | ssh-to-age
</code></pre>
<p>During NixOS activation, sops-nix takes those encrypted YAML files, decrypts them using the host’s age private key, and drops the plaintext into <code>/run/secrets/</code>. One of those secrets is typically <code>nix_builder_access_tokens</code> — a file containing GitHub PATs that Nix reads via <code>nix.extraOptions</code>:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">nix.extraOptions = ''
!include /run/secrets/nix_builder_access_tokens
'';
</code></pre>
<p>This is the mechanism. Encrypted at rest, decrypted on activation, consumed by Nix for authenticated fetches. It works beautifully — once the system is running. The problem is the first time.</p>
<h2 id="the-chicken-and-the-egg">The chicken and the egg</h2>
<p>After a fresh nixos-anywhere install, the host boots into a minimal NixOS. It has an SSH host key (nixos-anywhere generated one), so it <em>could</em> decrypt sops secrets — if those secrets had been deployed. But they haven’t been, because the first real <code>nixos-rebuild switch</code> is what deploys them.</p>
<p>And that first <code>nixos-rebuild switch</code> can’t complete, because it needs to fetch private flake inputs, which requires the access tokens, which are in the sops secrets, which haven’t been deployed yet.</p>
<pre><code>Fresh install
→ needs nixos-rebuild switch to deploy sops secrets
→ nixos-rebuild needs private flake inputs
→ private flake inputs need access tokens
→ access tokens are in sops secrets
→ sops secrets haven't been deployed yet
→ goto: needs nixos-rebuild switch
</code></pre>
<p>If you’ve ever written a Nix expression that infinitely recurses, this should feel familiar. Same energy, different layer of the stack.</p>
<h2 id="the-fix-build-somewhere-else">The fix: build somewhere else</h2>
<p>The insight is simple: the <em>target</em> host can’t build its own config, but some other host that already has decrypted tokens can. You just need to separate where the build happens from where the result gets activated.</p>
<p><code>nixos-rebuild</code> has two flags for exactly this:</p>
<ul>
<li><strong><code>--target-host</code></strong> — the machine where the built system closure gets copied to and activated</li>
<li><strong><code>--build-host</code></strong> — the machine where the Nix build actually runs</li>
</ul>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run nixpkgs#nixos-rebuild -- switch \
--flake .#myhost \
--target-host root@myhost.example.com \
--build-host root@builder.example.com
</code></pre>
<p>This tells nixos-rebuild: SSH into <code>builder.example.com</code>, run the build there (where access tokens are already decrypted and available), copy the resulting closure to <code>myhost.example.com</code>, and activate it. The target host never needs to fetch a single flake input. It just receives a pre-built system closure and switches to it.</p>
<h2 id="why-nix-run-nixpkgs-nixos-rebuild">Why <code>nix run nixpkgs#nixos-rebuild</code></h2>
<p>If you’re managing NixOS servers from a Mac (as I am), <code>nixos-rebuild</code> isn’t in your PATH — it’s a NixOS tool, not a Nix tool. <code>nix run nixpkgs#nixos-rebuild --</code> runs it from nixpkgs without installing it. The <code>--</code> separates <code>nix run</code> flags from <code>nixos-rebuild</code> flags. This is one of those things that’s obvious in hindsight and confusing for about fifteen minutes when you first encounter it.</p>
<h2 id="a-real-workflow-dedicated-server-after-nixos-anywhere">A real workflow: dedicated server after nixos-anywhere</h2>
<p>Here’s the concrete scenario. You’ve just deployed NixOS onto a Kimsufi dedicated server using nixos-anywhere. The server rebooted, you can SSH in, NixOS is running. But this is the bare-bones config — no sops secrets deployed yet.</p>
<p>First, grab the new host’s age key (nixos-anywhere generates a new SSH host key on every run):</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">ssh-keyscan anubis.example.com 2>/dev/null | ssh-to-age
</code></pre>
<p>Update <code>.sops.yaml</code> with the new key, then re-encrypt:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">sops updatekeys secrets/anubis.yaml
</code></pre>
<p>Now the bootstrap deploy, building on an existing host that already has tokens:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run nixpkgs#nixos-rebuild -- switch \
--flake .#anubis \
--target-host root@anubis.example.com \
--build-host root@builder.example.com
</code></pre>
<p>The builder compiles the full system closure — fetching private inputs with its own decrypted tokens — and ships it to the new host. NixOS activates, sops-nix runs, and suddenly <code>anubis</code> has its own decrypted access tokens sitting in <code>/run/secrets/</code>.</p>
<p>From this point forward, the host can build its own configuration. Subsequent deploys are straightforward:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">ssh anubis nixos-rebuild switch --flake .#anubis
</code></pre>
<p>The bootstrap problem is a one-time obstacle. You climb over it once with <code>--build-host</code>, and then it’s gone.</p>
<h2 id="when-the-build-host-is-your-local-machine">When the build host is your local machine</h2>
<p>You don’t always have a separate build server. If your local machine has the tokens (because you’re working from a checkout of the flake and your Nix config includes the access tokens), you can omit <code>--build-host</code> entirely:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run nixpkgs#nixos-rebuild -- switch \
--flake .#myhost \
--target-host root@myhost.example.com
</code></pre>
<p>This builds locally and copies the closure to the target. If you’re on a Mac, this means cross-compiling for Linux (or using a remote builder), which is its own adventure. But if you’re on a Linux workstation with the right tokens configured, this is the simplest path.</p>
<h2 id="beyond-bootstrap">Beyond bootstrap</h2>
<p>The <code>--build-host</code> / <code>--target-host</code> pattern isn’t only useful for the sops bootstrap. It’s the right tool any time the target can’t or shouldn’t build its own config:</p>
<ul>
<li><strong>Underpowered hardware</strong> — Raspberry Pis, small VPSes, anything where a full Nix build would take an hour or swap itself to death.</li>
<li><strong>Cross-compilation avoidance</strong> — building an aarch64 config on an x86_64 host (or vice versa) via a same-architecture build host is often faster than cross-compiling locally.</li>
<li><strong>Minimal attack surface</strong> — some production hosts intentionally don’t have Nix build capabilities. They receive pre-built closures and activate them. No compiler, no fetcher, no GitHub tokens on the box at all.</li>
</ul>
<p>The bootstrap problem is the most annoying instance of needing this pattern, but it’s far from the only one.</p>
<h2 id="the-full-picture">The full picture</h2>
<p>Putting it all together, here’s the lifecycle of a new NixOS host from bare metal to self-sufficient:</p>
<ol>
<li><strong>nixos-anywhere</strong> installs NixOS onto the target (minimal config, no secrets)</li>
<li><strong>ssh-keyscan | ssh-to-age</strong> gets the new host’s age public key</li>
<li><strong>sops updatekeys</strong> re-encrypts secrets for the new key</li>
<li><strong>nixos-rebuild –build-host</strong> does the first deploy using an existing host’s tokens</li>
<li><strong>sops-nix activates</strong> and decrypts access tokens on the target</li>
<li><strong>The host can now self-build</strong> — subsequent deploys need no external builder</li>
</ol>
<p>Steps 2-4 are the bootstrap dance. You do it once per host, and then you never think about it again — until the next fresh install, when you’ll wish you had bookmarked this post.</p>
Deploying NixOS on OVH Kimsufi with nixos-anywhere2026-03-14T20:00:00+00:002026-03-14T20:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/nixos-anywhere-ovh-kimsufi/<p>So you’ve found yourself a cheap dedicated server on OVH Kimsufi (now branded “OVH Eco”, because marketing), and you want to run NixOS on it. OVH doesn’t offer NixOS as an install option. What they do offer is a rescue mode that gives you root SSH access to a Debian live environment. That’s all <a rel="noopener noreferrer" target="_blank" href="https://github.com/nix-community/nixos-anywhere">nixos-anywhere</a> needs.</p>
<p>This is a walkthrough of how I deployed NixOS onto a Kimsufi server with two 4TB HGST disks, using disko for declarative partitioning and mdadm RAID-0, and the various traps I walked into along the way.</p>
<h2 id="the-disk-layout">The disk layout</h2>
<p>Kimsufi servers in the lower tiers come with two spinning rust disks. I wanted both of them in a RAID-0 for maximum usable space.</p>
<p>Here’s the full disko config:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
disko.devices = {
disk = {
disk0 = {
type = "disk";
device = "/dev/disk/by-id/ata-HGST_HUS726T4TALA6L1_V6GXVWPS";
content = {
type = "gpt";
partitions = {
esp = {
size = "512M";
type = "EF00";
content = {
type = "filesystem";
format = "vfat";
mountpoint = "/boot/ESP0";
mountOptions = [ "umask=0077" ];
};
};
mdadm-root = {
size = "100%";
content = {
type = "mdraid";
name = "root";
};
};
};
};
};
disk1 = {
type = "disk";
device = "/dev/disk/by-id/ata-HGST_HUS726T4TALA6L1_V6GEAMHS";
content = {
type = "gpt";
partitions = {
esp = {
size = "512M";
type = "EF00";
content = {
type = "filesystem";
format = "vfat";
mountpoint = "/boot/ESP1";
mountOptions = [ "umask=0077" ];
};
};
mdadm-root = {
size = "100%";
content = {
type = "mdraid";
name = "root";
};
};
};
};
};
};
mdadm = {
root = {
type = "mdadm";
level = 0;
content = {
type = "filesystem";
format = "ext4";
mountpoint = "/";
};
};
};
};
}
</code></pre>
<h2 id="the-boot-config-that-actually-works">The boot config that actually works</h2>
<p>Disko creates the RAID arrays, but it does <em>not</em> tell NixOS to assemble them at boot. That’s on you. If you forget this line:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">boot.swraid.enable = true;
</code></pre>
<p>…congratulations, you’ve just installed a system that will never boot. You’ll find this out after waiting for the server to come back up, panicking, entering rescue mode, and staring at an initrd that has no idea what mdadm is.</p>
<p>The full boot section:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">boot.loader.systemd-boot.enable = false;
boot.loader.grub = {
enable = true;
efiSupport = true;
mirroredBoots = [
{ devices = [ "nodev" ]; path = "/boot/ESP0"; }
{ devices = [ "nodev" ]; path = "/boot/ESP1"; }
];
};
boot.loader.efi.canTouchEfiVariables = true;
boot.swraid.enable = true;
</code></pre>
<p>The <code>mirroredBoots</code> is the key piece — GRUB installs itself to both ESPs, so either disk can boot the system. (In this case the data on the RAID-0 is purely ephemeral, but for those running RAID-1 with data that matters, this mirrored boot setup is the important part — your system stays bootable even if a disk dies.) The <code>"nodev"</code> device means “don’t try to install to an MBR, this is pure EFI”. And <code>swraid.enable</code> — don’t forget it. I’ll say it again. Don’t forget it.</p>
<h2 id="running-nixos-anywhere">Running nixos-anywhere</h2>
<h3 id="disable-ovh-monitoring-first">Disable OVH monitoring first</h3>
<p>Before you do anything else, go to the OVH panel and disable their monitoring. The kexec step replaces the running kernel in-place, which looks to OVH’s monitoring like your server just died. This triggers an automated “intervention” that can temporarily lock your netboot settings in the panel. Ask me how I know.</p>
<h3 id="the-install">The install</h3>
<p>Boot the server into OVH rescue mode from the panel. Once you can SSH in as root, from your local machine:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run github:nix-community/nixos-anywhere -- \
--flake .#my-server \
--phases kexec,disko,install \
root@x.x.x.x
</code></pre>
<p>The <code>--phases kexec,disko,install</code> is not optional. This is the single most important flag and the hill I will die on.</p>
<h3 id="why-no-reboot-phase">Why no reboot phase</h3>
<p>Kimsufi rescue mode is netboot-based. Every time the server reboots, it checks the OVH panel for which boot device to use. If it’s still set to “rescue”, you’re back in Debian. If nixos-anywhere includes the <code>reboot</code> phase (which is the default), it will happily reboot the server right back into rescue mode, not into your freshly installed NixOS.</p>
<p>The full workflow is:</p>
<ol>
<li>Disable OVH monitoring in the panel (if you haven’t already — seriously, do this)</li>
<li>Boot the server into OVH rescue mode</li>
<li>nixos-anywhere installs with <code>--phases kexec,disko,install</code> (no reboot)</li>
<li>Go to the OVH panel, change netboot from “rescue” to “Boot from the hard disk”</li>
<li>Reboot the server from the OVH panel</li>
<li>NixOS boots</li>
</ol>
<h3 id="secrets-management-dance">Secrets management dance</h3>
<p>nixos-anywhere generates a new SSH host key on every run. If you’re using sops-nix (and you should be), the age key derived from that host key will change. After installation completes:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># Get the new age key
ssh-keyscan x.x.x.x 2>/dev/null | ssh-to-age
# Update .sops.yaml with the new key, then:
sops updatekeys secrets/my-server.yaml
</code></pre>
<p>Then on first real boot, push the updated secrets:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix shell nixpkgs#nixos-rebuild -c nixos-rebuild switch \
--flake .#my-server --target-host root@x.x.x.x
</code></pre>
<h3 id="verify-before-you-deploy">Verify before you deploy</h3>
<p>Each failed boot on Kimsufi costs you real time — you have to enter rescue mode, wait for it, SSH in, re-run nixos-anywhere, wait for that, change the boot device again, reboot again. Before you commit to a deploy, check the critical bits:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix eval .#nixosConfigurations.my-server.config.boot.swraid.enable
# should print: true
nix eval .#nixosConfigurations.my-server.config.boot.loader.grub.efiSupport
# should print: true
</code></pre>
<p>A 30-second <code>nix eval</code> can save you 30 minutes of rescue mode cycling.</p>
<h2 id="the-java-web-start-kvm-a-portal-to-2008">The Java Web Start KVM: a portal to 2008</h2>
<p>Kimsufi servers don’t have standard IPMI/BMC consoles. What they have is an “IP KVM” that gives you a <code>.jnlp</code> file — a Java Web Start launcher. In 2026. Oracle killed Java Web Start in Java 11 (2018). OVH apparently did not get the memo.</p>
<p>You can’t just double-click a <code>.jnlp</code> file on a modern system and have anything happen. You need a Java 8 runtime and a way to parse the JNLP XML, download the referenced JARs, and launch them with the right classpath. This is exactly the kind of problem Nix was born to solve.</p>
<p>Here’s a standalone <code>jnlp-run.nix</code> you can use without pulling in anyone’s entire dotfiles flake:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix"># jnlp-run.nix — run ancient JNLP files with nix-shell
# Usage: nix-shell jnlp-run.nix --run "jnlp-run /path/to/kvm.jnlp"
let
pkgs = import <nixpkgs> { };
jdk = if pkgs.stdenv.isDarwin then pkgs.zulu8 else pkgs.jdk8;
in
pkgs.mkShell {
packages = [
(pkgs.writeShellScriptBin "jnlp-run" ''
export PATH="${pkgs.lib.makeBinPath [ jdk pkgs.xmlstarlet pkgs.curl pkgs.coreutils ]}"
if [ -z "$1" ]; then
echo "Usage: jnlp-run <file.jnlp>" >&2
exit 1
fi
jnlp="$1"
codebase=$(xml sel -t -v '/jnlp/@codebase' "$jnlp")
jar_href=$(xml sel -t -v '/jnlp/resources/jar/@href' "$jnlp")
jar_url="''${codebase}/''${jar_href}"
os=$(uname -s)
arch=$(uname -m)
case "''${os}-''${arch}" in
Linux-x86_64) native_href=$(xml sel -t -v \
'//resources[@os="Linux" and (@arch="x86_64" or @arch="amd64")]/nativelib/@href' \
"$jnlp" 2>/dev/null) ;;
Linux-i*86) native_href=$(xml sel -t -v \
'//resources[@os="Linux" and (@arch="x86" or @arch="i386")]/nativelib/@href' \
"$jnlp" 2>/dev/null) ;;
*) native_href="" ;;
esac
mapfile -t args < <(xml sel -t -v '/jnlp/application-desc/argument' -n "$jnlp")
tmpdir=$(mktemp -d)
trap 'rm -rf "$tmpdir"' EXIT
echo "Downloading $jar_url ..."
curl -ksSL -o "$tmpdir/app.jar" "$jar_url"
if [ -n "$native_href" ]; then
native_url="''${codebase}/''${native_href}"
echo "Downloading native lib $native_url ..."
curl -ksSL -o "$tmpdir/native.jar" "$native_url"
(cd "$tmpdir" && jar xf native.jar)
fi
echo "Launching with $(java -version 2>&1 | head -1) ..."
exec java -Djava.library.path="$tmpdir" -jar "$tmpdir/app.jar" "''${args[@]}"
'')
];
}
</code></pre>
<p>Save that to a file, download the <code>.jnlp</code> from the OVH panel, and:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix-shell jnlp-run.nix --run "jnlp-run ~/Downloads/kvm_x.x.x.x.jnlp"
</code></pre>
<p>Nix pulls in Java 8 (Zulu on macOS, OpenJDK on Linux), xmlstarlet, and curl into an ephemeral shell. The script parses the JNLP XML, downloads the KVM applet JARs, grabs any platform-specific native libraries, and launches the whole thing. When you close the shell, it’s all gone. No system-wide Java 8 installation polluting your machine.</p>
<p>It’s a thoroughly modern solution to a thoroughly ancient problem. You get a VGA console view of your server that looks like it was rendered by a GPU from the Bush administration, but it works — and sometimes “it works” is all you need to figure out why your boot is hanging.</p>
<h2 id="the-hard-won-lessons-summarized">The hard-won lessons, summarized</h2>
<p>For the skimmers and the future-me who will inevitably forget all of this:</p>
<ul>
<li><strong><code>--phases kexec,disko,install</code></strong> — always. No reboot phase. Ever.</li>
<li><strong>Disable OVH monitoring</strong> before kexec, or enjoy your locked panel.</li>
<li><strong><code>boot.swraid.enable = true</code></strong> — disko won’t set this for you.</li>
<li><strong>UEFI, not BIOS</strong> — use <code>EF00</code> partitions, not <code>EF02</code>.</li>
<li><strong>Age keys change</strong> on every nixos-anywhere run — update sops immediately.</li>
<li><strong><code>nix eval</code> your boot config</strong> before deploying — rescue mode round-trips are not fun.</li>
<li><strong>The KVM is Java Web Start</strong> — Nix can run it, your OS cannot.</li>
</ul>
<h2 id="wrapping-up">Wrapping up</h2>
<p>From a blank Kimsufi server in rescue mode to a fully declarative NixOS system with RAID-0, mirrored boot, WireGuard tunnels, encrypted secrets, and remote deployment — all in an evening’s work whilst sipping a good Spanish Alhambra beer. There are worse ways to spend a Friday night.</p>
A Lightweight Prometheus Exporter for Bhyve VMs2026-03-13T16:00:00+00:002026-03-13T16:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/prometheus-bhyve-exporter/<p>I run a handful of bhyve VMs on FreeBSD, managed by <a rel="noopener noreferrer" target="_blank" href="https://github.com/churchers/vm-bhyve">vm-bhyve</a>. Nothing exotic — just a straightforward setup without CBSD, RCTL, or RACCT. I wanted basic visibility into what each VM was doing: is it up, how much CPU is it burning, how much memory is it actually using. The kind of thing you glance at on a Grafana dashboard and move on.</p>
<p>The existing option for this on FreeBSD is <a rel="noopener noreferrer" target="_blank" href="https://github.com/tycho-kirchner/rctl_exporter">rctl_exporter</a>, which instruments FreeBSD’s resource control framework. It’s a solid tool, but it’s designed for a broader use case — RACCT/RCTL accounting across jails, processes, and login classes. If you’re already using RCTL limits and want to monitor those, it makes sense. But if you’re running a simple vm-bhyve setup without RACCT enabled, pulling in the full RCTL machinery just to see if your VMs are alive and how much memory they’re eating is overkill.</p>
<p>So I wrote <a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/prometheus-bhyve-exporter">prometheus-bhyve-exporter</a>.</p>
<h2 id="what-it-does">What it does</h2>
<p>The exporter is a small Rust binary that parses the output of <code>vm list</code> (from vm-bhyve) and queries <code>ps</code> for each running VM’s bhyve process. On each Prometheus scrape it collects:</p>
<ul>
<li><code>bhyve_vm_up</code> — whether the VM is running (1) or stopped (0)</li>
<li><code>bhyve_vm_cpu_allocated</code> — number of CPUs allocated</li>
<li><code>bhyve_vm_memory_allocated_bytes</code> — memory allocated in bytes</li>
<li><code>bhyve_vm_cpu_usage_percent</code> — current CPU usage from ps</li>
<li><code>bhyve_vm_memory_rss_bytes</code> — resident set size</li>
<li><code>bhyve_vm_memory_vsz_bytes</code> — virtual memory size</li>
<li><code>bhyve_vm_pid</code> — PID of the bhyve process</li>
</ul>
<p>All are gauges. All per-VM metrics carry a <code>vm</code> label with the VM name. There are also <code>bhyve_exporter_scrape_duration_seconds</code> and <code>bhyve_exporter_scrape_errors_total</code> for monitoring the exporter itself.</p>
<p>No kernel modules, no RACCT, no RCTL. Just <code>vm list</code> and <code>ps</code>.</p>
<h2 id="installation-on-freebsd">Installation on FreeBSD</h2>
<p>You need a Rust toolchain. The repo includes a Makefile and the scaffolding to build a proper FreeBSD package.</p>
<h3 id="from-source">From source</h3>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">git clone https://github.com/ijohanne/prometheus-bhyve-exporter.git
cd prometheus-bhyve-exporter
sudo make install
</code></pre>
<p>This drops the binary into <code>/usr/local/bin/</code> and an rc.d script into <code>/usr/local/etc/rc.d/</code>.</p>
<h3 id="as-a-freebsd-package">As a FreeBSD package</h3>
<p>If you prefer a proper <code>.pkg</code> that shows up in <code>pkg info</code> and can be cleanly removed:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">git clone https://github.com/ijohanne/prometheus-bhyve-exporter.git
cd prometheus-bhyve-exporter
make package
sudo pkg add prometheus-bhyve-exporter-0.1.0.pkg
</code></pre>
<h3 id="enable-and-start-the-service">Enable and start the service</h3>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">sudo sysrc bhyve_exporter_enable=YES
sudo service bhyve_exporter start
</code></pre>
<p>The exporter listens on <code>0.0.0.0:9288</code> by default. You can override the listen address and the path to the <code>vm</code> command via <code>rc.conf</code>:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">sudo sysrc bhyve_exporter_listen_address="127.0.0.1:9288"
sudo sysrc bhyve_exporter_vm_command="/usr/local/sbin/vm"
</code></pre>
<h2 id="scraping-with-prometheus">Scraping with Prometheus</h2>
<p>Point Prometheus at it the usual way:</p>
<pre data-lang="yaml" class="language-yaml "><code class="language-yaml" data-lang="yaml">scrape_configs:
- job_name: bhyve
static_configs:
- targets: ['your-freebsd-host:9288']
</code></pre>
<p>If you’re managing Prometheus through NixOS (like I am), the equivalent in your scrape config looks like:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
job_name = "bhyve";
honor_labels = true;
static_configs = [
{
targets = [ "your-freebsd-host:9288" ];
labels = { instance = "your-freebsd-host"; };
}
];
}
</code></pre>
<h2 id="grafana-dashboard">Grafana dashboard</h2>
<p>The repo includes a <a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/prometheus-bhyve-exporter/blob/master/grafana/dashboard.json">Grafana dashboard</a> that you can import directly. It gives you an overview of VM status, CPU usage, and memory consumption at a glance.</p>
<p>Here’s what the CPU and memory panels look like in practice:</p>
<p><img src="https://perlpimp.net/blog/prometheus-bhyve-exporter/cpu-usage.png" alt="CPU usage per VM" /></p>
<p><img src="https://perlpimp.net/blog/prometheus-bhyve-exporter/memory-rss.png" alt="Memory RSS per VM" /></p>
<p>Nothing fancy — just the metrics that matter when you want to know if a VM is misbehaving or if you need to rebalance resources.</p>
<h2 id="future-plans">Future plans</h2>
<p>I’m considering submitting this as a PR to the <a rel="noopener noreferrer" target="_blank" href="https://www.freebsd.org/ports/">FreeBSD ports tree</a> once I’ve run it for a while longer and I’m confident it’s stable enough. The port scaffolding is already in the repo, so the jump from “builds locally” to “available in ports” shouldn’t be a large one.</p>
<h2 id="contributing">Contributing</h2>
<p>The project is MIT licensed and lives at <a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/prometheus-bhyve-exporter">github.com/ijohanne/prometheus-bhyve-exporter</a>. If you run bhyve VMs and want to help make it better — forks and pull requests are very welcome. Whether it’s new metrics, bug fixes, or packaging improvements — all welcome.</p>
Bootstrapping NixOS with a Template Generator2026-03-12T12:00:00+00:002026-03-12T12:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/bootstrapping-nixos-with-a-template-generator/<p>Every NixOS machine I manage starts the same way. A <code>configuration.nix</code> that imports the right modules. A <code>home.nix</code> that wires up the user’s shell, editor, and dev tooling. An entry in <code>flake.nix</code> that ties it all together. A deploy script that knows whether to rebuild locally or push to a remote host. A user registry entry with SSH keys and metadata.</p>
<p>None of this is hard. All of it is tedious, and tedious means error-prone. You copy an existing host directory, find-and-replace the hostname, forget to update the architecture, wonder why your aarch64 server is trying to pull x86_64 packages, and lose twenty minutes to a mistake that shouldn’t have been possible. I got tired of this, so I wrote a tool.</p>
<h2 id="the-setup-template">The setup-template</h2>
<p>The <a rel="noopener noreferrer" target="_blank" href="https://github.com/ijohanne/dotfiles-ng">dotfiles repo</a> now ships a Rust CLI called <code>setup-template</code>. It scaffolds new host and user configurations for the flake — interactively or from a config file. You run it, answer some questions, and it produces the files you would have written by hand, minus the transcription errors.</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run .#setup-template -- new
</code></pre>
<p>The wizard prompts for two things: who you are and what the machine is.</p>
<p>For the user, it asks for a username, full name, email, preferred shell, whether you want developer tooling, and any SSH public keys. For the host, it asks for a hostname, platform (Linux or Darwin), architecture, role (desktop or server), nixpkgs channel, deploy mode, and which optional modules to enable — secrets, neovim, and language toolchains.</p>
<p>When it finishes, it writes three files and prints a flake snippet to stdout:</p>
<ul>
<li><code>configs/users.nix</code> — your user added to the shared registry</li>
<li><code>hosts/<name>/configuration.nix</code> — the system config for your new machine</li>
<li><code>hosts/<name>/home.nix</code> — the home-manager config with your selected modules</li>
<li>A ready-to-paste <code>nixosConfigurations</code> or <code>darwinConfigurations</code> block</li>
</ul>
<h2 id="what-this-looks-like-for-a-site-host">What this looks like for a site host</h2>
<p>This site — perlpimp.net — is a Zola static site packaged as a Nix flake. The flake builds the HTML, compiles a CV to PDF via typst, and exports a NixOS module that sets up nginx with ACME, virtual hosts, domain redirects, and optional Plausible analytics. The module interface is small:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">services.perlpimpnet = {
enable = true;
domain = "perlpimp.net";
extraDomains = [ "www.perlpimp.net" ];
analytics.plausible.enable = true;
};
</code></pre>
<p>To host this, you need a NixOS server. That server needs a system configuration, a user, deploy tooling, and probably secrets management for anything beyond the static site itself. This is exactly what the template generator produces.</p>
<p>You would run the wizard, select Linux, your architecture, the server role, and enable secrets. The generator writes the host configuration with systemd-boot, a user account, and a deploy script that either rebuilds from a local checkout or pulls from GitHub over SSH. You paste the flake snippet, add the perlpimpnet flake as an input, import its NixOS module into your host config, and you have a deployable server.</p>
<p>The manual version of this process involves copying files from another host, editing half a dozen values across three files, and hoping you got the deploy script arguments right. The template version is a two-minute wizard followed by adding your site-specific imports. One of these scales. The other is how I used to do it.</p>
<h2 id="non-interactive-mode">Non-interactive mode</h2>
<p>The wizard is fine for one-off setups, but if you are provisioning multiple hosts — say a web server, a build server, and a desktop — you write a JSON config and run the generator in batch mode:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run .#setup-template -- generate --config setup.json
</code></pre>
<p>The config file is a versioned schema with arrays of users and hosts. Each host references its primary user by username, selects its modules, and declares its deploy mode. The generator validates the whole thing — primary users must exist in the users list, language selections must be from the supported set, Darwin hosts can’t be servers — and either produces all the files or tells you what’s wrong. No partial output, no half-generated state.</p>
<pre data-lang="json" class="language-json "><code class="language-json" data-lang="json">{
"version": 1,
"repo_ref": "github:ijohanne/dotfiles-ng",
"users": [{
"username": "ij",
"name": "Ian Johannesen",
"email": "ij@perlpimp.net",
"shell": "fish",
"developer": true,
"ssh_keys": ["ssh-ed25519 AAAA..."]
}],
"hosts": [{
"name": "web01",
"platform": "linux",
"arch": "x86_64",
"role": "server",
"nixpkgs": "unstable",
"deploy_mode": "remote",
"primary_user": "ij",
"modules": {
"secrets": true,
"neovim": true,
"languages": ["nix"]
}
}]
}
</code></pre>
<p>This is the same config file you would check into version control if you wanted a record of how each machine was initialised. The generator is deterministic — same input, same output — so the config file doubles as documentation.</p>
<h2 id="a-concrete-example-from-zero-to-serving-perlpimp-net">A concrete example: from zero to serving perlpimp.net</h2>
<p>Say you have a VPS from any provider that supports kexec — Hetzner, netcup, OVH, most of the usual suspects. The box is running whatever stock Linux the provider gave you. You want it running NixOS, serving this site, with secrets management and a deploy script, and you want it done in under an hour.</p>
<p>Here is the whole process.</p>
<h3 id="step-1-scaffold-the-host">Step 1: scaffold the host</h3>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">cd ~/git/dotfiles
nix run .#setup-template -- new
</code></pre>
<p>The wizard runs:</p>
<pre><code>── User ──────────────────────────────────
Username: ij
Full name: Ian Johannesen
Email: ij@perlpimp.net
Shell [fish/zsh/bash]: fish
Developer mode? [Y/n]: y
SSH public key (empty to finish): ssh-ed25519 AAAA... ij@macbook
SSH public key (empty to finish):
── Host ──────────────────────────────────
Hostname: web01
Platform [linux/darwin]: linux
Architecture [x86_64/aarch64]: x86_64
Role [desktop/server]: server
Nixpkgs channel [unstable/stable]: unstable
Deploy mode [local/remote]: remote
Enable secrets? [Y/n]: y
Enable neovim? [Y/n]: y
Languages [nix, rust, lua, markdown, flutter]: nix
</code></pre>
<p>It writes <code>configs/users.nix</code>, <code>hosts/web01/configuration.nix</code>, <code>hosts/web01/home.nix</code>, and prints the flake snippet.</p>
<h3 id="step-2-add-disko-and-the-site-module">Step 2: add disko and the site module</h3>
<p>The generator gives you a working system config, but the VPS needs disk partitioning and your site needs its NixOS module. You create a <code>hosts/web01/disko.nix</code> for the VPS disk layout — a small BIOS boot partition and the rest as ext4:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{
disko.devices = {
disk.main = {
type = "disk";
device = "/dev/vda";
content = {
type = "gpt";
partitions = {
boot = {
size = "1M";
type = "EF02";
};
root = {
size = "100%";
content = {
type = "filesystem";
format = "ext4";
mountpoint = "/";
};
};
};
};
};
};
}
</code></pre>
<p>Then you edit <code>hosts/web01/configuration.nix</code> to import disko and the perlpimpnet module:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">{ inputs, config, pkgs, lib, user, ... }:
let
deploy = import ../../configs/deploy { inherit pkgs; };
in
{
imports = [
../../configs/server.nix
./hardware-configuration.nix
./disko.nix
inputs.disko.nixosModules.disko
inputs.perlpimpnet.nixosModules.default
];
networking.hostName = "web01";
services.perlpimpnet = {
enable = true;
domain = "perlpimp.net";
extraDomains = [ "www.perlpimp.net" ];
analytics.plausible.enable = true;
};
environment.systemPackages = [
(deploy.mkDeployScript {
name = "deploy-web01";
host = "web01";
})
];
sops = {
defaultSopsFile = ../../secrets/web01.yaml;
age = {
sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
keyFile = "/var/lib/sops-nix/key.txt";
generateKey = true;
};
};
system.stateVersion = "25.11";
}
</code></pre>
<p>You add <code>perlpimpnet</code> and <code>disko</code> as flake inputs, paste the generated flake snippet, and commit.</p>
<h3 id="step-3-deploy-with-nixos-anywhere">Step 3: deploy with nixos-anywhere</h3>
<p>The VPS is running stock Debian or Ubuntu. You don’t need to install NixOS manually — <a rel="noopener noreferrer" target="_blank" href="https://github.com/nix-community/nixos-anywhere">nixos-anywhere</a> handles it. It SSH’s into the running Linux, kexec’s into a NixOS installer environment, partitions the disk using your disko config, and installs your flake configuration. One command:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">nix run github:nix-community/nixos-anywhere -- --flake .#web01 root@<vps-ip>
</code></pre>
<p>This partitions <code>/dev/vda</code> according to <code>disko.nix</code>, installs the full NixOS configuration including nginx, ACME, and the perlpimpnet module, and reboots. When it comes back up, the site is live. The whole thing takes a few minutes, most of which is the build and transfer.</p>
<h3 id="step-4-secrets-and-ongoing-deploys">Step 4: secrets and ongoing deploys</h3>
<p>After the first boot, grab the host’s age key for sops-nix:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">ssh-to-age-remote root@<vps-ip>
</code></pre>
<p>Add it to <code>.sops.yaml</code>, create <code>secrets/web01.yaml</code>, and you have encrypted secrets that only this host can decrypt.</p>
<p>From here on, deployments are a push and an SSH command:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">git push
ssh web01 deploy-web01
</code></pre>
<p>The deploy script on the host checks for a local checkout, falls back to GitHub if there isn’t one, and runs <code>nixos-rebuild switch --flake</code>. That is the entire workflow — template, customise, nixos-anywhere, deploy. No ISO to boot, no manual partitioning, no ansible playbooks, no forgetting which host has which configuration.</p>
<h2 id="what-happens-after-generation">What happens after generation</h2>
<p>The generator gets you to a buildable configuration. It does not do everything. After the files land, you still need to:</p>
<ol>
<li>Merge the generated <code>users.nix</code> into any existing user registry</li>
<li>Paste the flake snippet into <code>flake.nix</code></li>
<li>Add your site-specific module imports — in this case, the perlpimpnet NixOS module</li>
<li>If you enabled secrets, set up sops-nix: get the host’s age key, add it to <code>.sops.yaml</code>, create the secrets file</li>
<li>Build and deploy</li>
</ol>
<p>The tool prints these steps when it finishes. It does not try to be clever about automating things that require human judgment — like whether your secrets should use age or GPG, or which network interface your server uses. It handles the boilerplate. You handle the decisions.</p>
<h2 id="why-rust">Why Rust</h2>
<p>The generator is built with clap for argument parsing, dialoguer for the interactive prompts, and serde for config serialisation. It has unit tests for rendering determinism and schema validation. It builds with <code>rustPlatform.buildRustPackage</code> in the flake and participates in <code>nix flake check</code>.</p>
<p>I could have written this as a bash script. I have written this as a bash script, more than once, for other projects. Bash is fine for gluing together commands, but the moment you need structured input validation, schema versioning, or deterministic output, you are fighting the language instead of using it. A Rust binary that either produces correct output or fails with a clear error is worth the marginal extra effort over a shell script that silently does the wrong thing when you typo an argument.</p>
<h2 id="the-broader-point">The broader point</h2>
<p>NixOS already solves the “works on my machine” problem for system configuration. But the act of creating that configuration — the scaffolding step — is still a manual, copy-paste-and-edit workflow in most setups. The reproducibility of the system doesn’t help if the process of defining the system is ad hoc.</p>
<p>A template generator is not a novel idea. Every web framework has <code>create-app</code>. Every language has <code>init</code>. What’s less common is applying this to infrastructure definitions, where the cost of a wrong default is not a broken dev server but a misconfigured production host.</p>
<p>The goal is not to remove the need to understand what the configuration does. It is to remove the need to remember it. The generator encodes the conventions of the flake — where files go, how deploy scripts are wired, which modules exist — so that adding a new host is a matter of answering questions about the host, not about the flake’s internal structure. The knowledge lives in the tool, not in your head.</p>
Flutter and Nix: When Your SDK Thinks It Owns the Filesystem2026-03-08T10:00:00+00:002026-03-08T10:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/flutter-and-nix/<p>So this started, as these things usually do, with a build that wouldn’t build. I was setting up a Flutter project inside a Nix flake — nothing exotic, just wanting reproducible dev environments like a civilised person — and iOS builds were failing in ways that made no sense until they made too much sense.</p>
<p>What followed was a few hours of digging through Xcode build phases, Gradle internals, Apple code signing documentation, and Flutter issue trackers. I came out the other side with a working — if impure — set of workarounds and a much clearer picture of why Flutter and Nix are fundamentally at odds. This is that write-up.</p>
<h2 id="the-symptoms">The symptoms</h2>
<p>The immediate failures were straightforward enough. Builds would die because <code>codesign</code> couldn’t write to framework files that lived in the Nix store. <code>rsync</code> would fail copying frameworks because the source was read-only and the tooling expected to be able to modify what it copied. On the Android side, Gradle would try to create build output directories inside the Flutter SDK path itself — which, being in <code>/nix/store/</code>, wasn’t going to happen.</p>
<p>The error messages all pointed the same direction: Flutter expected to write to places that Nix had made read-only. Not build output directories — the SDK’s own installation.</p>
<h2 id="what-i-found-the-ios-side">What I found: the iOS side</h2>
<p>When you build a Flutter iOS app, Xcode invokes <code>xcode_backend.sh</code> during its Build Phases. This script copies <code>Flutter.framework</code> from the engine artifacts cache — under <code>bin/cache/artifacts/engine/ios</code> in the SDK tree — into the project and then into the built products directory. Older Flutter versions dumped it into <code>ios/Flutter/Flutter.framework</code>, newer ones target <code>BUILT_PRODUCTS_DIR</code> (see <a rel="noopener noreferrer" target="_blank" href="https://github.com/flutter/flutter/issues/70224">flutter/flutter#70224</a>).</p>
<p>These copies need to be writable because Apple’s <code>codesign</code> embeds signature data directly into the Mach-O binary. It physically modifies the file. There’s no detached signing option for embedded frameworks — the binary itself gets rewritten with the developer’s identity. That’s an Apple platform constraint, not something Flutter invented.</p>
<p>The thing is, the build output directory is already writable — that’s the whole point of a build directory. The correct approach is to read from the immutable SDK, copy into the mutable build output, and sign there. Flutter’s pipeline mostly does this now, but the assumption of SDK-level writability is still baked into enough places that it breaks on read-only filesystems.</p>
<h2 id="what-i-found-the-android-gradle-side">What I found: the Android/Gradle side</h2>
<p>The Flutter Gradle plugin tries to build under <code>$FLUTTER_SDK/flutter_tools/gradle</code>. That path is inside the SDK distribution. When <code>$FLUTTER_SDK</code> lives in the Nix store, Gradle fails with <code>Failed to create parent directory</code> errors pointing at <code>/nix/store/...</code>.</p>
<p>This is documented in <a rel="noopener noreferrer" target="_blank" href="https://github.com/NixOS/nixpkgs/issues/260278">NixOS/nixpkgs#260278</a> and <a rel="noopener noreferrer" target="_blank" href="https://github.com/NixOS/nixpkgs/issues/289936">#289936</a>. The workaround is to patch <code>build.gradle.kts</code> to redirect <code>buildDir</code> via an environment variable (<code>FLUTTER_GRADLE_PLUGIN_BUILDDIR</code>). It works, but it’s a patch against a design that shouldn’t need patching.</p>
<h2 id="what-i-found-the-sdk-itself">What I found: the SDK itself</h2>
<p>Beyond the platform-specific issues, Flutter’s SDK is architecturally hostile to immutability. It caches downloaded engine artifacts under <code>bin/cache/</code>. It runs self-update checks. Its version detection expects a <code>.git</code> directory to be present and writable. It was designed as a self-managing tool — think <code>rustup</code> or <code>nvm</code> — that owns its entire directory tree.</p>
<p>There’s a key issue on the Flutter tracker that captures the whole problem: <a rel="noopener noreferrer" target="_blank" href="https://github.com/flutter/flutter/issues/118162">#118162</a>, titled “Immutable and Offline Flutter SDK mode”. The filing states it plainly — Flutter requires both write access to itself and internet access during initialisation. This makes it impossible to ship as a system package in RPM, DEB, or Flatpak format. The end user doesn’t have write access to <code>/usr/lib/flutter</code>, and a Flatpak SDK extension doesn’t get internet during builds.</p>
<p>The issue was closed as a duplicate. As far as I can tell, the underlying problem hasn’t been addressed.</p>
<h2 id="what-the-nix-community-has-done-about-it">What the Nix community has done about it</h2>
<p>There’s a whole ecosystem of workarounds at this point. The nixpkgs Flutter package creates a <code>flutter-wrapped</code> symlink forest around the read-only store path. Developers write patch scripts that intercept tools like <code>codesign</code> and <code>rsync</code> to make files writable before the real binaries run. Others patch Gradle build files, symlink <code>.git</code> directories from wrapped to unwrapped SDK paths, and various other creative horrors.</p>
<p>Manuel Plavsic documented a comprehensive set of steps in <a rel="noopener noreferrer" target="_blank" href="https://manuelplavsic.ch/articles/flutter-environment-with-nix/">his guide</a>. The NixOS Discourse has threads going back years of people hitting the same wall. Every workaround exists for the same reason: Flutter doesn’t separate its distribution from its build state.</p>
<h2 id="what-i-ended-up-doing">What I ended up doing</h2>
<p>I wrote an <code>apply-ios-fixes</code> script for the flake. It installs wrapper scripts for <code>codesign</code> and <code>rsync</code> that <code>chmod</code> files writable before the real tools touch them, and it patches <code>project.pbxproj</code> to prepend a scripts directory to <code>PATH</code> in the <code>xcode_backend.sh</code> build phases:</p>
<pre data-lang="nix" class="language-nix "><code class="language-nix" data-lang="nix">apply-ios-fixes =
pkgs.writeShellScriptBin "apply-ios-fixes" ''
set -euo pipefail
root="$(${pkgs.git}/bin/git rev-parse --show-toplevel 2>/dev/null || pwd)"
nix_scripts="$root/nix/ios-scripts"
ios_scripts="$root/ios/scripts"
pbxproj="$root/ios/Runner.xcodeproj/project.pbxproj"
if [ ! -f "$pbxproj" ]; then
echo "error: project.pbxproj not found" >&2
exit 1
fi
# Install wrapper scripts
mkdir -p "$ios_scripts"
cp "$nix_scripts/codesign" "$ios_scripts/codesign"
cp "$nix_scripts/rsync" "$ios_scripts/rsync"
chmod +x "$ios_scripts/codesign" "$ios_scripts/rsync"
echo "installed ios/scripts/{codesign,rsync}"
# Patch project.pbxproj – prepend PATH to
# xcode_backend.sh build phases
if grep -q 'PROJECT_DIR}/scripts' "$pbxproj"; then
echo "project.pbxproj already patched"
else
${pkgs.gnused}/bin/sed -i \
'/xcode_backend\.sh\\" embed_and_thin/s|shellScript = "/bin/sh|shellScript = "export PATH=\\"''${PROJECT_DIR}/scripts:''${PATH}\\"\\n/bin/sh|' \
"$pbxproj"
${pkgs.gnused}/bin/sed -i \
'/xcode_backend\.sh\\" build"/s|shellScript = "/bin/sh|shellScript = "# Nix codesign fix: wrapper makes files writable before signing\\nexport PATH=\\"''${PROJECT_DIR}/scripts:''${PATH}\\"\\n/bin/sh|' \
"$pbxproj"
echo "patched project.pbxproj"
fi
echo "done – iOS Nix fixes applied"
'';
</code></pre>
<p>It’s impure. It’s the least of all evils. It gets the job done.</p>
<h2 id="the-takeaway">The takeaway</h2>
<p>The root cause here isn’t technical complexity — it’s a design philosophy. Flutter treats its SDK directory as a mutable workspace. The SDK is simultaneously the distribution, the build cache, the artifact store, and the update mechanism. Everything co-located, everything writable.</p>
<p>Compare this to how other toolchains handle it. Rust’s <code>cargo</code> never writes back to the toolchain directory. Go’s <code>GOROOT</code> is read-only by design — <code>GOPATH</code> and the module cache live elsewhere. Even Node.js keeps <code>node_modules</code> in the project tree, not inside the runtime installation. The pattern of separating your distribution from your build state is well-established. Flutter just doesn’t follow it.</p>
<p>I’d call this horrendous citizenship on the internet. The “SDK is the world” mentality — where one tool absorbs dependency management, version control, artifact caching, and the build itself into a single mutable blob — is convenient for getting started. <code>flutter run</code> works on your laptop, in your home directory, with internet access. But it means your SDK can’t participate in Nix, can’t be distributed as a system package, can’t run in air-gapped CI, and can’t be shared read-only across users on a build server.</p>
<p>More focus should be put into doing things right, not just what’s easy. Flutter, with all of Google’s resources behind it, has no excuse for not having figured this out by now.</p>
Lead the LLM, Don't Let It Lead You2026-03-07T18:00:00+00:002026-03-07T18:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/lead-the-llm/<p>There is a misconception taking hold among developers using LLMs: that the code the model produces is the artifact worth keeping. It is not. The code is disposable. The prompt is the source.</p>
<p>The correct workflow is not prompt, accept, build on top. It is:</p>
<pre><code>prompt
→ evaluate
→ reset to HEAD
→ revise prompt
→ evaluate
→ repeat
</code></pre>
<p>Every time you accept generated code and start layering on top of it, you are accumulating debt against a foundation you did not write, do not fully understand, and cannot efficiently modify. You have traded authorship for speed and lost both.</p>
<h2 id="why-the-code-has-no-value">Why the code has no value</h2>
<p>An LLM will produce working code on the first try often enough to be dangerous. The problem is that “working” is the lowest bar. The generated code might be correct but:</p>
<ul>
<li>Structured in a way that fights the rest of your codebase</li>
<li>Full of unnecessary abstractions “just in case”</li>
<li>Using patterns the model was trained on rather than patterns that fit your problem</li>
<li>Subtly wrong in ways that only surface later</li>
</ul>
<p>None of this matters if you treat the code as a disposable proof of concept. All of it matters if you commit it and move on.</p>
<h2 id="the-prompt-is-the-artifact">The prompt is the artifact</h2>
<p>When you reset to HEAD and revise your prompt instead of patching the output, you are doing something that looks wasteful but is actually efficient. You are:</p>
<ol>
<li>
<p><strong>Keeping your specification clean.</strong> The prompt is a declarative description of what you want. The code is one possible implementation. When you fix the code directly, your specification and implementation diverge immediately.</p>
</li>
<li>
<p><strong>Getting a fresh generation every time.</strong> LLMs do not carry the baggage of their previous attempts unless you let them. A revised prompt produces revised code — not a patch on top of a patch.</p>
</li>
<li>
<p><strong>Staying in control.</strong> You are editing a document you wrote (the prompt) instead of editing a document the machine wrote (the code). One of these you understand completely. The other you are guessing at.</p>
</li>
</ol>
<h2 id="when-to-stop-iterating">When to stop iterating</h2>
<p>You stop when the generated code meets your standards on a clean read. When you can look at the output, understand every line, and judge it as something you would have written yourself given enough time. That is the commit point — not before.</p>
<p>This means the prompt has to get specific. But not too specific.</p>
<h2 id="the-prompt-balance">The prompt balance</h2>
<p>Vague prompts produce vague code. That much is obvious. But the instinct to fix this by writing maximally detailed prompts creates its own problems.</p>
<p>An overly concise prompt leaves too much to the model’s discretion and you get output shaped by its training data rather than your intent. But an overly elaborate prompt introduces a subtler risk: you start encoding your own assumptions as constraints, and some of those assumptions are wrong. This is human tech debt — knowledge that is outdated, incomplete, or simply incorrect — leaking into the specification. The model would have made a better choice if you had not told it otherwise.</p>
<p>Worse, long and detailed prompts tend to accumulate conflicting reasoning. You specify one thing in paragraph two and contradict it in paragraph six. You may not notice, but the model will be pulled in both directions. LLMs are not deterministic to begin with — the same prompt will always produce some variance in output. But a prompt full of internal contradictions amplifies this dramatically. Instead of reasonable variation, you get wildly divergent outputs from the same input. The generation becomes unpredictable in ways that make evaluation harder and iteration slower.</p>
<p>The sweet spot is a prompt that is precise about what matters and silent about what does not. State the constraints that actually constrain. Describe the behaviour you need. Leave the implementation decisions you do not care about to the model — it may know more current patterns than you do. Each iteration should make the prompt more precise, not longer. If your prompt is growing but your output quality is not improving, you are adding noise, not signal.</p>
<h2 id="scaling-up-the-two-phase-approach">Scaling up: the two-phase approach</h2>
<p>Everything above works well for small, self-contained projects — a CLI tool, a single module, a script. But once a project has real scope, ad hoc prompts stop scaling. You need to formalise them.</p>
<p>The approach is two phases. In the first phase, you are not writing code at all. You are iterating on documents: a <code>SPEC.md</code> that defines what the system does, and an <code>ARCHITECTURE.md</code> that defines how it is structured. You use the same reset-and-revise loop, but what you are evaluating is the documents themselves, not code. You iterate until these documents are precise, consistent, and complete enough to serve as the source of truth for the entire project.</p>
<p>In the second phase, when you generate code, the LLM works under these documents. Every prompt references them. Every evaluation checks the output against them. The documents are the constitution; the code is legislation that must comply with it.</p>
<p>These documents will evolve as new features are added — they are not frozen after the first phase. But they must always be held to the highest standard, because they are what keep the LLM from steering off course. Without them, each generation drifts further from your intent. With them, you have guardrails that compound in value over time.</p>
<p>This only works if you protect the documents. When you iterate on features or changes, the LLM will sometimes want to modify the spec or the architecture to accommodate its implementation. This is where you need to be most critical. Changes to these documents should either be made without an LLM — by you, deliberately — or subjected to a much higher degree of scrutiny than changes to code. A bad line of code the LLM can rewrite by refining a prompt. A bad line in your spec is a policy error that propagates into every future generation.</p>
<p>The spec drifts, the project drifts. Hold the model accountable to the documents, not the other way around.</p>
<h2 id="the-implication">The implication</h2>
<p>Whether it is a one-off prompt or a <code>SPEC.md</code> that governs an entire project, the artifact that matters is the one you wrote — not the one the model generated. The skill is not “using an LLM.” It is writing precise specifications under constraints. That is the same skill it has always been. The tool changed. The job did not.</p>
Hello, World2026-03-07T10:00:00+00:002026-03-07T10:00:00+00:00
Ian Johannesen
https://perlpimp.net/blog/hello-world/<p>Welcome to perlpimp.net. This site is built with <a rel="noopener noreferrer" target="_blank" href="https://www.getzola.org/">Zola</a>, packaged as a Nix flake, and served by nginx on NixOS.</p>
<p>More to come.</p>