Impermanent NixOS: on a VM + tmpfs root + flakes + LUKS

2024-01-29T00:00:00+00:00

Ever since reading Graham Christensen's blog post, Erase your darlings</a>, I've been intrigued by the idea of opt-in state persistence. The concept has become known as impermanence 1</a> 2</a>, but I like to think of it as:

I say we take off and nuke the entire site from orbit. It’s the only way to be sure.</a> </blockquote> 
Check out the video I made</a> introducing this blog post. It might be a good starting point to add some context of what you're getting yourself into.
So what is impermanence? To quote the nix-community impermanence readme</a>:

Lets you choose what files and directories you want to keep between reboots - the rest are thrown away.
Why would you want this?

It keeps your system clean by default.</li>
It forces you to declare settings you want to keep.</li>
It lets you experiment with new software without cluttering up your system.</li> </ul> </blockquote>
The other day I couldn't log in to the graphical environment on my system. I had to switch to a TTY to find out why using journalctl</code>:
systemd-xdg-autostart-generator[2375]: Exec binary 'teams' does not exist: No such file or directory systemd-xdg-autostart-generator[2375]: /home/will/.config/autostart/teams.desktop: not generating unit, error parsing Exec= line: No such file or directory</code></pre>
A ~/.config/autostart/teams.desktop</code> was left over after uninstalling teams3</a>. Crap like this trying to bootstrap autostart on its own is exactly the sort of thing I want to nuke from orbit. So I'm finally going to give impermanence a try.
The plan</a> </h1> Create a custom NixOS live ISO</li> Fire up the live ISO in a VM</li> assume UEFI system (no instructions for legacy BIOS)</li> tmpfs as root</li> Partition with optimal alignment</li> Optional LUKS encryption on root</li> EXT4 for persistence 4</a></li> With swap 5</a> optionally encrypted with random key</li> An opinionated install using nix flakes</a> 6</a></li> </ul> This post is going to be a walk-through of how to try out impermanence with NixOS in a VM. I have also tested this on a Framework laptop with a NVMe drive. I have run through the steps many times in a VM to make sure they work and are easy to copy, paste, and run. I encourage reviewing the code before running. </div> </blockquote> I imagine the audience for this post is people who are already familiar with NixOS, but I'll try to keep in mind those crazy enough to try NixOS, flakes, and impermanence for the first time. I personally try this sort of thing in a VM first. For example, when migrating to NixOS. And again, when I rewrote my config for nix flakes. This time, as I'm migrating to impermanence, I want to document the process better. Hopefully, this will make it easier for others to try it out too. I'm largely using https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/</a> as a guide to use tmpfs</code> (i.e. RAM) as root. I find the simplicity of its setup appealing. However, power outages create a risk of data loss. I use a UPS with my system, and want to give this a try before trying the ZFS / Btrfs snapshot approach. Creating a custom NixOS live ISO</a> </h1> It's nice to be able to copy and paste between the host and the VM, and the following code blocks are written to make them easy to paste and run. Fortunately, it's easy to create a custom live CD/ISO</a>: If that's something you rather not mess with, then skip this section. iso.nix</code>: { modulesPath, pkgs, ... }: { imports = [ "${modulesPath}/installer/cd-dvd/installation-cd-graphical-plasma5.nix" ]; # Enables copy / paste when running in a KVM with spice. services.spice-vdagentd.enable = true; # Use faster squashfs compression isoImage.squashfsCompression = "gzip -Xcompression-level 1"; }</code></pre> ChangingsquashfsCompression</code> is not required, but it speeds up the build dramatically for a modest trade-off in space. </div> </blockquote> Then build the iso.nix</code> with: nix-build '<nixpkgs/nixos>' \ -A config.system.build.isoImage \ -I nixos-config=iso.nix</code></pre> The resulting ISO will be in ./result/iso/</code>. If you don't have Nix installed, I recommend using the Determinate Systems Nix installer</a> (reasons given on the website): </div> </blockquote> In iso.nix</code> I use the graphical installer because X11 is needed for spice to work. Check out the imported file</a> to get a sense of how the live CD is configured. I also made a flake based example on GitHub</a> which, in addition to the above, also brings in home-manager</a> to add some nice-to-haves such as: ZSH configured with completion and auto suggestions </li> Alacritty</a> with starship prompt</a> </li> fzf</a> with ZSH integration (Ctrl + R etc.) </li> neovim</a> with a minimal config for my custom keybindings made for Colemak-DH</a>. I'm actually an Emacs user (with evil-mode), but that's overkill for this situation. </li> xclip for clipboard support in neovim, and other packages I could have installed on the fly with nix-env. </li> </ul> Feel free to fork it and modify it to your liking. KVM management tool</a> </h1> I'm using virt-manager</a> as the front-end for libvirt. If you're on NixOS, see: https://nixos.wiki/wiki/Virt-manager</a> for install instructions. I also experimented with https://cockpit-project.org/</a>, but ran into problems. These days, most computers use UEFI instead of BIOS for the firmware. I'm going to assume no one needs BIOS to simplify these instructions. For me, virt-manager uses BIOS by default, to change this: Open virt-manager ⇒ View ⇒ set x86 Firmware to UEFI ⇒ Close</li> </ol> If you don't want to change the default, you can change it at VM creation time after selecting the disk size: Check the Customize configuration before install</code> ⇒ Finish. </li> In the Overview section, change the Firmware from BIOS to UEFI ⇒ Apply </li> </ol> Fire up the VM</a> </h1> Using the ISO from the section about custom ISOs, or an official one</a>, sudo mv</code> it into /var/lib/libvirt/images/</code> which is the default directory it looks for images. I always end up having permission issues otherwise. Open virt-manager ⇒ Select File ⇒ New virtual machine ⇒ Forward. </li> Choose ISO ⇒ Forward (it should auto-detect that it's NixOS). </li> Choose Memory and CPU amount (I personally use 25% - 50% of what I have available) ⇒ Forward. </li> Choose available disk size ⇒ Forward. </li> Begin Installation </li> </ol> Once booted into the ISO, the first thing I do is right-click on the desktop and Configure Display Settings</code> to change to a higher resolution. Partitioning</a> </h1> For me, lsblk</code> shows a disk named vda</code>. Replace vda</code> with your disk below. Set up a variable for the disk which will be used in the following commands: export DISK=/dev/vda</code></pre> I'm going to use 1MiB</code> (sector 2048) as the starting sector for optimal alignment, and that will probably work for you too. However, if you run into alignment issues, then see the "Finding optimal alignment" section at the end of the blog post. </div> </blockquote> sudo parted $DISK --script \ unit MiB \ mklabel gpt \ mkpart ESP fat32 1 513 \ set 1 boot on \ mkpart swap linux-swap 513 8705 \ mkpart nix 8705 100% \ print</code></pre> Here are the annotated parted</code> commands from above: # default units for `print` and `mkpart` commands unit MiB # initialize the partition table mklabel gpt # Make boot partition 512 MiB. # Note that 1MiB is where my partition needs to start for optimal alignment. mkpart ESP fat32 1 513 # mark the partition is bootable set 1 boot on # Make swap partition 8705 = 513 + 8192 (note 8192MiB = 8GiB) mkpart swap linux-swap 513 8705 # Make root partition the rest of the drive mkpart nix 8705 100% # see the results print</code></pre> The lsblk</code> command should now show the partitions on the disk: $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 3.1G 1 loop /nix/.ro-store sr0 11:0 1 3.2G 0 rom /iso vda 253:0 0 40G 0 disk ├─vda1 253:1 0 512M 0 part ├─vda2 253:2 0 8G 0 part └─vda3 253:3 0 31.5G 0 part</code></pre> Check alignment: for i in {1..3}; do sudo parted $DISK -- align-check optimal $i; done</code></pre>Format</a> </h1> The following will set variables PART1</code>, PART2</code>, and PART3</code> to the partition names. I realized when trying this out on a laptop with a NVMe drive that simply using ${DISK}1</code> wouldn't work when the disk ends with a number such as nvme0n1</code>, the partitions end up looking like nvme0n1p1</code> etc. for i in {1..3};\ do export "PART$i"=$(lsblk -lp | grep part | grep ${DISK} | awk -v line=$i 'NR==line{print $1}');\ done;\ echo $PART1; echo $PART2; echo $PART3</code></pre> The output should look similar to: /dev/vda1 /dev/vda2 /dev/vda3</code></pre> Optionally, LUKS encrypt the root partition: sudo cryptsetup luksFormat $PART3</code></pre> Open the encrypted partition and change the variable to point to the decrypted partition: sudo cryptsetup luksOpen $PART3 crypted && \ export PART3=/dev/mapper/crypted</code></pre> Now format the partitions: sudo mkfs.fat -F 32 -n boot ${PART1} && \ sudo mkswap -L swap ${PART2} && \ sudo mkfs.ext4 -L nixos ${PART3}</code></pre> I always ignore the warning: lowercase labels might not work properly on some systems </blockquote> Visually verify the partitions and file systems using: sudo parted $DISK -- unit MiB print</code></pre> Here's what I get with a 40GiB virtual disk: Model: Virtio Block Device (virtblk) Disk /dev/vda: 40960MiB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1.00MiB 513MiB 512MiB fat32 ESP boot, esp 2 513MiB 8705MiB 8192MiB linux-swap(v1) swap swap 3 8705MiB 40959MiB 32254MiB ext4 nix</code></pre>Mount</a> </h1> I'm going to base the following on the excellent https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/</a> guide. cat << 'EOF' > ~/mount.sh #!/usr/bin/env bash set -e # Mount your root file system as tmpfs mount -v -t tmpfs none /mnt # Create mount directories mkdir -v -p /mnt/{boot,nix,etc/nixos,var/log} # Mount /boot and /nix mount -v $PART1 /mnt/boot -o umask=0077 mount -v $PART3 /mnt/nix # Create persistent directories mkdir -v -p /mnt/nix/persist/{etc/nixos,var/log} # Bind mount the persistent configuration / logs mount -v -o bind /mnt/nix/persist/etc/nixos /mnt/etc/nixos mount -v -o bind /mnt/nix/persist/var/log /mnt/var/log # Make config directory temporarily easier to work with chmod -v 777 /mnt/etc/nixos EOF</code></pre>chmod u+x ~/mount.sh && sudo -E ~/mount.sh</code></pre>Configure hardware</a> </h1> Enable swap now. Otherwise, it won't get automatically configured in your hardware-configuration.nix</code>: sudo swapon $PART2</code></pre> Generate the initial configuration.nix</code> and hardware-configuration.nix</code>: nixos-generate-config --root /mnt && cd /mnt/etc/nixos</code></pre> If you see the following, just ignore it: ERROR: Not a Btrfs filesystem: Invalid argument </blockquote> Per Elis's blog post, we need to set some options on the tmpfs root in the hardware-configuration.nix</code>: The most important bit is the mode</code>, otherwise certain software (such as openssh) won't be happy with the permissions of the file system. The size</code> is something you can adjust depending on how much garbage you are willing to store in ram until you run out of space on your root... </blockquote> { #... fileSystems."/" = { device = "none"; fsType = "tmpfs"; + options = [ "defaults" "size=25%" "mode=755" ]; }; #... }</code></pre> The following will add the options to the tmpfs root and format the two nix files using nixpkgs-fmt</code>: sed -i '/fsType = "tmpfs";/a options = [ "defaults" "size=25%" "mode=755" ];' \ ./hardware-configuration.nix && \ nix-shell -p nixpkgs-fmt --run 'nixpkgs-fmt .'</code></pre>Increase security of the boot mount</a> </h2> You may have noticed when I mounted boot</code> I used umask=0077</code>. This was to avoid the following warning: ⚠️ Mount point '/boot' which backs the random seed file is world accessible, which is a security hole! ⚠️ ⚠️ Random seed file '/boot/loader/.#bootctlrandom-seedc878009c19a876dc' is world accessible, which is a security hole! ⚠️</code></pre> There's a nixpkgs issue</a> about it. The generated hardware-configuration.nix</code> does not currently pick up these permission options. So let's fix that again using sed</code>. sed -i '/fsType = "vfat"/a options = [ "umask=0077" ];' \ ./hardware-configuration.nix && \ nix-shell -p nixpkgs-fmt --run 'nixpkgs-fmt .'</code></pre>Optional encrypted swap</a> </h2> If you're using LUKS on the root partition, then you might want to encrypt swap</a>. However, the by-uuid</code> generated for the swapDevice</code> in the hardware-configuration</code> needs to change to the by-partuuid</code> because when using the randomEncryption.enable = true</code> the by-uuid</code> changes every boot. To keep with the theme of the post, I threw together a script to do the work for you: cat << 'EOF' > ~/encrypt-swap.sh #!/usr/bin/env bash set -e if [[ -z $PART2 ]]; then echo "PART2 is undefined or empty" exit 1 fi hwConfig=/mnt/etc/nixos/hardware-configuration.nix backupHwConfig=/mnt/etc/nixos/hardware-configuration.backup.nix main() { swapPart=$(echo $PART2 | awk -F'/' '{print $NF}') swapDiskUUID=$(ls -l /dev/disk/by-uuid | grep $swapPart | awk '{print $9}') swapPartUUID=$(ls -l /dev/disk/by-partuuid | grep $swapPart | awk '{print $9}') echo "swapDiskUUID: $swapDiskUUID" echo "swapPartUUID: $swapPartUUID" sed -i "s|by-uuid/$swapDiskUUID|by-partuuid/$swapPartUUID|g" $hwConfig sed -i "/$swapPartUUID/s/\";/\";\n/" $hwConfig sed -i "/$swapPartUUID\"/ a\\randomEncryption.enable = true;" $hwConfig nix-shell -p nixpkgs-fmt --run "nixpkgs-fmt $hwConfig" } cp $hwConfig $backupHwConfig trap 'cp $backupHwConfig $hwConfig' ERR main EOF</code></pre>chmod u+x ~/encrypt-swap.sh && ~/encrypt-swap.sh</code></pre>Configure with flakes</a> </h1> If you're not familiar with nix flakes, I recommend reading at least the first link below: https://nix.dev/concepts/flakes.html</a></li> https://www.tweag.io/blog/2020-05-25-flakes/</a></li> https://nixos.wiki/wiki/Flakes</a></li> </ul> We're getting to the rest of the owl</a> section that seems inevitable with configuring NixOS. To help out, I made an example starter config</a> that I configured using Misterio77/nix-starter-configs</a> minimal template. So if you're starting out with flakes, then I recommend checking out both. I'll talk about this more in the last section of the blog post. Here's how to use my starter config: git clone https://github.com/willbush/ex-nixos-starter-config.git && \ mv hardware-configuration.nix ./ex-nixos-starter-config && \ cd ex-nixos-starter-config && \ git add .</code></pre> New files in a git repo must be staged at least for flake related commands to be aware of them. This is why git add .</code> is needed so that the hardware-configuration.nix</code> is staged. Otherwise, an error like the following will happen: error: getting status of '/mnt/nix/store/21zpkqcn55a73x9y8yy4lrrd7ja3mjvc-source/nixos/hardware-configuration.nix': No such file or directory </div> </blockquote> Install</a> </h1> Before installing, I recommend looking over the generated /mnt/etc/nixos/configuration.nix</code>. It's not going to be used in the installation, so copy any settings you want into the /mnt/etc/nixos/ex-nixos-starter-config/configuration.nix</code> file. If you like, rename blitzar</a> to whatever hostname you like. I usually name my hosts after astronomical bodies or concepts for fun. From the /mnt/etc/nixos/ex-nixos-starter-config</code> directory run: NIX_CONFIG="experimental-features = nix-command flakes" \ sudo nixos-install --flake .#blitzar --no-root-passwd</code></pre> Change /mnt/etc/nixos</code> permissions back and reboot: sudo chmod -v 755 /mnt/etc/nixos && \ reboot</code></pre> Since we're using Nix flakes, there's no requirement for the repo to be under /etc/nixos</code>. Feel free to change it after rebooting, but make sure you persist the directory you decide to store it! </div> </blockquote> If you're using my example starter config, then see its readme</a> for credentials and how to change them. Rest of the owl</a> </h1> One important difference</a> I made when configuring from the minimal Misterio77/nix-starter-configs</a> template is I switched home manager</a> to a module</a> within a NixOS system configuration. This means all the home.nix</code> configuration is applied when nixos-install</code> is run (above) or in the more normal case when using nixos-rebuild</code>. I rather not have to use the home-manager</code> standalone CLI tool. I suppose one might have a use-case where they want to configure their home independently of their system. I'm not sure what the best approach is to avoid having to run the tool on every boot. Finding what to persist</a> </h2> Hi, it's me from the future. I've been using this setup for a couple of weeks. One thing that's helpful is to be able to find differences between the /nix/persist</code> and /</code> directories. I use rsync</code> with --dry-run</code> to do this: sudo rsync -amvxx \ --dry-run \ --no-links \ --exclude '/tmp/*' \ --exclude '/root/*' \ / /nix/persist/ \ | rg -v '^skipping|/$'</code></pre> Here is some documentation on the options used: --no-OPTION turn off an implied OPTION (e.g. --no-D) -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) -c, --checksum skip based on checksum, not mod-time & size -m, --prune-empty-dirs prune empty directory chains from file-list -n, --dry-run perform a trial run with no changes made -v, --verbose increase verbosity -x, --one-file-system don't cross filesystem boundaries If this option is repeated, rsync omits all mount-point directories from the copy. Otherwise, it includes an empty directory at each mount-point it encounters (using the attributes of the mounted directory because those of the underlying mount-point directory are inaccessible).</code></pre> However, note I'm not using -c, --checksum</code> because it's slower and seems to be overkill. I exclude /tmp</code> and /root</code> because I don't want to persist those directories. I use rg</code> (ripgrep</a>) to filter out results. Such as lines starting with skipping</code>: skipping non-regular file "home/will/.local/state/home-manager/gcroots/current-home" skipping non-regular file "home/will/.local/state/nix/profiles/home-manager" ...</code></pre> or ending with /</code>: home/will/.config/hypr/ home/will/.config/mako/ home/will/.config/nvim/ home/will/.config/swaylock/ home/will/.config/systemd/ ...</code></pre> You can also reverse the source / destination to find orphaned files in the /nix/persist</code> directory. Documentation</a> </h2> Of course, the real "rest of the owl" is actually configuring your system, and there's no way around reading documentation. Here are a few links to get you started: https://nixos.org/learn</a></li> https://nix.dev/</a></li> https://zero-to-nix.com/</a></li> </ul> The links above have a lot of jumping off points to other resources. Let me know if I left anything out. The nix-community impermanence readme</a> is a must-read unless you decided to not use: # flake.nix { inputs = { # ... impermanence.url = "github:nix-community/impermanence"; }; # ... }</code></pre>Finding optimal alignment</a> </h2> When using parted</code>, a simple way around alignment issues is to use percentages or MB / GB units. When this is done, parted</code> effectively searches a calculated distance around the given location to find optimal alignment. However, I like to have my partitions in nice clean power of 2 IEC unit sizes (e.g. MiB) because that's what most tools default to (e.g. gparted</code>, cfdisk</code>, lsblk</code>, df -h</code> etc.). The problem is when using IEC units or exact sectors, parted</code> will not search for optimal alignment. It basically assumes you know what you're doing. 7</a> 8</a> 9</a> What we need to find is the starting sector of the first partition. A simple trick to do this with parted</code> is to: sudo parted $DISK --script \ mklabel gpt \ mkpart primary 0% 100% \ unit MiB print \ rm 1</code></pre> This will print out something like the following. Note the Start</code>. Model: Virtio Block Device (virtblk) Disk /dev/vda: 40960MiB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1.00MiB 40959MiB 40958MiB primary</code></pre> Now wipe the device to start over: sudo wipefs -a $DISK</code></pre> Head back to the "Partitioning" section and adjust the values for your Start</code>. Harmless error on shutdown</a> </h2> When shutting down or rebooting, you might see a flash of red in the logs: [FAILED] Failed unmounting /nix.</code></pre> Seems to be related to this issue</a>. Fortunately, it seems to be harmless and doesn't cause any delay. If you want to see the error more clearly, try sudo halt</code>. ^1Nixos Wiki Impermanence</a> </div> ^2Nix Community Impermanence</a> </div> ^{3 Why this breaks the graphical environment I still don't know. Perhaps I need to disable xdg.autostart.enable</code> which defaults to true? </div>}^{4 I spent way too much time considering whether I wanted to switch away from EXT4. After considering the features and performance of file systems such as: </div>} Btrfs</li> ZFS</li> f2fs</li> XFS</li> bcachefs</li> </ul> I decided to stick with EXT4 for my workstation. Though I plan to revisit this topic again in the future. ^5This is always a controversial topic. And I can't blame either position because I have yet to find any argument with experiments to back up claims. Here's an argument in favor of swap</a> which I keep running across. It should be pretty straight forward to remove swap from the script snippets above. </div> ^{6 This blog post is already taking too long. Making it "opinionated" makes it easier to write. </div>}^7Archlinux Parted Alignment</a> </div> ^8HQ Code Shop Blog Post on GNU Parted</a> </div> ^9Unix StackExchange Question on Parted Alignment</a> </div>

Format

Other appliance anecdotes
Discussion </a> </h1>

https://news.ycombinator.com/item?id=39238010</a></li>
https://slrpnk.net/post/4915674</a></li> </ul>

Discussion </a> </h1>

https://news.ycombinator.com/item?id=39238010</a></li>
https://slrpnk.net/post/4915674</a></li> </ul>

Run upgrade-packages.sh locally </a> </h1>
^1</sup>
action-inputs</a></p> </div>

Not caring until it's gone
3-2-1 Backup Rule</a>, did I realize I need to up my game.</p>

Cleanup

willbush::blog

Impermanent NixOS: on a VM + tmpfs root + flakes + LUKS

Goodbye Non-KISS Appliances

Monorepo NuGet Package Updates with GitHub Actions

Synology NAS Backups with Minimal Bus Factor

Fast OCR to clipboard

willbush::blog

Impermanent NixOS: on a VM + tmpfs root + flakes + LUKS

The plan</a> </h1> Create a custom NixOS live ISO</li> Fire up the live ISO in a VM</li> assume UEFI system (no instructions for legacy BIOS)</li> tmpfs as root</li> Partition with optimal alignment</li> Optional LUKS encryption on root</li>

Goodbye Non-KISS Appliances

Monorepo NuGet Package Updates with GitHub Actions

Synology NAS Backups with Minimal Bus Factor

External Hard Drive</a> </h2> On Ebay, in late 2021, I bought a:</p> ~$55 WD My Passport 4TB Certified Refurbished Portable Hard Drive.</li> </ul> Since using the space doesn't incur any additional cost like cloud storage, I backup more than just the "don't want to lose" category.</p>

Synology Diskstation</a> </h2> Model: DS918+</li> Software Version: 7.1.1-42962 Update 4</li> RAM: 16GB</li> CPU: INTEL Celeron J3455</li>

Wasabi</a> </h2> </path></svg> </div>

Fast OCR to clipboard