'Recovered VM: Chroot, Pacman, and Pain

'

src/content/recovering_an_archlinux_qemu_vm_in_proxmox.md

@@ -0,0 +1,72 @@
+# Recovering an Archlinux QEMU VM in Proxmox: A Day in Hell and Back Again
+
+Well that was a morning. Today I wanted to try and fix my Longhorn installation in Kube... (again 😥). It turns out, things didn't go as planned.
+
+## The Unexpected Downfall
+
+I went to perform my usual update and reboot... except today for whatever reason, the upgrade decided to fail to install the kernel and left me with an unbootable system.
+
+### Dropping into Grub Rescue
+
+At this point I dropped back down to grub rescue mode (which is always fun). Honestly? I hate that environment! And then it hit me: these systems are just QEMU disks, right? Surely I can mount them, chroot in, and fix the install.
+
+## The Quest for Recovery
+
+It took 2 hours of frantic Googling through Proxmox and Arch forums until I stumbled upon something... almost magical.
+
+### Mounting QEMU Disks Made Easy
+
+I found an amazing suite of tools to make mounting these qemu disks a breeze. Check out this [guide](https://www.howtogeek.com/devops/how-to-mount-a-qemu-virtual-disk-image/) for all the details on libguestfs-tools and guestmount.
+
+#### Mounting in Action
+
+```bash
+sudo apt install libguestfs-tools
+sudo guestmount --add /var/lib/pve/local-btrfs/images/100/vm-100-disk-0/disk.raw --mount /dev/sda3 /tmp/kube_disk/
+```
+
+### Enter Chroot Land
+
+Now that I've got my disk mounted, it's time to chroot in. But hold up! I need it as root this time.
+
+#### Setting Up Arch-Chroot
+
+```bash
+sudo apt install arch-installation-scripts
+arch-chroot /tmp/kube_disk/
+```
+
+### Pacman: The Hero We Deserve (But Need Permission)
+
+Oh boy, pacman threw 23 million permission errors my way. Last year they changed it to work rootless by default… but I found out you can turn off the `DefaultUser` flag in `/etc/pacman.conf`. Here's how:
+
+```bash
+# Disable DefaultUser temporarily for this session (or remove if permanent)
+pacman -Syu --noprofile --overwrite "*"
+```
+
+### Clean Up and Exit
+
+Once we're done, we need to exit the chroot. Remember that crucial step: umounting correctly.
+
+```bash
+exit
+sudo umount /tmp/kube_disk/
+```
+
+## The Reboot Saga (And How Not To Do It)
+
+Reboot was supposed to be a smooth sail… but I made one fatal mistake.
+
+### Corruption Nightmare
+
+I didn't unmount before starting the VM. This led me down an unfortunate path of corrupting my btrfs partition beyond recognition and having to rebuild not just my master node, but *entire* cluster! Backups saved the day... barely!
+
+#### Lessons Learned
+
+*   **Never** reboot without first properly umounting.
+*   Seriously need more backups for those images. 🚀
+
+## Conclusion: A Cluster-Fucked Day Turned Into a Learning Experience
+
+All in all it was chaos, but hey – learning happens at 2 AM after midnight reboots and frantic Googling. Hope this helps anyone else stuck with Proxmox woes! 🚀