Merge pull request 'recovering_an_archlinux_qemu_vm_in_proxmox' (#15) from recovering_an_archlinux_qemu_vm_in_proxmox into master

Reviewed-on: #15

src/content/recovering_an_archlinux_qemu_vm_in_proxmox.md

@@ -0,0 +1,93 @@
+Title: Recovering Archlinux Qemu VM in Proxmox
+Date: 2025-07-01 20:00
+Modified: 2025-07-01 20:00
+Category: SysAdmin
+Tags: System Admin, Proxmox, Qemu, Arch, Kubernetes
+Slug: recovering-arch-vm-proxmox
+Authors: Andrew Ridgway
+Summary: An absolute nightmare of a day trying to recover my kube cluster from a silly update error
+
+### Human Edit
+This is probably the most amazing run of the blog creator, I've started using the new gemma3n and also upgrade the box ollama runs on so it can run slightly bigger models. Using phi4 and gemma:27b has produced some amazing results see below
+
+I *did* need to update some of the pacman stuff as it conflated to seperate issues so bear in mind I have made some little edits in that place but otherwise... this is straight from the mouth of the llm. Enjoy!
+
+# Recovering an Archlinux QEMU VM in Proxmox: A Day in Hell and Back Again
+
+Well that was a morning. Today I wanted to try and fix my Longhorn installation in Kube... (again 😥). It turns out, things didn't go as planned.
+
+## The Unexpected Downfall
+
+I went to perform my usual update and reboot... except today for whatever reason, the upgrade decided to fail to install the kernel and left me with an unbootable system.
+
+### Dropping into Grub Rescue
+
+At this point I dropped back down to grub rescue mode (which is always fun). Honestly? I hate that environment! And then it hit me: these systems are just QEMU disks, right? Surely I can mount them, chroot in, and fix the install.
+
+## The Quest for Recovery
+
+It took 2 hours of frantic Googling through Proxmox and Arch forums until I stumbled upon something... almost magical.
+
+### Mounting QEMU Disks Made Easy
+
+I found an amazing suite of tools to make mounting these qemu disks a breeze. Check out this [guide](https://www.howtogeek.com/devops/how-to-mount-a-qemu-virtual-disk-image/) for all the details on libguestfs-tools and guestmount.
+
+#### Mounting in Action
+
+```bash
+sudo apt install libguestfs-tools
+sudo guestmount --add /var/lib/pve/local-btrfs/images/100/vm-100-disk-0/disk.raw --mount /dev/sda3 /tmp/kube_disk/
+```
+
+### Enter Chroot Land
+
+Now that I've got my disk mounted, it's time to chroot in. But hold up! I need it as root this time.
+
+#### Setting Up Arch-Chroot
+
+```bash
+sudo apt install arch-installation-scripts
+arch-chroot /tmp/kube_disk/
+```
+
+### Pacman: The Hero We Deserve (But Need Permission)
+
+Oh boy, pacman threw 23 million permission errors my way. Last year they changed it to work rootless by default… but I found out you can turn off the `DefaultUser` flag in `/etc/pacman.conf`. Here's how:
+
+```bash
+# Disable DefaultUser temporarily for this session (or remove if permanent)
+pacman -Syu 
+```
+
+I did have a couple issues installing the kernel (which is what got borked in the update)
+```bash
+# Sometimes some files got written so use overwrite to get rid of them
+# be warned this *could* be destructive
+pacman -S linux --overwrite "*"
+```
+
+### Clean Up and Exit
+
+Once we're done, we need to exit the chroot. Remember that crucial step: umounting correctly.
+
+```bash
+exit
+sudo umount /tmp/kube_disk/
+```
+
+## The Reboot Saga (And How Not To Do It)
+
+Reboot was supposed to be a smooth sail… but I made one fatal mistake.
+
+### Corruption Nightmare
+
+I didn't unmount before starting the VM. This led me down an unfortunate path of corrupting my btrfs partition beyond recognition and having to rebuild not just my master node, but *entire* cluster! Backups saved the day... barely!
+
+#### Lessons Learned
+
+*   **Never** reboot without first properly umounting.
+*   Seriously need more backups for those images. 🚀
+
+## Conclusion: A Cluster-Fucked Day Turned Into a Learning Experience
+
+All in all it was chaos, but hey – learning happens at 2 AM after midnight reboots and frantic Googling. Hope this helps anyone else stuck with Proxmox woes! 🚀