Title: Recovering Archlinux Qemu VM in Proxmox
Date: 2025-07-01 20:00
Modified: 2025-07-01 20:00
Category: SysAdmin
Tags: System Admin, Proxmox, Qemu, Arch, Kubernetes
Slug: recovering-arch-vm-proxmox
Authors: Andrew Ridgway
Summary: An absolute nightmare of a day trying to recover my kube cluster from a silly update error

### Human Edit
This is probably the most amazing run of the blog creator, I've started using the new gemma3n and also upgrade the box ollama runs on so it can run slightly bigger models. Using phi4 and gemma:27b has produced some amazing results see below

I *did* need to update some of the pacman stuff as it conflated to seperate issues so bear in mind I have made some little edits in that place but otherwise... this is straight from the mouth of the llm. Enjoy!

# Recovering an Archlinux QEMU VM in Proxmox: A Day in Hell and Back Again

Well that was a morning. Today I wanted to try and fix my Longhorn installation in Kube... (again 😥). It turns out, things didn't go as planned.

## The Unexpected Downfall

I went to perform my usual update and reboot... except today for whatever reason, the upgrade decided to fail to install the kernel and left me with an unbootable system.

### Dropping into Grub Rescue

At this point I dropped back down to grub rescue mode (which is always fun). Honestly? I hate that environment! And then it hit me: these systems are just QEMU disks, right? Surely I can mount them, chroot in, and fix the install.

## The Quest for Recovery

It took 2 hours of frantic Googling through Proxmox and Arch forums until I stumbled upon something... almost magical.

### Mounting QEMU Disks Made Easy

I found an amazing suite of tools to make mounting these qemu disks a breeze. Check out this [guide](https://www.howtogeek.com/devops/how-to-mount-a-qemu-virtual-disk-image/) for all the details on libguestfs-tools and guestmount.

#### Mounting in Action

```bash
sudo apt install libguestfs-tools
sudo guestmount --add /var/lib/pve/local-btrfs/images/100/vm-100-disk-0/disk.raw --mount /dev/sda3 /tmp/kube_disk/
```

### Enter Chroot Land

Now that I've got my disk mounted, it's time to chroot in. But hold up! I need it as root this time.

#### Setting Up Arch-Chroot

```bash
sudo apt install arch-installation-scripts
arch-chroot /tmp/kube_disk/
```

### Pacman: The Hero We Deserve (But Need Permission)

Oh boy, pacman threw 23 million permission errors my way. Last year they changed it to work rootless by default… but I found out you can turn off the `DefaultUser` flag in `/etc/pacman.conf`. Here's how:

```bash
# Disable DefaultUser temporarily for this session (or remove if permanent)
pacman -Syu 
```

I did have a couple issues installing the kernel (which is what got borked in the update)
```bash
# Sometimes some files got written so use overwrite to get rid of them
# be warned this *could* be destructive
pacman -S linux --overwrite "*"
```

### Clean Up and Exit

Once we're done, we need to exit the chroot. Remember that crucial step: umounting correctly.

```bash
exit
sudo umount /tmp/kube_disk/
```

## The Reboot Saga (And How Not To Do It)

Reboot was supposed to be a smooth sail… but I made one fatal mistake.

### Corruption Nightmare

I didn't unmount before starting the VM. This led me down an unfortunate path of corrupting my btrfs partition beyond recognition and having to rebuild not just my master node, but *entire* cluster! Backups saved the day... barely!

#### Lessons Learned

*   **Never** reboot without first properly umounting.
*   Seriously need more backups for those images. 🚀

## Conclusion: A Cluster-Fucked Day Turned Into a Learning Experience

All in all it was chaos, but hey – learning happens at 2 AM after midnight reboots and frantic Googling. Hope this helps anyone else stuck with Proxmox woes! 🚀