Andrew O'Brien (Sep 19 2024 at 13:20): Andrew O'Brien (Sep 19 2024 at 13:20):Justin mentioned trying Kubevirt out (possibly with frustration) recently. I'd love to hear more about that experience (or anyone else's). Is it generally hard to set up properly? And if so, where are the sharp edges?
The possibility of managing VMs as resources is pretty appealing, but if libvirt is involved I can see how it might be challenging...
Justin Garrison (Sep 25 2024 at 15:23):Imagine if VMware was built in the age of microservices or if OpenStack had Kubernetes available when it was being developed. It removes some of the "old school" set up a linux server this way stuff and makes everything a container/controller with CRDs and etcd.
Justin Garrison (Sep 25 2024 at 15:24):I got the whole thing working at home. Tested everything. Tore it down. Deployed it again (still working) then packed everything up and went to London. I got there and none of it worked. I spent 2 hours re-deploying it and got a partial working system (3 VMs deployed) but when I came back to it to scale it back up it wouldn't deploy any more VMs.
Justin Garrison (Sep 25 2024 at 15:25):The whole project was built by red hat, for red hat (it's part of OpenShift) and feels very fragile if you're outside of that ecosystem. I know other tools build on it too (eg Rancher Harvester) but at this point I wouldn't use the "raw manifests" it provides to deploy and manage it. There's too many moving parts
Justin Garrison (Sep 25 2024 at 15:26):FWIW I used https://github.com/ContainerCraft/Kargo to deploy it which worked but I still ran into problems multiple times and I had Kat (the lead dev of Kargo) on a call with me to help me work through it
Andrew O'Brien (Sep 25 2024 at 17:04):I'll check out Kargo.
My basis for comparison with Kubevirt is a home grown (non-K8s) server that calls out to libvirt, iptables, and Ansible on the node, so the bar isn't terribly high for me personally. :sweat_smile: But the current implementation is thought to be "simple" (even if the implementation is very gnarly), so good to know that it be on the more challenging end.
Justin Garrison (Sep 25 2024 at 17:32):I will also say that my coworker deployed kubevirt and had it working in just a couple hours. I looked at alternatives like kata, edera, firecracker, etc and all of them don't give a full VM experience (it's more for isolation). I was actually hoping proxmox would be possible to run on top of k8s because I think that would be simpler than openstack but more features than kubevirt
Andrew O'Brien (Sep 25 2024 at 18:03):Have you looked at Incus? I just learned about it yesterday (via Bluefin DX, BTW so thanks for that). First impression makes it seem like it’s a “more modern” take on libvirt with clustering and an HTTP API already included. I wonder if it would be feasible/a good idea to build an operator that interacts with that so that one could get declaratively defined full VMs…
(Of course this is based on less than 24 hours of experience with it.)
Justin Garrison (Oct 07 2024 at 01:46):Incus is a fork of lxc so everything you could do with lxc you can do with incus. One of the benefits is it appears to be better packaged for enterprise Linux distros
Andrew O'Brien (Oct 07 2024 at 14:27):I watched the Kubevirt on Talos talk and am terrified that there would be so much more to both using Kubevirt on K8s or trying to get Incus integrated myself than I’m competent to do. :sweat_smile:
Especially storage management. That looks hard.
Last updated: Dec 12 2024 at 15:40 UTC