Using Kubevirt to run VMs on kubernetes
By Aravind
Using Kubevirt to run VMs on kubernetes clusters
Kubernetes has been the defacto standard for ochestration container and VM workloads. The APIs have been extended using Kubevirt which is a CNCF project to instantiate VM workloads. This post talks about how to spin up Juniper’s VMs, particularly vSRX on kubevirt. Kubevirt under the hood uses libvirt , so KVM based VMs can be spun up.
most of the documents talks about 0.35 and prior versions. it is important for k8s version and kubevirt versions to match the release times. I believe around 3 releases gap is fine based on my testing in my case i had the below k8s versions and none of the kubevirt would install. So had to try the newer ones. so v0.50.0 worked based on the k8s version i had.
Kubernetes installation
Follow the link here to setup kubernetes cluster. Talks about 3 servers running Ubuntu 20.04 version
root@k8s-master:~/# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:48:05Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
root@k8s-master:~/# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:49:13Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:43:11Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
root@k8s-master:~/# kubelet --version
Kubernetes v1.23.6
Install Kubevirt
You can change the version v0.51.0 to the newer ones accordingly posted on https://github.com/kubevirt/kubevirt/branches/active and https://kubevirt.io/user-guide/latest_release_notes/
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v0.51.0/kubevirt-operator.yaml
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v0.51.0/kubevirt-cr.yaml
Verify pod status
root@k8s-master:~/vsrx-kubevirt# kubectl get pods -n kubevirt
NAME READY STATUS RESTARTS AGE
virt-api-667c9d49c7-btcjd 1/1 Running 1 (8h ago) 9h
virt-api-667c9d49c7-wblj5 1/1 Running 1 (8h ago) 9h
virt-controller-fb8776755-26tld 1/1 Running 1 (8h ago) 9h
virt-controller-fb8776755-lm5sv 1/1 Running 1 (8h ago) 9h
virt-handler-6wjgt 1/1 Running 0 9h
virt-handler-zc8rs 1/1 Running 0 9h
virt-operator-54986cb6df-d6wms 1/1 Running 0 9h
virt-operator-54986cb6df-lrfj9 1/1 Running 0 9h
Install virtctl
Virtctl is used as a tool to upload images which can then be used to spin up teh VMs. The procedure to upload and spin will be covered in the later sections
Download the respective version of virtctl . Use the same as kubevirt. In this case v0.51.0. The latest version can be found here https://github.com/kubevirt/kubevirt/releases
export VERSION=v0.51.0
wget https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/virtctl-${VERSION}-linux-amd64
add respective permissions to the executable
sudo chmod +rx <virtctl file>
cp <file> /usr/local/bin/virtctl
Install libvirt client
This needs to be installed on all the worker nodes where the VM needs to be spun up.
root@k8s-worker1:~# sudo apt install libvirt-clients
In case the VM work loads needs to be spun up on the master node, then the master node has to be tainted. In order to taint, run the below.
kubectl taint nodes --all node-role.kubernetes.io/master-
Install CDI (Containerized data importer)
Containerized data importer is needed in order to upload VM images which can be used to run the VMs.
Find versions here
https://github.com/kubevirt/containerized-data-importer/releases
replace version with values and install
VERSION=$(curl -s https://github.com/kubevirt/containerized-data-importer/releases/latest | grep -o "v[0-9]\.[0-9]*\.[0-9]*")
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml
Verify the pods
root@k8s-master:~/vsrx-kubevirt# kubectl get pods -n cdi -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cdi-apiserver-76ffb454c9-4mjsr 1/1 Running 0 9h 10.244.1.9 k8s-worker2 <none> <none>
cdi-deployment-668465f546-4xc87 1/1 Running 0 9h 10.244.2.7 k8s-worker1 <none> <none>
cdi-operator-64d6dc7f6f-2997n 1/1 Running 0 9h 10.244.1.8 k8s-worker2 <none> <none>
cdi-uploadproxy-664c777f8f-cb2fn 1/1 Running 0 9h 10.244.1.10 k8s-worker2 <none> <none>
Expose node port
Expose node port for the CDI by creating a node port service so that teh CDI proxy is reachable from outside of the cluster
save the file as cdi-service.yaml
root@k8s-master:~/vsrx-kubevirt# more CDI/cdi-service.yaml
apiVersion: v1
kind: Service
metadata:
name: cdi-uploadproxy-nodeport
namespace: cdi
labels:
cdi.kubevirt.io: "cdi-uploadproxy"
spec:
type: NodePort
ports:
- port: 443
targetPort: 443
nodePort: 31001
protocol: TCP
selector:
cdi.kubevirt.io: cdi-uploadproxy
Apply the file
kubectl apply -f cdi-service.yaml
Verify service status
root@k8s-master:~/vsrx-kubevirt# kubectl get svc -n cdi
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cdi-api ClusterIP 10.108.255.248 <none> 443/TCP 9h
cdi-prometheus-metrics ClusterIP 10.98.76.169 <none> 8080/TCP 9h
cdi-uploadproxy ClusterIP 10.96.92.83 <none> 443/TCP 9h
cdi-uploadproxy-nodeport NodePort 10.104.121.237 <none> 443:31001/TCP 8h
Upload the images to CDI
In order the upload the images to CDI, there are few things that are required, depending on how the image needs to be spun up. This talks specifically about persistent disks. This may not be needed if a containerDisk is used.
- datavolume
- storage class
- persistent volume
when the datavolume is created, it automatically creates an upload pod and which triggers a PVC (persistent volume claim) which ultimately gets bound to a PV (persistent volume). it creates a scratch paritition (PV) would also be needed. Once the upload is complete this will be released.
Create Storage class
root@k8s-master:~/vsrx-kubevirt/CDI# more storage_profile.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
storageclass.kubernetes.io/is-default-class: "true"
name: hostpath-provisioner
provisioner: kubernetes.io/hostpath-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
Apply
kubectl apply -f storage_profile.yaml
Verify the storage class
root@k8s-master:~/vsrx-kubevirt# kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
hostpath-provisioner (default) kubernetes.io/hostpath-provisioner Delete WaitForFirstConsumer false 156m
Create persistent volume
Create 2, one for scratch and another for boot disk. The hostPath is a path on the local server. The storageClassName is important because that is the tying in factor between the DV, PV and PVC. so ensure the correct names are used. Otherwise, the pvc, will reamin in pending stage and will never get bound.
root@k8s-master:~/vsrx-kubevirt/CDI# more pv1.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv0001
spec:
storageClassName: hostpath-provisioner
accessModes:
- ReadWriteOnce
capacity:
storage: 25Gi
hostPath:
path: /root/vsrx-kubevirt/pvdata2
root@k8s-master:~/vsrx-kubevirt/CDI# more pv2.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv0002
spec:
storageClassName: hostpath-provisioner
accessModes:
- ReadWriteOnce
capacity:
storage: 25Gi
hostPath:
path: /root/vsrx-kubevirt/pvdata2
Apply the file
kubectl apply -f pv1.yaml
kubectl apply -f pv2.yaml
Verify the PV status
root@k8s-master:~/vsrx-kubevirt/CDI# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv0001 25Gi RWO Retain Released default/vsrx-boot-scratch hostpath-provisioner 12m
pv0002 25Gi RWO Retain Bound default/vsrx-boot hostpath-provisioner 12m
Create DataVolume (DV)
root@k8s-master:~/vsrx-kubevirt/CDI# more dv.yaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
name: vsrx-boot
spec:
source:
upload: {}
pvc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 25Gi
Apply the file
kubectl apply -f dv.yaml
Verify the DV status
root@k8s-master:~/vsrx-kubevirt/CDI# kubectl get dv
NAME PHASE PROGRESS RESTARTS AGE
vsrx-boot Succeeded N/A 3m26s
Once the DV is applied, a pod is kicked off for image upload and will be in waiting, until the PVCs are BOUND. once BOUND, the pod will transition to a running state. Once running, upload the image.
Verify the PVC status
root@k8s-master:~/vsrx-kubevirt/CDI# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
vsrx-boot Bound pv0002 25Gi RWO hostpath-provisioner 12s
vsrx-boot-scratch Bound pv0001 25Gi RWO hostpath-provisioner 12s
Upload the image using virtctl
use the dv which was created. The Uploadproxy IP is the pod IP and 8443 is the port exposed. Ideally the nodePort exposed should be used which is outside the cluster. Since the connection did not succeed, local clusterIP has been used.
The image vsrx3_21.4R2_kubevirt.qcow2 is locally present on the master node.
root@k8s-master:~/vsrx-kubevirt# virtctl image-upload dv vsrx-boot --size=25Gi --image-path=vsrx3_21.4R2_kubevirt.qcow2 --uploadproxy-url https://10.244.1.10:8443 --insecure --storage-class hostpath-provisioner
Once kicked off, below will be the output if no errors are noticed. Errors include ceritificates, PVC unbound and waiting for pod to go into running.
Using existing PVC default/vsrx-boot
Uploading data to https://10.244.1.10:8443
746.81 MiB / 746.81 MiB [=================================================================================================] 100.00% 6s
Uploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress
Processing completed successfully
Uploading vsrx3_21.4R2_kubevirt.qcow2 completed successfully
At this time, the upload to the registry is completed. the scratch PVC would be released at this time.
root@k8s-master:~/vsrx-kubevirt/CDI# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
vsrx-boot Bound pv0002 25Gi RWO hostpath-provisioner 3m23s
Use multus for multiple interfaces for VMs
Multus is needed incase multiple interfaces is needed for the VM. The interfaces can come from multiple CNIs, or bridge, macvlan or SRIOV. Below is the process to use multus and connect to linux bridges
Install Multus
Install multus using the below. These are obtained directly from the multus quick start guide.
git clone https://github.com/k8snetworkplumbingwg/multus-cni.git && cd multus-cni
cat ./deployments/multus-daemonset-thick-plugin.yml | kubectl apply -f -
Verify the status
root@k8s-master:~# kubectl get pods -A | grep multus
kube-system kube-multus-ds-dmttm 1/1 Running 0 155m
kube-system kube-multus-ds-dpxrw 1/1 Running 0 155m
kube-system kube-multus-ds-qtnmk 1/1 Running 0 155m
Create Network Attachment Definition
root@k8s-master:~/vsrx-kubevirt/NAD# more nad-fxp.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: fxp
spec:
config: '{
"cniVersion": "0.3.0",
"name": "fxp",
"type": "bridge",
"bridge": "br0",
"isGateway": true,
"ipam": {
"type": "host-local",
"subnet": "192.168.5.0/24"
}
}'
Apply the file
kubectl apply -f nad-fxp.yaml
Similarly create multiple NADs so that it can be added to the VM manifest file.
Finally Create VM and Start
Finally create the VM using all the above parameters and start. Ensure you use the same data volume name is being used as that was created
root@k8s-master:~/vsrx-kubevirt# more vsrx-kubevirt_cdi.yaml
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: vsrx
spec:
running: True
template:
metadata:
labels:
kubevirt.io/vm: vsrx
spec:
domain:
ioThreadsPolicy: auto
cpu:
sockets: 1
cores: 4
threads: 1
resources:
requests:
memory: 8Gi
cpu: "4"
limits:
cpu: "4"
memory: 8Gi
devices:
useVirtioTransitional: true
disks:
- name: boot
disk:
bus: virtio
interfaces:
- name: default
masquerade: {}
- name: fxp <<< 2nd interface
bridge: {}
- name: left <<< 3rd interface
bridge: {}
volumes:
- name: boot
dataVolume:
name: vsrx-boot <<< name should match
networks:
- name: default
pod: {}
- name: fxp <<< multus based interface
multus:
networkName: fxp
- name: left
multus:
networkName: left
Ensure you use useVirtioTransitional: true . Else the vSRX will end up in boot loop.
Apply the manifest file
kubectl apply -f vsrx-kubevirt_cdi.yaml
Verify the status.
Once applied, there will be a kubevirt pod which will go into running state. This pod can be used to run virsh commands.
root@k8s-master:~/vsrx-kubevirt# kubectl get pods
NAME READY STATUS RESTARTS AGE
virt-launcher-vsrx-ggf8d 1/1 Running 0 109m
check vm status
root@k8s-master:~/vsrx-kubevirt# kubectl get vm
NAME AGE STATUS READY
vsrx 111m Running True
Login to VM
Login to VM and verify if all network interfaces have been attached.
root@k8s-master:~/vsrx-kubevirt# virtctl console vsrx
Successfully connected to vsrx console. The escape sequence is ^]
root> show chassis fpc
Temp CPU Utilization (%) CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt 1min 5min 15min DRAM (MB) Heap Buffer
0 Online -------------------- CPU less FPC --------------------
root> show interfaces terse
Interface Admin Link Proto Local Remote
ge-0/0/0 up up
gr-0/0/0 up up
ip-0/0/0 up up
lsq-0/0/0 up up
lt-0/0/0 up up
mt-0/0/0 up up
sp-0/0/0 up up
sp-0/0/0.0 up up inet
inet6
sp-0/0/0.16383 up up inet
ge-0/0/1 up up
dsc up up
fti0 up up
fxp0 up up
fxp0.0 up up
gre up up
ipip up up
irb up up
lo0 up up
lo0.16384 up up inet 127.0.0.1 --> 0/0
lo0.16385 up up inet 10.0.0.1 --> 0/0
10.0.0.16 --> 0/0
128.0.0.1 --> 0/0
128.0.0.4 --> 0/0
128.0.1.16 --> 0/0
lo0.32768 up up
lsi up up
mtun up up
pimd up up
pime up up
pp0 up up
ppd0 up up
ppe0 up up
st0 up up
tap up up
vlan up down
Run VIRSH commands
If one needs to run virsh commands as on KVM systems for debugging, then the pod can be used.
kubectl exec -it virt-launcher-vsrx-ggf8d bash
bash-4.4# virsh list
Id Name State
------------------------------
1 default_vsrx running
API reference
The below link contains the API reference
Using emulation
Sometimes we may not have baremetal servers but a bunch of VMs. In such cases where nested virtualization is required, emulation should be turned on.
Perform the below in order to achieve the same.
Edit kubevirt
kubectl edit kubevirt kubevirt -n kubevirt
Add the below params
spec:
configuration:
developerConfiguration:
useEmulation: true
save the file using esc + :wq. Now you can spin up VMs on the cluster.
linux kubernetes