Are you trying to troubleshoot KVM virtualization problem?

This guide is for you.

There are various log files stored on the host system to assist with debugging KVM related problems.
Here at Ibmi Media, as part of our Server Management Services, we regularly help our Customers to resolve issues regarding KVM Virtualization.
In this context, we shall look into how to troubleshoot KVM Virtualization Problem with log files in Linux for our customers.

Some log files to be familiar with for Troubleshooting KVM Virtualization Problem

There are various log files on the host system to assist with debugging KVM-related problems.
The files are as follows:
1. $HOME/.virtinst/virt-install.log – virt-install tool log file.
2. $HOME/.virt-manager/virt-manager.log – virt-manager tool log file.
3. /var/log/libvirt/qemu/ – Log files for each running virtual machine.

We can use grep and other Linux tools to view these files:

# tail -f /var/log/libvirt/qemu/freebsd.log

# grep something $HOME/.virtinst/virt-install.log

$ sudo tail -f /var/log/libvirt/qemu/openbsd.log

To Connect to the Console

We can use the virsh command to connect to the guest serial console using the following command:

$ virsh list

$ virsh console freebsd

This helps to troubleshoot problems such as networking and much more from the host itself.
However, we need to configure the guest operating system for a serial console.

KVM Configuration Files

We can edit the configuration files to define hardware properties for VMs or other setups.
They are located in /etc/libvirt/qemu/ directory. Here is a sample FreeBSD guest configuration file.

# less /etc/libvirt/qemu/freebsd.xml

OR

$ virsh dumpxml freebsd

When we create guests with the virt-manager or virt-install, the guest configuration files are created automatically in the /etc/libvirt/qemu/ directory.

We can edit this file using a text editor or virsh command given below:

virsh edit freebsd

virsh edit centos

KVM tools

Using the following command we can install KVM-tools package which contains some diagnostics and debugging tools for KVM:

# yum -y install kvm_tools

For mounting debugfs we can use the following commands:

# mount -t debugfs debugfs /sys/kernel/debug

# kvm_stat

We can use normal tools to troubleshoot and get other information using the following tools:
i. ps, pstree and top
ii. vmstat, iostat and lsof
iii. tcpdump, brctl, ip and ifconfig

Tips to Troubleshoot KVM virtualization problem

We can see the way in which our Support Techs troubleshoot KVM virtualization problem.

1. Permission issues

Prior to libvirt 0.6.1, all virtual machines run through libvirt were run as root, giving full administrator capabilities.
In libvirt-0.6.1, security started to improve with the addition of svirt. It attempts to automatically apply selinux labels to every file a VM needs to use, like disk images. When a VM tries to open a file that are not labeled by libvirt, permission will be denied.
As of libvirt-0.6.5, VMs were now launched with less process capabilities. This prevents the VM from doing allowing to alter host network configuration.
The libvirt-0.7.0 VM emulator process  run as an unprivileged 'qemu' user other than as ‘root’ by default.

2. Changing the QEMU/KVM process user

We can change the user that libvirt will run the QEMU/KVM process by editing /etc/libvirt/qemu.conf.
For example, if we want to run KVM as the user 'ibmimedia', we would set the fields to:

user='ibmimedia'

group='ibmimedia'

After that we can restart libvirtd with the following command:

# service libvirtd restart

3. Changing SVirt/Selinux configuration

We can disable SVirt for the libvirt QEMU driver by editing /etc/libvirt/qemu.conf.
For this we have to uncomment and set the following:

security_driver='none'

After that we can restart libvirtd with the following command:

service libvirtd restart

We must keep in mind that changing the SVirt/SELinux settings may have security implications.

4. Changing QEMU/KVM process capabilities

Libvirt by default launches QEMU/KVM guests with less process capabilities.
We can disable it by editing /etc/libvirt/qemu.conf.
For this we have to uncomment and set the following:

clear_emulator_capabilities=0

After that we can restart libvirtd with the following command:

service libvirtd restart

5. Ensuring system is KVM capable

Verify that the KVM kernel modules are properly loaded.
We can use the following commands:

$ lsmod | grep kvm

kvm

kvm_intel

If that command did not list kvm_intel or kvm_amd, KVM is not properly configured.
If our hardware does not support virtualization extensions, we cannot use KVM acceleration, only plain QEMU is an option.
However, if it supports virtualization extensions we can try to reload the kernel modules using the following command:

su -c 'bash /etc/sysconfig/modules/kvm.modules'

After that, we can retry the above lsmod command and see if we get the desired output.
In case the kvm.modules command produces an error we can use the following command:

dmesg | grep -i kvm

6. Graphical console access

In order to get a graphical console on the guest, we can either use 'virt-manager' and select the console icon for the guest, or we can use the 'virt-viewer' tool to just directly connect to the console:

virt-viewer guestname

7. Accessing data on guest disk images

For the guest image to live, we must only use read-only access, otherwise we may end up corrupting the disk image. Hence it is always safe to use guestfish –ro.
The guestfish program allows us to manipulate guest disk images without running the guest:

su -c 'yum install guestfish'

guestfish -d NameOfGuest -i --ro

><fs> ll /

><fs> cat /boot/grub/grub.conf

[Still facing issues with KVM Virtualization? – We can help you. ]