[kernel-xen] Zombie domU

Tue Dec 4 20:32:27 EST 2012

I had a domU with a bad kernel= command line in grub, which crashed half way through the boot process.

When I tried to destroy it, it appeared to succeed, but the VM is still listed in 'xm list':

# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  1024     1     r-----     13.0
dmodftp                                         1024     1                 0.0

Notice there is now no ID for dmodftp domU. I've tried destroying many times but it won't go away. I've rebooted dom0, I even booted to a non-xen kernel, erased all the xen packages and kernel and reinstalled them, and it's still there! Any idea where this is coming from?

We've got 10 machines in the pool, and the VMs are all on shared storage, so I logged in to another Xen box and managed to boot up this particular VM just fine (with fixed kernel= in grub). I just need to figure out how to destroy this zombie.

I checked the xend.log and this is all I can find. After rebooting, this gets logged:

/var/log/xen/xend.log:[2012-12-04 10:12:37 2945] DEBUG (XendDomainInfo:260) XendDomainInfo.createDormant({'vcpus_params': {'cap': 0, 'weight': 256}, 'PV_args': '', 'features': '', 'cpus': [[]], 'use_tmp_kernel': False, 'devices': {'1e8aa463-4979-db41-f55c-9015d463d871': ('console', {'protocol': 'vt100', 'location': '2', 'uuid': '1e8aa463-4979-db41-f55c-9015d463d871'}), '0efc9a3a-eb93-7b34-59b5-1d5008bf7b1d': ('vbd', {'protocol': 'x86_64-abi', 'uuid': '0efc9a3a-eb93-7b34-59b5-1d5008bf7b1d', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'sda1:disk', 'uname': 'file:/shared/xen/dmodftp.img', 'mode': 'w', 'VDI': '', 'backend': '0'}), 'cf35b798-12d2-3131-780a-76aec5fb96da': ('vbd', {'protocol': 'x86_64-abi', 'uuid': 'cf35b798-12d2-3131-780a-76aec5fb96da', 'bootable': 0, 'driver': 'paravirtualised', 'dev': 'sda2:disk', 'uname': 'file:/shared/xen/dmodftp.swap', 'mode': 'w', 'VDI': '', 'backend': '0'}), 'fbd5c705-ab9b-6fa4-8728-dc2d4217d716': ('vif', {'bridge': 'xenbr0', 'uuid': 'fbd5c705-ab9b-6fa4-8728-dc2d4217d716', 'script': '/etc/xen/scripts/vif-bridge', 'ip': '83.98.160.23', 'mac': '00:16:3e:D5:45:93', 'backend': '0'})}, 'memory_sharing': 0, 'superpages': '0', 'VCPUs_live': 0, 'PV_bootloader': '', 'actions_after_crash': 'restart', 'vbd_refs': ['0efc9a3a-eb93-7b34-59b5-1d5008bf7b1d', 'cf35b798-12d2-3131-780a-76aec5fb96da'], 'PV_ramdisk': '', 'is_control_domain': False, 'name_label': 'dmodftp', 'VCPUs_at_startup': 1, 'HVM_boot_params': {}, 'platform': {'tsc_mode': '0', 'pci': [], 'videoram': '4', 'nomigrate': '0'}, 'PV_kernel': '', 'console_refs': ['1e8aa463-4979-db41-f55c-9015d463d871'], 'vif_refs': ['fbd5c705-ab9b-6fa4-8728-dc2d4217d716'], 'on_xend_stop': 'ignore', 'pool_name': 'Pool-0', 'memory_static_min': 0, 'HVM_boot_policy': '', 'description': '', 'VCPUs_max': 1, 'start_time': 1354540655.6700001, 'memory_static_max': 21474836480, 'actions_after_shutdown': 'destroy', 'use_tmp_ramdisk': False, 'on_xend_start': 'ignore', 'memory_dynamic_max': 1073741824, 'actions_after_suspend': '', 'is_a_template': False, 'PV_bootloader_args': '', 'memory_dynamic_min': 1073741824, 'uuid': '0206a085-36f9-6694-7873-6bbcf627d13f', 'cpu_time': 0.0, 'shadow_memory': 0, 'target': 0, 'vcpu_avail': 1L, 'notes': {'HV_START_LOW': '18446603336221196288', 'FEATURES': '!writable_page_tables|pae_pgdir_above_4gb', 'VIRT_BASE': '18446744071562067968', 'GUEST_VERSION': '2.6', 'PADDR_OFFSET': '0', 'GUEST_OS': 'linux', 'HYPERCALL_PAGE': '18446744071578849280', 'LOADER': 'generic', 'SUSPEND_CANCEL': '1', 'PAE_MODE': 'yes', 'ENTRY': '18446744071591563776', 'XEN_VERSION': 'xen-3.0'}, 'other_config': {}, 'auto_power_on': False, 'actions_after_reboot': 'restart', 'Description': '', 'status': '1', 'vtpm_refs': [], 's3_integrity': 1})

About the machine:

# uname -a
Linux ams1-xen-8.localdomain 3.6.7-1.el6xen.x86_64 #1 SMP Wed Nov 21 23:43:46 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

# xm info
host                   : ams1-xen-1-8.localdomain
release                : 3.6.7-1.el6xen.x86_64
version                : #1 SMP Wed Nov 21 23:43:46 EST 2012
machine                : x86_64
nr_cpus                : 4
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 2394
hw_caps                : bfebfbff:20100800:00000000:00000940:0000e3bd:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 8191
free_memory            : 7072
free_cpus              : 0
xen_major              : 4
xen_minor              : 2
xen_extra              : .0
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : dom0_mem=1024M cpufreq=xen dom0_max_vcpus=1 dom0_vcpus_pin
cc_compiler            : gcc (GCC) 4.4.6 20120305 (Red Hat 4.4.6-4)
cc_compile_by          : mockbuild
cc_compile_domain      : crc.id.au
cc_compile_date        : Sat Oct 27 04:00:26 EST 2012
xend_config_format     : 4

Any and all suggestions greatly appreciated!

Regards
Chris

--
Chris Wik
Anu Internet Services Ltd
www.cwik.ch | www.anu.net