Symptoms
Networking on the host machine is still functioning properly, but all networking stops working on your KVM virtual machines. You can’t even get dhclient to work in a VM. Even if you manually assign an IP address and default route to a VM, it still can’t ping its gateway. Rebooting the virtual machines does no good; they still can’t get an IP address.
Quick and Dirty
Basically, you need to do restart libvirtd and (probably) save your iptables afterword; read on for more info, i.e.- don’t just copy and paste this. You should understand what you’re doing, why you’re doing it, and never paste anything off the internet into a terminal!
$ service libvirtd restart && iptables-save > /etc/sysconfig/iptables
In Depth
Red Hat’s KVM (kernel-based virtual machine) framework, built on libvirt and qemu, has some great characteristics — but it can be fragile, particularly when dealing with networking.
Unlike VMware or VirtualBox virtualization products, KVM is open and transparent about the inner-workings of almost everything. But that has consequences. There can be unexpected effects on your KVM environment in RHEL (or CentOS/Fedora/Scientific Linux…) when you tweak system settings on the HOST machine(s).
VMware does a good job of “protecting” its NAT’ted networks from you and your customized settings on the host operating system, but KVM, not so much…actually I should have said–not at all. Sure, if you try hard enough you can break anything, but VMware’s vmnet.ko kernel module doesn’t inherently rely on the host’s iptables configuration as does KVM. Ah, see? There it is…there’s the issue at hand. This is why your KVM network (and all your KVM virtual machines) stopped working. You tweaked your iptables firewall and who would have known, right? Maybe you added an NFS server, or an Apache instance, a VNC service perhaps? Unlike many popular debian-based systems like Ubuntu, Red Hat .rpm’s don’t do much post-install configuration (don’t get into a holy war with me, it’s the !#@$ truth OK). This translates to, for better AND worse, the fact that when adding a listening network service to a RHEL machine it is unlikely that the post-install script will poke a hole in your iptables firewall automatically for you under the assumption that you actually want to use the new service.
…Like I said, that can be a good AND a bad thing. In an enterprise environment, it’s probably more good than bad because you’re dealing with possible security ramifications in a mission-critical infrastructure (where you should have real hardware firewalls in place anyway, but I digress…)
All of this adds up to one fact: if you added a new listening network service to your RHEL KVM host machine, you had to manually tweak your firewall yourself. To wit, no magically convenient post-install script for the nfs-kernel-server package will do this for you on RHEL. This necessitated the action that brought about your problem. You, like myself and many others, made the one-time mistake of forgetting that KVM needs you to leave alone the (overtly transparent) rules for it’s iptables.
You, like myself and many others probably typed $ sudo system-config-firewall-tui (or something like it).
And that’s all it takes. If you do that and “save” your changes, your KVM network is blown away!
Such handy-dandy iptables utilities as Red Hat’s system-config-firewall-tui should protect you from the mistakes you know you might make in an already-complex iptables setup. You probably thought that by using such a tool that it would generate the correct iptables commands and you wouldn’t thereby be able to mess things up. You thought you were smart to use a tool that avoids human error. But in this case…
YOU WERE WRONG.
You were wrong because you thought your iptables wizard-y tool would preserve your existing setup, or in the least not blow it away. *Sigh*. You, poor soul, are lucky though. Red Hat’s init script for KVM’s libvirtd knows how to fix the problem– and the awesome thing about it is that unlike system-config-firewall-tui, the libvirt init script will preserve existing iptables rules. You can check out the official Red Hat documentation about this if you like, but it comes down to the two commands up above in the “Quick and Dirty” section; call the libvirtd init script and then save your iptables to the persistent rules file in /etc/sysconfig/iptables and your rear is saved.
$ sudo -s $ service libvirtd restart $ iptables-save > /etc/sysconfig/iptables
In conclusion, remember that this should be a one-time mistake 😉