Talking About vSphere Integration With OpenStack

I mentioned in my previous post how much I appreciated being able to work with and to learn from really smart people, certainly folks smarter than myself.  Well earlier this week, I sat down with someone I had only heretofore spoken with via Twitter – a Nova Compute engineer with the Rackspace Public Cloud.  He and I spoke briefly about vSphere integration with OpenStack Nova Compute, given the contributions VMware made to the Grizzly release, and the level of functionality exposed via the vSphere API, such as vMotion and DRS.

We picked the discussion back up earlier today, via e-mail, after my colleague took a look at that section of the Nova code in Github.  We realized that there was a bunch of questions we both have, even after looking at the code.  So, with his permission, I decided to post our discussion from today with the hope that:

  1. We can get answers from some really smart people who know OpenStack and/or the vSphere API.
  2. We can have a constructive and hopefully thought-provoking discussion about vSphere integration with OpenStack.

So, posted below is the actual e-mail thread with my colleague’s name removed.  Please comment to the post as you see fit and share with other folks who may be interested in the discussion.


Date: Wednesday, May 1, 2013 11:58 PM
To: Kenneth Hui <>
Subject: vSphere with Nova

Hey Ken,

It was good chatting with you on Monday.

If you’ve got some OpenStack specific questions (especially public cloud), I can try to help!

By the way, the VMware implementation in Nova looks to have vMotion:

Doesn’t look like it has any HA functionality built-in.


From: Ken Hui
Sent: Thursday, May 02, 2013 11:28 AM
Subject: Re: vSphere with Nova

It was great chatting with you as well. I will be sure to reach out to you now that I know a Nova expert in-house. 🙂

I’ll look for you when I am back at the Castle for Rookie O in June.

I see a “VM Rescue” section that looks it might be the HA functionality. I also see some code for disk migration that looks like it might be Storage vMotion. Finally, it appears that the Live Migration code may support DRS.

What do you think?


Date: Thursday, May 2, 2013 11:36 AM
To: Kenneth Hui <>
Subject: RE: vSphere with Nova

I’d have to look but essentially rescue mode spawns a new VM of identical type/image (with a known good base disk) and attaches the instance’s root as a secondary disk. Ideal for recovery situations.

I think the live migration code supports DRS clusters.

This bit was interesting:

nova host-update –maintenance enable would evacuate a host and live migrate every instance off.

The VMware api tasks end in _Task, so just running

grep -R Task /nova/nova/virt/vmwareapi


From: Ken Hui
Sent: Thursday, May 02, 2013 11:42 AM
Subject: Re: vSphere with Nova

That last piece looks like the vSphere maintenance mode that is often used to service the server or to update ESXi; you put the host into maintenance mode and vCenter live migrates all VMs. Looks like Nova is essentially making API calls to vCenter and letting it choose VM replacement. Do you see API calls back to Nova to notify the controller of these changes?


Date: Thursday, May 2, 2013 11:50 AM
To: Kenneth Hui <>
Subject: RE: vSphere with Nova

I don’t.

But, come to think of it, if we’re using the VMwareVCDriver wouldn’t that abstract away the operations at a host level? When working with the vSphere API I never had to know the host I was working on to power on/off an instance, just the cluster.


From: Ken Hui
Sent: Thursday, May 02, 2013 11:59 AM
Subject: Re: vSphere with Nova

Good point. Nova may simply be doing the hand-offs to vCenter and relying on vC to track the VMs.

Btw, do you mind if I put some of our conversation on my blog?


Date: Thursday, May 2, 2013 12:01 PM
To: Kenneth Hui <>
Subject: RE: vSphere with Nova


Go for it. I would like to see some more conversations in the wild about the vSphere implementation with OpenStack. It’d move everyone forward. =)


I hope readers following Cloud Computing in general and OpenStack in particular will find the exchange above of interest.  Again, I encourage comments on this post.



  1. Hi guys, happy to see you taking a look at the VMware code in Nova. I expect significantly more contributions in this driver during Havana, so the more Nova experts who have familiarity with it (and can therefore review changes) the better 🙂

    You have basically hit on the key realization toward the end of the thread. The current driver models an entire VMware cluster as a single Nova “Host”, since it is a single unit of compute capacity. Thus, vMotion, DRS, HA, etc. are supported if the cluster is configured to support them, but, the migrations are essentially “behind Nova’s back”, not driven by Nova. Scott Lowe did a demo at the OpenStack that I’d love to point you to, but it looks like the demo videos weren’t posted (or just not yet?).

    There are some proposals in grizzly to model vSphere compute capacity at granularities smaller than a cluster (e.g., a resource pool), updating Host records in Nova as vCenter changes. From that point, its probably not a stretch to imagine having the option of modeling each ESX server as a Nova Host, such that nova driven live migration would work, but I haven’t yet heard people proposing that.

  2. Dave,

    Thank you for your valuable input. One question I have is that since OpenStack treats a vSphere cluster as single entity, does this mean that live migration of a single VM between nodes cannot be done via OpenStack? If that is the case, is that one of the use cases that would require the modeling of each individual ESXi host?

    My take is that the more functionality we enable though Nova, the more use cases can be extended for OpenStack. Right now, there are a large number of legacy workloads that only benefit from but requires the HA features provided by a hypervisor such as vSphere. So, tighter and broader integration between vSphere/vCenter and OpenStack will allow the former to be a more viable option for an OpenStack based Cloud and allow the latter to help end-users to bring along legacy workloads.

    Thanks again.


  3. I think this is very similar to how XenDesktop Provisioning Server treats vSphere from a VDI perspective. XenDesktop doesn’t have the capability to interact within the vSphere Cluster itself. vCenter handles all of the VM control within the Cluster.

  4. I wouldn’t be surprised if the reason they did this is because OpenStack doesn’t have a general VM HA solution of itself yet. There are several ways you can hook in a HA solution, but the different requirements of the different environments with on top of that all the differnet hypervisors and storage- and clustering solutions don’t make it easy to generalize it.

  5. Also if it is clear what the code of a generalized VM HA solutions should look like, currently in OpenStack there is no clear place where that code should live. Maybe if the first is solved it will be obvious though. 🙂 I have a feeling it is going to be lots of pieces of code spread over 3 or 4 areas.

  6. Ken, that’s correct. With the current integration, an admin can live migrate a VM using vCenter (which exposes per-host granularity), but not using OpenStack. I think modeling each host in the DB would be a requirement for enabling this in Nova, but I’d have to look at the code more closely.

    An interesting exercise would be to setup two vSphere clusters, run the nova command to migrate between them, and see if the vSphere driver gives you a sane explanation of why that won’t work.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s