[Alternative Blogpost Title: Follow your personal unicorn…]
One thing I’ve been investigating is whether I can get ESX, vSphere, vCloud Director all working WITHIN a vCloud environment. The answer is yes, this is of course possible. After all its principle behind our hands-on-labs at VMware. I’m pretty sure I could get this working within the domain of my own lab environment, but what I would really like to do is try to this out on a public cloud environment. There’s a couple of agendas here really. Firstly, the hardware resources required to get a REALISTIC vCloud environment up and running are not insignificant – not beyond the reach of the common-man but still a bit of an ask. You could go down the route of buying an array of inexpensive PCs or tower-servers which is popular amongst homelabbers – or you could go for the vTARDIS approach – where you have one jumbo PC with truckload of Memory/SSD to make it useable. Alternatively…. you could try taking the nested ESX concept to the next level and have it hosted in within a vCloud Server Provider environment – you call this by a number of terms – cloud nesting (where use one cloud to nest another cloud within it), nested clouds or clouds within clouds sounds rather nice too. Anyway, I’m going to play in my own lab to get this thing working (mainly so I can learn what the requirements are), some of my buddies from the HoL are going to give me an idea of what’s required – and I’m also going to try and persuade a vCloud Service Provider I know if they’d be prepared to support a PoC. I’m thinking VMUG Advantage+VMTN Subscription+LabCloud might make an interesting proposition – heck, chuck in some credits for VMware Education you’d have a good package of resources for learning everything VMware.
Personally, I know that I’m starting from a low-base. I’ve never done Nested ESX before – and I’m interested to see how some of the automation that vCNS Edge Gateway does, but…..
As I see it there are number of challenges. Firstly (HARDWARE), as you may or may not know nested ESX does require particular hardware requirements, as well as options available within the web-client. These are precisely the sort of controls that you might not have access in a vCloud environment. To be honest I personally think the hardware requirements whilst VERY IMPORTANT in homelabs can by and large be taken for granted in public cloud environment – where modern CPU sets should be in use, with the attributes turned on in the BIOS.
As for enabling “nested ESX” Life is much simpler in ESX 5.1 because of the ability to add the creation of ESX VMs in the /etc/vmware/config file – a good account of this sort of nesting where you have “ownership” of the physical ESX instance can be found – a good account of this process can be found on William Lam’s blog – How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5
Similiarly, vCloud Director 5.1 (assuming that’s what your provider is using vCD5.1) has an option to expose “hardware-assisted CPU virtualization” like so…
Secondly (VM Hardware Levels), you have little or no control over what hardware-level the vCloud Service Provider has enabled. As you may or may not know – when a Provider vDC is created in a vCloud Director the SysAdmin sets the hardware-level.
That can cause problems later on down the track when it comes to importing OVFs. OVFs make a virtual machine portable, but even it cannot buck the VM Hardware Levels. So VM created on HW Level9 in vSphere 5.1 and then exported as an OVF, and then imported to a Provider vDC that support HW Level 8/7 will create this error:
That can cause challenges with .OVFs and OVAs that have been distributed via 3rd parties….
So I think the moral of the story is… if you going to go for this configuration – you need to confirm with your Service Provider what the hardware-level is for their Provider vDC. Remember we don’t support downgrading the HW Level once a VM has been upgraded…
Thirdly (Networking) – the ESX host’s management network needs to be set to “Promiscuous” under the Portgroup settings – and so do another NICs used by nested-VMs (VMs running inside nested-ESX hosts – keep up! it’s like “Inception” this!). By default when vCloud creates Organization Network the security settings are configured to Reject, Reject, Reject…
The same is true of vApp Networks as well, and although from vCloud Director you can control IF you want a vApp Network – you cannot control the portgroup settings backing the vApp… without access to the vSphere layer… So that would mean having a vCloud Service Provider willing to make that customization for you… I’m also thinking of the networking requirements inside the inner-walls of your virtualized lab – how would VLAN tagging, VCNI, and VXLAN work within this virtual virtualization layer – this cloud within a cloud… That’s a bridge I will cross when I get there….
There’s a couple of things to consider about networking I think from a vCloud Director perspective. If you put the Cloud Nesting vApp on vApp Network, when you power it down the vCNS Edge Gateway is powered off, deleted along with its accompanying portgroup – unless you set the option to “Retain IP/MAC”. If you put the Cloud Nesting vApp on an Organization Network you have to make the change to the portgroup once. Of course “promicious mode” is not for “free” from a networking perspective. By defintiion there would be a number duplicate packets on your network and as every VM recieves every packet, the physical switch has to process it and discard it.
Conclusions…
All in all if you want your HomeLAb in the cloud with full rights to it – using a pay-as-you-go model to fund your homelab – I see many technical challenges, and you might find it winds up being no cheaper than the one off cost of some entry-level PCs stuffed with memory. That might change if a Service Provider perceived a market (in which they could make money) for selling VMware labs to people in the vCommunity OR alternatively…. if they saw an indirect value in doing so. Help a guy setup his homelab in a cloud… perhaps he might recommend that SP when it came to hosting in the cloud proper?
Anyway, I’m quite taken by the idea of trying emulate the VMworld HoL in my lab. Despite covering ground others far more able have trode before – its new to me, and I think I will learn loads along the way. So I might take a detour in my journal look at how this thing is setup. Plus I’m also think about fun ways to describe the layers upon layers of virtualization/cloud it entails. “Nesting” is nice, but I’m a fan of the “vINCEPTION” joke that circulates on twitter. So how about vINCEPTION0 to describe the physical layer with anncillary infrastructure VMs and your vCloud Environment “proper”. vINCEPTION1 to describe Cloud Suite running in a vApp (ESX, vCSA, vCNS Manager, vCD) and vIINCEPTION2 to describe vApps running in the vINCEPTION1 layer…
All I need then is some kind of totem to remind me of which layer I was in. I was thinking of silver foil unicorn…