By vSphere

Note: I’m not overly happy with the way the graphics are behaving in terms of resolution. If you’re finding this hard to read – in think in future I will have to crank up the font size in PuTTy – and I’m looking for a modern WordPress theme as this is looking a little too much 2011 for my tastes.

My fellow vExpert Edward Haletky has github crammed full of useful stuff – and last week I spent most of my time getting to grips with his LinuxVSM. As ever I found this really interesting and is a bit of distraction from my main personal project. But like to follow where ever my passions and interest take me. As for myself I’m chatting with Alastair Cooke of about how Edward’s work could be incorporated into his AutoLab Project. Additionally, I”m looking at updating the UDA for the next release of vSphere – adding LinuxVSM to it, as well the deployment of nested homelab – that way the UDA can be used to deploy not just physical, but virtual ESX hosts.

Edward’s LinuxVSM is essentially a Linux version of VMware’s Software Manager. This is a sadly neglected Windows tool which hasn’t had any love from VMware since 2016. I tried downloading VMware’s Windows version and using it, but it didn’t work. In case you don’t know the “software manager” is meant to the ease the pain of downloading software from vmware.com. You can see the LinuxVSM as text-based, and scriptable version of VMware Software Manger allowing access to the main VMware’s VSM Metadata site.

So, using an ordinary personal MyVMware account LinuxVSM can:

  • Download practically almost any piece of software you need from vmware.com
  • You can “mark” a certain product suite as a “favourite” – and using cron LinuxVSM will update your repository with new version of software as they are released.
  • Your repository could be just a local .VMDK or else you could mount from the LinuxVSM to a CIFS or NFS share/export – and store your download on your NAS device.
  • Your account can be a personal  MyVMware account, and you do not need to be a customer account (although there are some bits that are not downloadable except for customers). For many of us this is a godsend. I’ve lost track of the number of “mailinator” accounts I’ve set up in effort to get hold of software from VMware – something I’ve experienced since 2003. Don’t forget you can use the VMUG Advantage to gain access to 1year NFRs if you are not in the vExpert club. VMUG Advantage is great – although sometimes its “downloads” lag behind what is available from the live site. So perhaps the real advantage of the VMUG Advantage are the licensing keys rather than the access to the media.

Step1: Install Linux (Skip if familiar already!)

In my case I used CentOS7 minimal although Debian, Ubuntu and RHEL should work just as well. I’ve been out of the loop for two years so rather than re-using out of date templates it made sense to download the .ISO and do clean installation. This part is too tedious to document but a couple of caveats.

  • Create a generic “user” account of some description. You DO NOT need “root” to ‘install’ the LinuxVSM, you should really use an ordinary account with sudo privileges for any of the setup routine – and just using that account to execute the VSM scripts.
  • Don’t forget to enable the NIC during the installation – unless your idea of fun is editing the network-scripts file.
  • You might want to decide now if you are going to download the software to the local VMDK or else configure LinuxVSM to store the software on your NAS. As this will determine how big you make your VMDK file or whether to add an additional disk for storing the software.
  • I like to add my ordinary user account to the wheel group so I can sudo using usermod -aG wheel user – this will allow you to use sudo to carry out certain tasks rather than using the root account.
  • If you don’t want to bugged for your “user” password when using the sudo command, I would use sudo visudo to and un-remark the line that reads %wheel ALL=(ALL) NOPASSWD: ALL

After the installation I decided to install the open-source VMware Tools. I had this idea of asking Edward the permission to build and distribute all his work as an appliance. I later discovered this was probably a bad idea because I could have so easily have fallen foul of breaching VMware’s acceptable usage policy by accident. The breach would have been accidental and not deliberate. But you can’t be too careful in this day and age. It can be installed and redistributed – but it cannot be pre-configured with MyVMware user accounts.

If you don’t know how to install the open-source version of the tools – this is what you do:

1. Login as root to the CentOS installation and use the yum updater to download and install the Open Source tools

sudo yum install open-vm-tools

During the install select [Y] to the various options. If you are using the virtual machine as a template or if it will be protected by SRM (Site Recovery Manager), then install the deployPkg Tools plug-in. To install this plug-in:

2. Create a file with vi

sudo vi /etc/yum.repos.d/vmware-tools.repo

with the following content:

[vmware-tools]
name = VMware Tools
baseurl = http://packages.vmware.com/packages/rhel7/x86_64/
enabled = 1
gpgcheck = 1

3. Install the deployPkg Tools plug-in with:

sudo yum install open-vm-tools-deploypkg

4. Restart the vmtoolsd service to ensure the service functions correctly. This should clear any warning in vCenter that VMware Tools is not installed to CentOS.

sudo systemctl restart vmtoolsd

Step2: Download, Install and First Run Setup LinuxVSM

Edward has a GitHub page stuffed with all his goodies at https://github.com/Texiwill/aac-lib – I downloaded his entire library with the intention of looking at each of his bits – but for the moment LinuxVSM was my main interest. Next, we transfer the contents of the VSM directory to the Linux system setup in Step2. I just used WinSCP to do that – if you want to make life a little easier you can actually pull down the VSM scripts directly from GitHub if you can locate the “RAW” url for the files.

1. Log on as user using SSH and run the installer with

chmod 755 install.sh
./install.sh 'Europe/London'

If you are unsure of your current times zone you can use the command “date” to view simple version or else this command will tell you more precisely:

timedatectl | grep 'Time zone' | cut -d' ' -f 10

This pulls down all the software and scripts and also an update process to ensure you always can easily update the LinuxVSM. New software is endlessley being release by vmware.com, and Edward needs to keep on top of changes to the software depot on VMware’s website.

The core script for LinuxVSM is vsm.sh which after the install is located in /usr/local/bin which means you don’t need to be in the path to run it – it is marked as being executable so there is no need to use the sh command to execute it. The first run of vsm.sh does a download of the metadata backing the VMware download site – mainly in the form of .xhtml files. Sadly, because of the way the site is constructed the only reliable way of making sure LinuxVSM will always work is downloading all these .xhtml files. This does take a litle time (around 3-5mins on my 20MB domestic broadband), and it is a one-time event.

IMPORTANT: To run vsm.sh you need to log out as root or else use su – user to execute the script.

2. Ensure you login with a standard user account – vsm.sh –help will give you all the information you need

[user@LinuxVSM ~]$ vsm.sh --help
 /usr/local/bin/vsm.sh [--dlg search] [-d|--dryrun] [-f|--force] [--favorite] [-e|--exit] [-h|--help] [-l|--latest] [-m|--myvmware] [-mr] [-nq|--noquiet] [-ns|--nostore] [-nc|--nocolor] [--dts|--nodts] [--oem|--nooem] [--oss|--nooss] [-p|--password password] [--progress] [-q|--quiet] [-r|--reset] [-u|--username username] [-v|--vsmdir VSMDirectory] [-V|--version] [-y] [--debug] [--repo repopath] [--save]
 --dlg - download specific package by name or part of name
 -d|--dryrun - dryrun, do not download
 -f|--force - force download of packages
 --favorite - download suite marked as favorite
 -e|--exit - reset and exit
 -h|--help - this help
 -l|--latest - substitute latest for each package instead of listed
 Only really useful for latest distribution at moment
 -m|--myvmware - get missing suite and packages from My VMware
 -mr - reset just the My VMware information, implies -m
 -nq|--noquiet - disable quiet mode
 -ns|--nostore - do not store credential data and remove if exists
 -nc|--nocolor - do not output with color
 -p|--password - specify password
 --progress - show progress for OEM, OSS, and DriverTools
 -q|--quiet - be less verbose
 -r|--reset - reset repos
 -u|--username - specify username
 -v|--vsmdir path - set VSM directory
 -V|--version - version number
 -y - do not ask to continue
 --dts - include DriversTools in All-style downloads
 --nodts - do not include DriversTools in All-style downloads
 --oss - include OpenSource in All-style downloads
 --nooss - do not include OpenSource in All-style downloads
 --oem - include CustomIso in All-style downloads
 --nooem - do not include CustomIso in All-style downloads
 --debug - debug mode
 --repo path - specify path of repo
 --save - save settings to $HOME/.vsmrc, favorite always saved on Mark

All-style downloads include: All, All_No_OpenSource, Minimum_Required
 Requires packages:

wget python python-urllib3 libxml2 perl-XML-Twig ncurses bc

To Download the latest Perl CLI use (to escape the wild cards):
 ./vsm.sh --dlg CLI\.\*\.x86_64.tar.gz

Use of the Mark option, marks the current product suite as the favorite. There is only 1 favorite slot available. Favorites can be downloaded without traversing the menus.

I’ve put together some samples/examples:

vsm.sh -u me@me.com -p  ‘P@ssw0rd!’-ns -y --nodts --nooss --nooem --repo /home/user

Note: This logs you into VMware’s VSM Metadata, and pulls the metadata of the site, but doesn’t include Drivers & Tools (so no PowerCLI/Perl included!), and skips the Open Source and OEM version of thing like VMware ESXi (beware of this – if you use a Custom ISO such as the HP ISO like that I use on my HP ML350e Gen8s servers). LinuxVSM can store your MyVMware credentials securely and is a requirement for this “favorite” process – if you feel uncomfortable with that – then user the switch -ns to “not store” your password. Even if you provide the password on the CLI prompt with -u and -p your password would still be visible in the bash.history. Something that only became apparent to me when working with Edward. My Linux skills were never what they should be – and in the last two years are decidedly rusty! The default is for LinuxVSM to store your credentials – if you don’t like this idea you need always remember to use the switch -ns to ensure that LinuxVSM does not store your credentials. Finally, if you are providing a password on the CLI remember to use single quotes ‘around your password’ as special characters could be seen as escape code. The –repo switch redirects your downloads to a location other than /tmp/vsm which is the default. You need to be careful about downloading large amounts of data and filling up / (root). This “obvious” to Linux people, but less so to Windows folks.

vsm.sh --nooss

Note: This runs vsm.sh in an “interactive” mode – where you are prompted for your username and password. Drivers and Tools and OEM ISOs would appear in the LinuxVSM menus – but Open Source components would not.

IMPORTANT: Remember at this stage you’re not actually downloading any software from vmware.com, merely accessing the VMware VSM metadata needed to traverse the website in a text-mode way which in turn allows you to select the software you want to download. Despite this the volume of .xhtml data is not insignificant and can take nearly 25mins to download depending on your internet speeds. Edward is working on a method to radically reduce this time.

The main thing to say is the vmware.com website holds a lot of files – so I would pay close attention to the “no” style switches on vsm.sh that allows you to limit the volume of material you wind-up downloading as favourite. To really limit what you download to individual file the –dlg switch is a must – however some of the power of LinuxVSM is being able to “subscribe” to a “favorite” and have that downloaded whenever it changes.

Note: Edward is always improving – i think since this screen grab was taken we are now on to release 4.0.4

LinuxVSM will be busy downloading the .xhtml files

Now that the VMware VSM metadata pages are download we can navigate them using the LinuxVSM menus.

If you have stored your credentials within LinuxVSM subsequent connections will not need a user name password. In the case I connected using vsm.sh -y –nooss

Note: Notice the 1 next to use credstore indicates credentials have been stored. If you have done this accidentally remove the VSM directory with rm –rf /tmp/vsm

Step3: Change Download Location (Optional)

I normally store all my software on cheap NAS device I have which has NFS export on it called “software”. It’s quite old this NAS device and was one of the first domestic NAS devices to support VMware ESXi

1. Create a mount-point for the NFS export within the LinuxVSM

sudo mkdir -p /mnt/nfs/software/vmware/vSphere65U1

2. Next install the NFS Client to tools CentOS7 with:

sudo yum -y install nfs-utils

3. To mount for this boot time use the following:

 sudo mount -t nfs 10.20.30.253:/nfs/software/ /mnt/nfs/software/

4. This changes the way I run vsm.sh to reflect the new repo-sitory locations like so:

vsm.sh -u me@me.com -p -ns ‘P@ssw0rd!’ -y --nodts --nooss --nooem --repo /mnt/nfs/software/

5. To make this permamant I can edit fstab and add fixed entry like so:

sudo vi /etc/fstab

[i] to enter insert mode

10.20.30.253:/nfs/software /mnt/nfs/software/ nfs defaults 0 0

[ESC] to exit insert mode
[:] to enter command-mode
[w!] to write the file
[q!] to quit vi

Step4: Start Downloading!

Manual One-Time Download

Using the menus, we can navigate for binaries we wish to download. For instance, the menu options [1], [4], [1], [5] will get me to the vSphere 6.5 Enterprise 6.5 suite within which I can see the components such as ESX, NSX, VC and so on

If I type [5] this will expand the ESX node in the menu I can see the generic ESX “VMvisor-Installer” also side the .zip updater, the .iso rollup and the .pdf file.

If I press [4] I can take just the ISO file

If I press option [9] it downloads all the CustomISO images including the supporting files. This is a large amount of data – and another chance to something else.

If we examine the contents of /tmp/vsm we will see there is a “download group” called dlg_ESX65U1 and within it a CustomISO directory which contains the files downloaded.

Note: Remember its good practise to relocate download location to somewhere else using the –repo switch. In my case I made this my Synology NAS which back my homelab. As an aside DLG refers to a “Download Group”. This is actually a VMware term and you will see it used in the URL’s that back the official vmware.com/downloads site. So in the example below the Download Group called “OVF420” contains all the version of the OVFTOOL’s that work across Win32/Win64/Linux and MacOS.

 

Mark a Favourite and Download All (Recommended)

This is probably THE most powerful feature of LinuxVSM – the ability download entire suites of products and “subscribe” any changes, updates or new drops of software. Let say you wanted to download the entire contents of particular suite within LinuxVSM – you can “mark” a particular product suite as “favorite”, and then download all of that suite. Additionally, we can setup a cron job to check daily to see if any new releases have been added to the favourite. This is the recommended method although it does download a large amount of data. It’s recommended because it is the MOST reliable method and the most complete method of downloading – and also ensure that as new software becomes available a crontab job will check and download the new software as it becomes available.

In this case I ran vsm.sh -y –nooss again and navigated to the vSphere 6.5 release using the menu options – [1], [4], [1], [5]. Typing option [14] marks this suite as my preferred distribution.

Once the favorite has been set. Exit VSM and run the command:

crontab -e

[i] to enter insert mode

 0 2 * * * /usr/local/bin/vsm.sh -y -mr --favorite

[ESC] to exit insert mode
[:] to enter command-mode
[w!] to write the file
[q!] to quit crontab

Note: This would cause LinuxVSM to check the VMware VSM metadata for new distributions available within your marked suite as favourite. In my case on 20MB broadband “Minimum Required” started at 11pm and finished at 2am in the morning.

This is a breakdown of download groups created and how much space they take up which amounts to about 61GB in total.

 3.7G ./vsm/dlg_BDE_232
 6.4G ./vsm/dlg_ESXI65U1/CustomIso
 4.9M ./vsm/dlg_ESXI65U1/DriversTools
 7.6G ./vsm/dlg_ESXI65U1
 5.1G ./vsm/dlg_NSXV_640
 7.2G ./vsm/dlg_VC65U1G
 11G ./vsm/dlg_VDP617
 3.2G ./vsm/dlg_VIC131
 1.4G ./vsm/dlg_VR6512
 0 ./vsm/dlg_VRLI_451_VCENTER
 8.3G ./vsm/dlg_VROPS_661
 1.9G ./vsm/dlg_VROVA_731

The time taken to download this varies on your speed – that works out roughly as:

8 Mbit/s 17hrs
10 Mbit/s 14hrs
24 Mbit/s 5hrs
80 Mbit/s 1.4hrs
100 Mbit/s 1.2hrs

IMPORTANT: In the current release there is only 1 favourite slot.

Manual One-Time Download of a specific file

Using the –dlg switch we can specify a specific file to download. One use of –dlg command is too massive reduce the download time by not pulling down components within a suite or download group that are not required. For instance by merely downloading the Custom .ISO rather than entire download group – I was able to reduce the download from 6.4GB to 1.8GB.

However, due to the sheer number of the files hosted with the VMware VSM Metadata its is impossible to LinuxVSM to list every file by its specific name. Additionally, some files appear more than once in the repository, as well often having very similar names. It’s therefore possible to query of specific a name that is too exact, which result in negative response – or alternatively too broad a query which results in too many results. It’s for this reason you will see some of the strings do not use the full filename – but a partial one.

The recommendation is to use favourites as outlined previously – if you must use the –dlg switch we would recommend testing first before embedding any calls into scripts and such like.

Generic VMware ESXi6.5 U1 .ISO Image

vsm.sh -y -f --dlg VMware-VMvisor-Installer-6.5.0.update01-5969303.x86_64.iso

Custom Vendor specific VMware ESXi6.5 U1 .ISO Image

vsm.sh -y -f --dlg VMware-ESXi-6.5.0-Update1-6765664-HPE-650.U1
vsm.sh -y -f --dlg ESXi-6.5.0-Update1-7388607-HPE-650.U1
vsm.sh -y -f --dlg VMware-ESXi-6.5.0-Update1-P-6765664-hitachi-1201-HA8000V.iso
vsm.sh -y -f --dlg VMware-ESXi-6.5.0-Update1-5969303-hitachi-0203-Blade-HA8000.iso
vsm.sh -y -f --dlg Vmware-ESXi-6.5.0-5969303-Custom-Cisco
vsm.sh -y -f --dlg VMware-ESXi-6.5.0.update01-5969303-Fujitsu-v411-1.iso
vsm.sh -y -f --dlg ESXi-6.5.0.update01-7388607-LNV-20180201.iso

VMware vCenter6.5 U1 DVID .ISO Images vCenter Server Appliance (VCSA) and Windows vCenter

vsm.sh -y -f --dlg VMware-VCSA-all-6.5.0-8024368.iso
vsm.sh -y -f --dlg VMware-VIM-all-6.5.0-8024368.iso

Perl CLI using escape codes:

vsm.sh -mr -y -f --dlg CLI\.\*\\.x86_64.tar.gz

Note: The use of the -f causes vsm.sh to forcibly download the file and over-write if it already exists. If the file has never been downloaded before – it’s not strictly necessary.

Step5: Enable Daily Update LinuxVSM

Enabling a daily update is useful as VMware are endlessly creating new releases which then changes the way vmware.com/download functions. This causes Edward to have push-out and update to ensure LinuxVSM remains aware of this change. Using vi we can create a small script for instance called update.sh which will force LinuxVSM to see if any updates to it have happened. This file can be stored in /etc/cron.daily (which usually runs at 3AM).

IMPORTANT: The script aac-base.install has its a call to set the timezone. I would recommend editing this to ensure it matches your location/region. Look for the variable TZ=’America/Chicago” and set this to be your preferred timezone, for example Europe/London

1. cd /etc/cron.daily
2. sudo vi update.sh

[i] to enter insert mode

#!/bin/sh
cd /home/user/aac-base
./aac-base.install -u
./aac-base.install -i vsm 'Europe/London'

[ESC] to exit insert mode
[:] to enter command-mode
[w!] to write the file
[q!] to quit vi

IMPORTANT: The script aac-base.install has the option to set the timezone. I would recommend doing this in your update.sh to ensure your TZ settings aren’t clobbered. If you don’t change this you will find your TZ become’s CDT or American/Chicago.

3. Save the file, and make it executable with

sudo chmod u+x update.sh

Note: You can confirm your version of LinuxVSM with the command vsm.sh -V

Conclusions:

I’m sooooo pleased Edward has put the effort into building VSM – personally for a hard-core VMware enthusiast like myself its a god-send. I’ve been struggling with access media from vmware.com for more than a decade – and this fixes a long-term bug bear for me big time.