CSIM Technical Notes

When a student leaves AIT, there is no automatic procedure to save and removes the files that were created on the local disks of guppy.

Further more, because the students are using some tools to virtualize environment, there are many file duplicates that should be taken into account to reduce the space on the final backup. In some instances, a space reduction of 30% could be achieved by simply removing the duplicated files.

1. Connect to guppy and become super user;
2. Compress the files of the user:

   find /home2/user -type f -exec xz -f -9 {} \;
   

3. Remove duplicates:

   freedups.pl -a /home2/user
   

4. Create some directory on banyan to receive the files, for example /home/corse/deleted/user/guppy;
5. Transfer the files:

   cd /home2/user; tar cf - . |ssh someone@banyan sudo -A 'sh -c "cd /home/corse/deleted/user/guppy; tar xfBp -"'
   

6. Remove the files from guppy.

Starting with ESXi 6.5, under certain conditions, ESXi server will loose the permissions that was defined based on Active Directory. The server is still joint to the domain, but the user will not be able to access their VM anymore. All permissions have to be regenerated by hand.

That affects the permissions set on the VMs but also the permissions set at the server level.

While the cause is not exactly known, I suspect it is linked to a temporary unavailability of the AD (like a reboot): when ESXi server cannot reach the AD, it wipes off any permission corresponding to that AD. This is critical with the server hosting students VMs, especially the VMs for WAE course that has two or three students per VM.

To solve this problem, I have created a collection of tools that will automatically back-up the permissions based on AD and restore them if they disappear. The tools have been bundled in a VMWare VIB (vSphere Installation Bundle) for easy deployment.

Download and install the VIB

Get the VIB from /home/pc-application/WINAPPS/VMware/offline_bundle_persist_perms.zip copy it on the ESXi server. Note that only the offline version of the VIB is available.

Connect to the server shell and run:

esxcli software vib install -f -d /offline_bundle_persist_perms.zip

That's all you have to do. From now on, the AD permissions will persist.

The tools have been tailored for CSIM environment, AD permissions are in the form SMB4\user.

update ESXi

Third parties VIB that are not officially signed by VMware prevent ESXi update. And this is diagnosed quite late in the update process, so precious time can be lost. Note that this is true also for ghettoVCB.

Hence, the VIB must be removed before any ESXi update:

1. login on the server and copy the file /etc/persist_perms/permissions to some place safe across reboots, for example /vmfs/volumes/datastore1/;
2. remove the VIB (see the command below) without removing AD permissions;
3. proceed to ESXi update;
4. reinstall the VIB (see above) and restore the permissions file.

Remove the last AD permission

When the script detects that there is no more AD permissions, it will automatically restore them from the back-up. So when you remove the last permission, the script will try and restore it.

To prevent that you can remove the VIB before removing the last AD permission:

esxcli software vib remove -f -n persist_perms

Alternatively, you can stop cron:

kill `cat /var/run/crond.pid`

remove the permission, update the file /etc/persist_perms/permissions:

echo  \#\!/bin/sh >/etc/persist_perms/permissions

and restart cron:

/usr/lib/vmware/busybox/bin/busybox crond

What is installed by the VIB

The VIB installs the following files:

• /opt/persist_perms/test-perms.sh the script that checks that the AD permissions are present, makes a back-up and restores the permissions;
• /opt/persist_perms/perm.awk and /opt/persist_perms/role.awk the scripts that are used in generating the back-up;
• /etc/persist_perms/permissions the list of AD permissions, this is a shell script that contains all the commands needed to restore the permissions;
• /etc/rc.local.d/999.persist_perms.sh the script in charge of setting the crontab when ESXi boots (at boot, the crontab consists of a fixed list of tasks, call to our test needs to be inserted during every boot).

Revision History

1.0.0-0.0.2

Sept. 2021, correct a bug in the .awk files that did not take into account role-names that could have blank spaces in them. A matter of handeling quoted strings.

1.0.0-0.0.1

Aug. 2019, initial version,

How to construct a VIB

VMWare used to offer the tool vibauthor to package a VIB. This tool is not available on VMWare website, but it has been packaged in a docker container by William Lam. You can follow the instructions on William's page or follow the instructions bellow:

1. install docker and docker.io on an Ubuntu machine, follow the current documentation;
2. pull vibauthor docker:

   sudo docker pull lamw/vibauthor
   

3. run the container:

   docker run --rm -it lamw/vibauthor
   

4. you should be able to access the command vibauthor

Preparing the files to create a VIB has been inspired from the page by William about Creating Custom VIBs For ESXi 5.0 & 5.1 with VIB Author Fling. All the files needed for this VIB can be downloaded from /home/pc-application/WINAPPS/VMware/persist_perms-1.2.tgz.

After extracting the tar file, you should see the following file hierarchy:

$ tree stage
stage
├── descriptor.xml
└── payloads
    └── payload1
        ├── etc
        │   ├── persist_perms
        │   │   └── permissions
        │   └── rc.local.d
        │       └── 999.persist_perms.sh
        └── opt
            └── persist_perms
                ├── perm.awk
                ├── role.awk
                └── test-perms.sh

7 directories, 6 files

Note that:

• In stage/payloads/payload1 you see the files that will be installed by the VIB;
• The file descriptor.xml describes the VIB and how it should be installed;
• The exact hierarchy of stage, payloads, payload1, descriptor.xml seems to be defined by vibauthor and should not be modified;
• The file permissions must have the sticky bit set, that allows the script to modify the file and ensure that the modifications persist across reboot.
• The file 999.persist_perms.sh is executed by the VIB installation while other files are simply copied to their destinations. My understanding is that because that file is located in /etc/rc.local.d is is being executed after installation. This serves our purpose as it ensures the cron file is updated immediately and not only after a reboot. So the back-up script is launched shortly and a back-up is made on the next cron execution.

Once the stage hierarchy has been finalized, the VIB may be created with the command:

vibauthor -C -t stage -f -O offline_bundle_persist_perms.zip