vCloud Director 5.1.2 Removing Org VDC Networks

I ran into a little issue the other day configuring vcloud director. I wanted to remove an organization VDC completely. Of course to do this you must disable the VDC and remove all of it's resources (networks, templates, and vApps). Upon trying to remove the networks I kept getting the error

Entity xxx.xxx.xxx.xxx cannot be deleted, because it is in use.

This error occurs even though there is absolutely nothing using the network; no templates, VMs, vApps, ....nothing. Our networks were of the direct connect type, although this may also occur with an Edge Gateway network. We didn't want to remove the upper level networks from the entire cloud. The system seems to want to hold on to these networks once they were added for the first time. It seems that we may not be able to remove these networks from the system, but we were able to move them to a new organization VDC. We did this directly on the vcloud database.

/******
First lets see what is in the table
******/
SELECT TOP 1000 [id]
      ,[vdc_id]
      ,[lr_type]
      ,[name]
FROM [vcloud].[dbo].[vdc_logical_resource]

Take a look at what you have in the table. The records you care about have a lr_type of NETWORK.

Once you have the IDs from the table you can update the records accordingly.

In the change query below the first parameter (vdc_id) is the organization you want the network to move to. You can usually find the proper id by noting the vcd_id of the record that has the organization name in the 'name' field. The second parameter (id) is the network you want to change. In other words, you are going to set the network's organization id field.

UPDATE vcloud.dbo.vdc_logical_resource
SET vdc_id = 0xabunchofUUIDnumbersfortheorganization
WHERE id = 0xabunchofUUIDnumbersforthenetwork;

Another unique issue we were seeing occurs when we try to disable sharing this network with other VDCs in the organization. If we try to uncheck the box in it's properties we get the following error in vCloud Director:

Index: 0, Size: 0

Not sure what this one is about. Seems it could be an out of bounds exception with the program accessing a list.

Using a Script to Add a Network Printer with Custom Drivers

You can use a simple .vbs file to add a printer to a Windows 7 computer. This script will create a new TCP/IP local port for use with the printer.  Before you begin you will want to get ahold of the drivers that will allow this printer to be used on your target system (32 or 64bit).

Often the drivers are gathered in a folder of the download the manufacturer provides. You may need to sift through some of the files to find the correct .inf file for your printer. If you view the contents of the .inf file you may find a number of printers listed. It's important to note how your printer model is designated. You will need to specify the exact text of the printer listing in the /m switch of the printui.dll command.

One way to get the exact name of the printer you need to specifiy is to start the installation on a test system using the "Add Printer" wizard in "Devices and Printers".  Once you get to the "Install the printer driver" step click on "Have Disk..."  Browse to the .inf file and hit OK.  You will be presented with a list of printers. Using the exact text of one of options presented in the selection box as your printer listing text will achieve the desired result. You can cancel out of the wizard once you have the text copied to your script.



Let's assume the following scenerio:
- We have a networked Dell 5350 at network address 192.168.1.70
- The printer is in room 207 of your building
- You have pushed the drivers for the printer to a local folder (C:\Drivers\Dell 5350dn) on your target system.
- You want the printer name to show up as "Room 207 - Dell Laser" 
- The local port name will match the IP address.


Save the following into a text file and rename the file extention to .vbs

Set WSHNetwork = WScript.CreateObject("WScript.Network")
set shell = WScript.CreateObject( "WScript.Shell" )
CompName = shell.ExpandEnvironmentStrings("%COMPUTERNAME%")
Set objWMIService = GetObject("winmgmts:\\" & CompName & "\root\cimv2")

Set objNewPort = objWMIService.Get("Win32_TCPIPPrinterPort").SpawnInstance_
Set oShell = WScript.CreateObject("WScript.shell")
Set objPrinter = objWMIService.Get("Win32_Printer").SpawnInstance_

sub createPort (name, ip)
    objNewPort.Name = name
    objNewPort.Protocol = 1
    objNewPort.HostAddress = ip
    objNewPort.SNMPEnabled = False
    objNewPort.Put_
end sub


'-- Call the create port function with the address and port name parameters
 
createPort "192.168.1.70", "192.168.1.70"


oShell.run "cmd /K rundll32 printui.dll,PrintUIEntry /if /f ""C:\Drivers\Dell 5350dn\DKACLC40.inf"" /n ""Room 207 - Dell Laser"" /m ""Dell 5350dn Laser Printer"" /r 192.168.1.70 /b ""Room 207 - Dell Laser"" /q"

Set oShell = Nothing




Check your  "Devices and Printers" window to see if the new printer has appeared.

If you are finding that something is not working but you see that the port was created you can try just the following command on the target system.



rundll32 printui.dll,PrintUIEntry /if /f "C:\Drivers\Dell 5350dn\DKACLC40.inf" /n "Room 207 - Dell Laser" /m "Dell 5350dn Laser Printer" /r 192.168.1.70 /b "Room 207 - Dell Laser"

If things aren't working with just the command you may get an error in the form of 0x00000*.  This can often indicate that the driver file specified can't be found or is invalid.

Password-less SSH Connections & What Can Go Wrong

So, you would like to jump from server to sever over SSH without being prompted for a password each time. Or perhaps you have a script or application that needs to access information on a remote server via a SSH tunnel. Using SSH shared key authentication makes this possible.

Shared key authentication is easy enough to set up, but there are a couple of pitfalls that can have you pulling out your hair if things don't work after you have followed the instructions exactly.

Let's consider the following two server scenario:

Server 1:  AIR
Server 2:  WATER

Suppose there is a user "bill" on both servers. Bill usually works on AIR and frequently needs to perform tasks on WATER, he would like to use shared key authentication.

AIR has a /etc/hosts file entry for WATER and vice versa. 


First Bill logs onto AIR and issues the following command
ssh-keygen -t rsa

Now might be a good time to mention that there are other options you could use the ssh-keygen command with to create different types of keys with a variety of key lengths. For this example we'll just use the default rsa key type.

When you hit enter after this command you will see:
Generating public/private rsa key pair.
Enter file in which to save the key (/home/bill/.ssh/id_rsa):

Accept this default path and hit enter.

You'll be prompted for a pass phrase, once to create it then again to verify, leave it blank and hit enter each time.

Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/bill/.ssh/id_rsa.
Your public key has been saved in /home/bill/.ssh/id_rsa.pub.
The key fingerprint is:
XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX bill@air

For this next step to work there should be at least a ~/.ssh/ directory on the WATER server under bill's account. An easy way to achieve this is to perform the same key generation sequence on WATER that we did on AIR. Once this is done we'll have .ssh directories on each of the servers. The next step is to add the public key of the server we are coming from to the authorized_keys file of the server we are going to.

When we are on AIR we can issue the following command:
cat ~/.ssh/id_rsa.pub | ssh bill@WATER "cat - >> ~/.ssh/authorized_keys"

You might be prompted to store the remote server's RSA info to the local system if this is the first time you are connecting. Bill will also be prompted for his WATER password this time.

This command will load the local (AIR) public key into the remote (WATER) server's authorized_keys file. From this point forward bill's SSH authentication from AIR to WATER can now be handled through a key exchange.

Now if bill is on AIR and issues a simple SSH command:
ssh WATER

He will be granted a direct password-less connection. This is great if bill wants to establish a script using scp to copy items from AIR to WATER, the script won't prompt for a password.


Congratulations!  Oh.. wait.. it didn't work ??


Now.. what to check if things don't work.
There are a few not so obvious things that can leave you pulling your hair out.

Improper permissions are usually the most obscure type of issue.

• Permissions on your home directory. The user you are remoting as should be able to read the remote user's home directory. Think about our example above, Bill may not have the same user number on both servers. If the numbers are different then Bill looks like a different person on each server OS. You can achieve this via a group or public permission depending on your security needs.

• Permissions on the authorized_keys file should be set to:
  -rw-r--r--   in other words:  644
  Also, you (bill in this case) should own these files, respectively on each server.

• If you have copied keys manually by copy and pasting or FTP, you may have broken the key, check for proper file encoding and stray characters or carriage returns.

• If you are setting this up on a root account you should check the /etc/ssh/sshd_config file to ensure the PermitRootLogin without-password option is enabled.

• If you have your home directory NFS 4 mounted on the remote server you'll need to use something like rcp.idmapd to ensure the user and group ownership information is associated correctly on the remote system. The default is usually to place the nobody user and group as the owner of the files. A quick fix is to add the specify vers=3 in the in the fstab file mount option.

Hopefully these tips get you up and running, if you find another issue please tell us about it.

Sendmail 550 Access denied with 127.0.0.1 Relay

Ran into this little sendmail issue today on a CentOS box.

My Linux server was configured to relay mail to a main corporate exchange server. The M4 configuration already had a proper SMART_HOST configured.

I was doing a simple test with sendmail and was getting an access denied error like this:


[root@server]# sendmail -v root
ppp
ppp
.
root... Connecting to [127.0.0.1] via relay...
220 server.com ESMTP Sendmail 8.13.8/8.13.8; Mon, 15 Oct 2012 13:41:19 -0400
>>> EHLO server.com
250-server.com Hello server.com [127.0.0.1], pleased to meet you
250 ENHANCEDSTATUSCODES
>>> MAIL From:<user@server.com>
550 5.0.0 Access denied
user... Using cached ESMTP connection to [127.0.0.1] via relay...
>>> RSET
250 2.0.0 Reset state
>>> MAIL From:<>
550 5.0.0 Access denied
postmaster... Using cached ESMTP connection to [127.0.0.1] via relay...
>>> RSET
250 2.0.0 Reset state
>>> MAIL From:<>
550 5.0.0 Access denied
Closing connection to [127.0.0.1]
>>> QUIT
221 2.0.0 server.com closing connection


This shows us that the first hop in the relay process is to itself.  This was where it was failing.

The fix was to add the following line to /etc/hosts.allow file:
sendmail: ALL :allow

This cleared up the issue and got mail flowing once again. This works since the hosts entries also apply to traffic for system daemons.

Viewing communication using the sendmail -v command is one way to view what's happening, however in my case once things were working again with the local relay I needed to look at the /var/log/maillog file to see that messages were making the next hop out to the corporate mail server. I could then see the relay=corporate.mail.server.com.   in the logs with a status of "Sent"

Slow SSH Connections

If you are SSH'ing to a server and having to wait for the user name and/or password prompt the issue could be more than just a slow connection.  There are a couple of common things to check if you are having to wait anywhere from 10 seconds to over 1 minute to get your session established.

1. DNS reverse mapping not resolving:


The SSH server may be trying ot perform a reverse lookup on the client trying to connect. If DNS doesn't response quickly, either with the host name or a 'not found' reply, then this attempt will continue until it times out. Modify your /etc/ssh/sshd_config to:

UseDNS no

2. SSH may be trying too many authentication types:

SSH may be configured to try PAM, GSSAPI, or some flavor of shared key authentication. You can change the setting:

GSSAPIAuthentication no

If you are using PuTTY you may also want to check the settings there.  If you are trying to connect using GSSAPI in putty but the server isn't set to use it then you will create a delay while it attempts this.  One tell tale sign that GSSAPI is enabled on the client side and is failing is getting an "Access denied" message at the prompt (illustrated below) yet authentication eventually succeeds.

login as: user
Access denied
user@someserver.com's password:
Last login: Tue Oct 1 01:23:40 2012 from localhost.com


Uncheck the "Attempt GSSAPI authentication" box and see if this speeds things up.

Fix Unstable Fedora Running on Cruical M4 SSD

Worked through an interesting experience with a SSD recently.  Apparently the Cruical M4 2.5" SDD has a stability issue that occurs after 5000 hours of actual use.

Crucial admitted this issue here:
http://forum.crucial.com/t5/Solid-State-Drives-SSD/BSOD-Crucial-M4/td-p/79098

I saw this manifest itself in the form of an unstable Fedora system running on a 128GB version of the M4. When rebooted the system would appear stable until an hour passed at which point it steadily degraded. There were files that were coming up not found and command execution was giving an "input output error", which is usually indicative of the system not finding the command.

Using smartd we noticed that the "Power_On_Hours" RAW_VALUE was around 8000. The command we used was:

smartctl -a /dev/sda

Just a side note but some other notable info in this output to indicate a possible drive failure are the "Reallocated_Sector_Ct" and the "Seek_Error_Rate".

The fix in this case was to upgrade the firmware. We did this and the system returned to a stable state. Crucial provides an ISO image you can boot to to perform the upgrade, this is a much better option if you are using it as your boot drive.

Getting the pesky browser plug-in to work for VMware Lab Manager

You've probably googled your eyes out trying to find an update, patch, or workaround to get the Lab Manger plugin to work with the browser or OS of your choice.  In some cases it's a lost cause, VMware has dropped the product and moved toward vCloud Director.  However, on almost all platforms there is a little hope you can get it / keep it working but you may need to make some compromises.

Below are some of the procedures we have found to alleviate some of the plug-in issues.  We first review the issues on Windows then Linux.

  

A note about MAC based Issues
At this time, the plugin is not natively supported on any MAC platform. You should consider running the plugin in IE on a Windows VM installed on your MAC using VMware Fusion. 

A note about the Plugin Files

For a few of these fixes you should grab the plugin source files, these are sitting on the lab manager server. You will need them if you run through a manual installation.

WINDOWS BASED SYSTEMS 
In some cases the initial console browser plugin installation can be a bit troublesome. The problem can usually be overcome by correcting browser settings and performing a clean administrative installation. Outlined below is the easier process first (resetting the browser and installing as an admin) followed by a manual uninstall and clean installation.

Default IE Settings and Install as Admin
Even though the plugin seems to be installed you may experience a browser crash or simply get a blank box where you expect to see the console window. Here are some fixes to try.

Run IE administratively:
Right click on the IE shortcut in your start menu and select “run as administrator”
Go to: Tools → Internet Options → Advanced → Click on the “Reset” button.

Restart the browser as an administrator and try to access the console once again.

Add your lab manager website to the trusted sites list:
Go to Tools → Internet Options → Security →Trusted Sites Add the site

Be sure to enable the QuickMksAxCtl Class:
With IE closed, go to Control Panel → Internet Options → Programs Hit the “Manage Add-ons” button use the drop down menu to show downloaded controls and double click on the QuickMksCtl Class add-on. Select the button to allow it to run on all websites. Close this window then enable to add-on.

Manually uninstall the VMware console browser plugin from Internet Explorer

Run the following command as an administrator:
regsvr32 /s /u "C:\Program Files\Internet Explorer\PLUGINS\quickMksAx.dll"

Delete the following files:
C:\Program Files\Internet Explorer\PLUGINS\msvcr71.dll
C:\Program Files\Internet Explorer\PLUGINS\quickmksax.inf
C:\Program Files\Internet Explorer\PLUGINS\ssleay32.dll
C:\Program Files\Internet Explorer\PLUGINS\vmware-remotemks.exe

Manual VMware console browser plugin installation for Internet ExplorerBefore you begin the manual installation remove the plugin using the manual uninstallation procedure.

Extract the contents of the .cab file to
C:\Program Files\Internet Explorer\PLUGINS\ 

Depending on your configuration you might run into system permission issues if you expand the cabinet file and try to directly copy to this folder using the GUI. If you experience this issue you may try opening the command prompt as an administrator and using the ‘expand’ command.

expand -F:* C:\ClientSoftware\VMware-mks.cab “C:\Program Files\Internet Explorer\PLUGINS”

To register the plugin, run the following command as an administrator:
regsvr32 /s "C:\Program Files\Internet Explorer\PLUGINS\quickMksAx.dll"

If none of this seem to help:
Check the settings outlined in http://kb.vmware.com/kb/10233/

These VMware KB articles outline some of the above procedures:
- Running as admin
- Adding security exceptions
- Failed installation, manual intervention

Manual Installation for Firefox

Note: At this time the console plugin is not compatible with FireFox 4.
You may find 3.6 still available here: ftp://ftp.mozilla.org/pub/mozilla.org/firefox/releases/3.6.28
Windows Linux

Use the plugin zip file, unzip the file's contents into:

%ProgramFiles%\Mozilla Firefox\plugins

Also, copy ssleay32.dll and libeay32.dll to
%ProgramFiles%\Mozilla Firefox

LINUX BASED SYSTEMS

Manual Installation for Firefox 3

Note: At this time the console plugin is not compatible with FireFox 4.
You may find 3.6 still available here: http://www.mozilla.com/en-US/firefox/all-older.html

Download the zip file using the links above

Unzip the file's contents into
~/.mozilla/plugins

Other Linux Issues

Using Mozilla Firefox on Linux to access the Lab Manager Web console can cause problems with the console plugin.

There are a number of possible issues and solutions:

In Firefox on Linux, if error messages appear when you try to use a virtual machine's console, you might not have all required libraries installed.
For RHEL 64bit, you need to install compat-libstdc+-33-3.2.3 on the setup (ideally using yum, which also installs libstdc+.so.5), and for Ubuntu, go to http://packages.debian.org/stable/base/libstdc++5 and install the missing library.

If Firefox reports that it could not install the plugin (Cancelled -227), create a directory named "plugins" in $HOME/.mozilla on the client computer. Log in to Lab Manager and install the plugin. Restart Firefox.

If Firefox reports LoadPlugin: failed to initialize shared library /root/.mozilla/plugins/libmks.so, create a soft link to libexpat.so.

Lab Manager Web console page shows an empty box in Mozilla Firefox 3.6 on Linux

Some versions in Firefox 3.6 series strip executable permissions on files that are extracted from the XPI plugin binary (see http://blog.mozilla.com/addons/2010/01/22/broken-executables-in-extensions-in-firefox-3-6).

The console plugin does not load correctly and the console page appears blank. To resolve the issue, browse to the console plugin installation folder at
"/<Firefox_profile_folder>/extensions/VMwareMKSNPRTPlugin@vmware.com/plugins/" and run the command "chmod 755 *" to manually enable permissions on the files of that folder.

You may have luck on Ubuntu using an older version of firefox. (Links to older versions)

You may find that the solution is to use Firefox 3.5 or below as 3.6 or higher doesn't work with the VMware remote console plugin - since this is already not getting security updates, it's best to install it separately to the main Firefox, and use a new profile. To avoid messing up any Ubuntu version of firefox, just untar the Firefox 3.5 tar.gz under something like /opt/firefox-3.5

Here's a shell script that invokes this Firefox with the right profile, even if you have a more recent Firefox running (via the -no-remote):

#!/bin/sh    

# Run Firefox 3.5, for VMware 2.0 only

prog=/opt/firefox-3.5/firefox/firefox 

exec $prog -no-remote -P vmware-FF3.5

After you are done with the console, it's best to close the Firefox 3.5 instance, otherwise links clicked in other applications may open in the 3.5 instance.

Converting your VMs and Template Disks to Thin

The tool we mainly make use of is vmkfstools to convert the thick disks to thin.

We'll be using the ESX console to perform this operation.

First, power off the VM.

Use vCenter to find the host the VM is on and the associated datastore location.

Log into the ESX host's remote console as root user.

Convert the original thick .vmdk into a thin one

vmkfstools -i THEVMNAME.vmdk -d thin thinTHEVMNAME.vmdk

Move the original .vmdk flat file out of the way.

vmkfstools -E THEVMNAME.vmdk orig-THEVMNAME.vmdk

Move new thin .vmdk flat file into place

vmkfstools -E thinTHEVMNAME.vmdk THEVMNAME.vmdk

In vCenter remove the VM from inventory and add it back again, otherwise the disk may not show as 'Thin'.

Ensure that the VM boots up and all drives are available.

If things test out ok you can remove the original disk file, using vmkfstools is the proper way so the datastore knows it's gone.

vmkfstools -U orig-THEVMNAME.vmdk

Setup CentOS / Redhat as a VNC server for all users

Lets suppose you have a Linux server you would like the users to access via VNC. On a basic setup a user may need to SSH into the server and find out what display socket they will be connecting to. This is overhead that can be avoided. Using the method below a user can connect directly to the server using a VNC client as their first and only step.

Open the ports:

vi /etc/sysconfig/iptables

Add the following lines

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 5900 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 5900 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 5901 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 5901 -j ACCEPT

Restart the firewall service:

service iptables restart

Be sure vnc-server and other components are installed:

yum –ty install vnc-server xinetd

Also grab vnc-ltsp-conf and install it using:
rpm –Uvh vnc-ltsp-config-4.0-4.el5.noarch.rpm

Start the services and configure to start on reboot:

service vncserver start

Check the service configuration in webmin or using system-config-services or using these commands:

/sbin/chkconfig xinetd on
/sbin/chkconfig vncts on
/sbin/chkconfig vncserver on
/sbin/service xinetd restart

Configuration:

As root edit the file "/etc/gdm/custom.conf"

To the next blank line below the "[security]" section add "DisallowTCP=false"

To the next blank line below the "[xdmcp]" section add "Enable=true"

Make sure you are in a position to either run "gdm-restart" for default Gnome installs or just reboot the CentOS box.

Valid users should now be able to use the VNC client to connect to the server.