Host Disconnects from vCenter in a vCloud Director Environment

So I ran into an issue the other day regarding host disconnects within vCenter in a vCD environment.  SSH’d into the host and noticed right off the bat that vmkernel.log was showing the following error:

WARNING: VisorFSObj: 893: Cannot create file tmp:/current.png5557 for process vmx-mks:Cell Profiler x64 (b9d6aa64-d789-4954-bde9-f2875406c159) because the visorfs inode table is full. 2012-10-05T12:38:49.808Z cpu27:6014573)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process esxcfg-advcfg because the visorfs inode table is full.

And in hostd.log I saw this:

WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process hostd because the visorfs inode table is full.

It is obvious through those errors what the issue is.  Once you come to that conclusion you will need to do some checking around.  The first thing I did was look in /tmp.  Low and behold my ls -l command showed me an ever scrolling list of current.png[1-9]*.  Thousands upon thousands of these were there.  For those of you not aware of what these are, they are the screencaps you would see in the vCloud Director Portal of your VMs.  The first thing one would think to try is to remove all of these with “rm -f current.png*”.  WHOOPS!  That won’t work in this case because the “argument list is too long”.  The way to get around this error is to do one of the following:

cd /tmp
for i in `seq 1 9`;do rm -f current.png[$i]*;done

or

cd /tmp
find . | grep current | xargs rm -f

This will remove the current.png files, releasing the inodes.  If you watch the logs (hostd, vmkernel) you will see that everything starts to process again.  After a few minutes it will reconnect to vCenter again.  Enjoy!

UPDATE: After I did some research on this I found that there is a KB Article regarding this (KB #2031839).