Wednesday, February 12, 2014

WARNING: Restarting the Agent of the Control Domain will stop this Guest (it might get even fully destroyed) !!!

Yikes, not the type of warning you want to see after upgrading your Oracle Enterprise Manager - Ops Center to Update 4! I received this error after upgrading my EC/Proxy and Agents to 12c U4, along with OCDoctor 4.26. Here is the error in its entirety (unique identifiers stripped for privacy):

Checking guest [LDOM_NAME] ...

ERROR: Critical issue detected !!!

The following config file does not list this Control Domain as current owner of guest [LDOM_NAME]:



#Store the owner during create itself. DO NOT HAND EDIT.

#Thu Aug 15 09:51:57 EST 2013

WARNING: Restarting the Agent of the Control Domain will stop this Guest (it might get even fully destroyed) !!!

This issue needs to get corrected immediately (Bug 16935439) 

There is a related config file '' which is listing a wrong owner as well:



#The future owner file. DO NOT HAND EDIT.

#Thu Aug 15 09:51:58 EST 2013

This file should get corrected as well!


Turns out, there are two files that live within your LDom Meta directory for each logical domain, that dictate which Control Domain currently owns the LDom, and which Control Domain will be the "future owner".

Source code comments, define the future owner as:

39 /* Key under which the owner of this virtserver shall be stored
40 on the device rather than the database.
41 This key shall be having the latest and current owner always
42 and any differences in the ownership shall result in the
43 destruction of the instantiated virtserver runtime objects on
44 any given xVM Server system.

So basically, if there's an internal decision to make the CDom owner of the LDom's configuration (not the EC's and it's database), it will go to the CDom, whose key is listed in this file.

Knowing all this, we need to backup these two files, and modify them to reflect the current CDom Owner, and that CDom's "", or key. To get the current CDom's key, I listed all .properties files for each LDom to find one that was set correctly. Something like the following should list everything:

# find /var/mnt/virtlibs/XXXXXXXXX/ -name | xargs cat

Modify these files and run your OCDoctor --troubleshoot again, and the issue should now be gone! The bug that the error quotes (16935439) is not public, but I did manage to get the following information from Oracle:

BUG 16935439 ---> Bug 17995697 - WRONG METADATA AFTER LDOM MIGRATION FAILURE (Related bug)

So I can only assume a failed migration has caused this condition. Hopefully it's resolved in Update 4, but I think I'll keep an eye on it following any future failed migration tasks.

Friday, February 7, 2014

Updating OCDoctor on Guest Domains Lacking Internet Connectivity

I've recently been upgrading Oracle Enterprise Manager Ops Center from Update 2, to Update 4. One of my many post-checks was to run an OCDoctor on every system, be it LDom, CDom, EC or Proxy Controller. Our guest domains can't reach the internet, and so I needed a quick way to push out and install the latest OCDoctor. The following line of code enumerates the list of LDoms (dropping the last character to match the hostname), copy over the bundle, and install.

# for x in $(ldm list | egrep -v NAME\|primary | awk '{print $1}' | sed 's/.$//'); do echo ${x}:; scp /var/tmp/ root@${x}:/var/opt/sun/xvm/; ssh root@${x} "cd /var/opt/sun/xvm; mv OCDoctor OCDoctor-20140207; unzip"; done