Wednesday, February 12, 2014

WARNING: Restarting the Agent of the Control Domain will stop this Guest (it might get even fully destroyed) !!!

Yikes, not the type of warning you want to see after upgrading your Oracle Enterprise Manager - Ops Center to Update 4! I received this error after upgrading my EC/Proxy and Agents to 12c U4, along with OCDoctor 4.26. Here is the error in its entirety (unique identifiers stripped for privacy):

---------------
Checking guest [LDOM_NAME] ...


ERROR: Critical issue detected !!!


The following config file does not list this Control Domain as current owner of guest [LDOM_NAME]:


File: /var/mnt/virtlibs/XXXXXXXXX/XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXX/guest-domain.properties


Content:


#Store the owner during create itself. DO NOT HAND EDIT.

#Thu Aug 15 09:51:57 EST 2013
owner.hostname=[INCORRECT_CDOM_NAME]
owner.vsc.id=[INCORRECT_CDOM_IDENTIFIER]

WARNING: Restarting the Agent of the Control Domain will stop this Guest (it might get even fully destroyed) !!!


This issue needs to get corrected immediately (Bug 16935439) 


There is a related config file 'guest-domain-FO.properties' which is listing a wrong owner as well:


File: /var/mnt/virtlibs/XXXXXXXXX/XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXX/guest-domain-FO.properties


Content:


#The future owner file. DO NOT HAND EDIT.

#Thu Aug 15 09:51:58 EST 2013
previous.owner.processing=false
owner.vsc.id=[INCORRECT_CDOM_IDENTIFIER]

This file should get corrected as well!

----------------------------------------

Turns out, there are two files that live within your LDom Meta directory for each logical domain, that dictate which Control Domain currently owns the LDom, and which Control Domain will be the "future owner".

Source code comments, define the future owner as:


39 /* Key under which the owner of this virtserver shall be stored
40 on the device rather than the database.
41 This key shall be having the latest and current owner always
42 and any differences in the ownership shall result in the
43 destruction of the instantiated virtserver runtime objects on
44 any given xVM Server system.


So basically, if there's an internal decision to make the CDom owner of the LDom's configuration (not the EC's and it's database), it will go to the CDom, whose key is listed in this guest-domain-FO.properties file.

Knowing all this, we need to backup these two files, and modify them to reflect the current CDom Owner, and that CDom's "vsc.id", or key. To get the current CDom's key, I listed all .properties files for each LDom to find one that was set correctly. Something like the following should list everything:

# find /var/mnt/virtlibs/XXXXXXXXX/ -name guest-domain-FO.properties | xargs cat

Modify these files and run your OCDoctor --troubleshoot again, and the issue should now be gone! The bug that the error quotes (16935439) is not public, but I did manage to get the following information from Oracle:

BUG 16935439 ---> Bug 17995697 - WRONG METADATA AFTER LDOM MIGRATION FAILURE (Related bug)

So I can only assume a failed migration has caused this condition. Hopefully it's resolved in Update 4, but I think I'll keep an eye on it following any future failed migration tasks.