oracle.sysman.emSDK.emd.fetchlet.FetchletException

Hi Friends,

While experimenting with Oracle Enterprise Manager Grid Control alongside SOA Management Pack Plus, i was getting following Metric collection errors for BPEL Process Manager Partner Link Metrics.

Issue:
The error description was:
"oracle.sysman.emSDK.emd.fetchlet.FetchletException: java.rmi.UnmarshalException: Error deserializing return-value: java.io.InvalidClassException: javax.xml.namespace.QName; local class incompatible: stream classdesc serialVersionUID = -916876369326528164, local class serialVersionUID = -9120448754896609940"

Reason:
Well, if you carefully observe the error description tries to say, "local class incompatible"!
This generally means a JDK Version conflict. If you research further, you will easily see that this happens due to a JDK Version conflict between your Grid Control Agent & the BPEL Process JDK.

You can verify the JDK Version using: "java -version"

Solution:
The only solution to this issue is to upgrade your BPEL Process Target JDK version. You cannot downgrade your Agent's JDK Version.
Please be careful while upgrading your BPEL Process Target JDK as it might destabilize your system.

Please ensure that the new JDK is compatible with your BPEL Process version and you know correct steps to upgrade the same.
After you upgrade the JDK, restart the Grid Control Agent and force an Metric Collection upload as follows:

prompt:/>emctl stop agent - Windows
bash-$ ./emctl stop agent - Unix/Linux

prompt:/>emctl start agent - Windows
bash-$ ./emctl start agent - Unix/Linux

prompt:/>emctl upload agent - Windows
bash-$ ./emctl upload agent - Unix/Linux

Keywords:
OEM, OMS, Enterprise Manager, Grid Control, Grid Control Agent, OMS Agent, OBIEE Management Pack, Siebel Management Pack, SOA Management Pack, AGENT_TZ_MISMATCH, Timezone, 10.2.0.5, PSU 3, JDK Mismatch, BPEL Process, etc

Oracle Enterprise Manager - AGENT_TZ_MISMATCH

Hi Friends,

Last entire week i was busy playing with Oracle Enterprise Manager Grid Control alongside Siebel, OBIEE & SOA Management Packs.

Believe me, its not a piece of cake!
Though PSU 3 on top of 10.2.0.5 provides great stability, still few nagging issues around Metric collections for different target types & other areas keep affecting us.

One of the most annoying issue that took me long time to figure out was my Grid Control Agent, simply failing to startup :-(

It was working perfectly fine earlier, but didn't know when it stopped working :-(
After much due diligence in the logs, the emagent.nohup had following dumps:
Issue:
----- Wed Jun 23 11:32:51 2010::Agent Launched with PID 3852 at time Wed Jun 23 11:32:51 2010 -----
(pid=3852): starting emagent version 10.2.0.5.0
(pid=3852): emagent started successfully
OMS decided to shutdown the agent because of the following reason sent from OMS: AGENT_TZ_MISMATCH(pid=3852): emagent now exiting abnormally - initialization failure. Consult '.trc' and '.log' files.
----- Wed Jun 23 11:34:39 2010::Pid 3852 not found. reaped=3852, processStatus=14080 -----
----- Wed Jun 23 11:34:40 2010::Checking status of EMAgent : 3852 -----
----- Wed Jun 23 11:34:40 2010::EMAgent exited at Wed Jun 23 11:34:40 2010 with return value 55. -----
----- Wed Jun 23 11:34:40 2010::EMAgent has exited due to initialization failure. -----
----- Wed Jun 23 11:34:40 2010::Stopping other components. -----
----- Wed Jun 23 11:34:40 2010::Commiting Process death. -----
----- Wed Jun 23 11:34:40 2010::Exiting watchdog loop

I was scratching my head, what does AGENT_TZ_MISMATCH means?? Googled it around no luck! I was shocked & surprised that there was no results in Google!

Reason:
Felt like i have hit some kind of jackpot ;-)
After much research, i came to know that such kind of issue might happen when your Agent's host has its Time Zone changed.

Now, first of all, i didnt accept it! What does Agent's Timezone has to do with its startup? It should automatically accept the new time...

But then, it was clear that Agent did startup correctly, but when it use to communicate with my OMS Server where the Agent's Timezone is recorded, the changed TimeZone hits a mismatch here!!!

Solution:
Well, in order to resolve this, the solution is simple!
From a command prompt or terminal go to %Agent_Home%/bin directory.
Execute below command:

prompt:/>emctl resetTZ agent - Windows
bash-$./emctl resetTZ agent - Unix/Linux

This will update your Agent's emd.properties with the changed TimeZone. But wait this does not complete the changes. You will need to execute a procedure on your Enterprise Manager Repository Database to reflect the changes in the timezone of the Agent.

So connect to your OMS Repository Database (generally 'emrep') using the repository owner (generally 'sysman')

SQL>exec mgmt_target.set_agent_tzrgn(<agent_name>, <new_tz_rgn>);

Well, the exact statement for this procedure is automatically generated at the end of 'emctl resetTZ agent' command :-)

Well, this completes your Agent's TimeZone change. Now to try to start the agent using below command:

prompt:/>emctl start agent - Windows
bash-$./emctl start agent - Unix/Linux

I am sure it will startup correctly! ;-)

Keywords:
OEM, OMS, Enterprise Manager, Grid Control, Grid Control Agent, OMS Agent, OBIEE Management Pack, Siebel Management Pack, SOA Management Pack, AGENT_TZ_MISMATCH, Timezone, 10.2.0.5, PSU 3, etc

SEVERE: EM is not configured for this database

Hi Friends,

I was doing this Oracle 10gR1 database upgrade to Oracle 10gR2 using DBUA (Database Upgrade Assistant) when my upgrade failed due to this error:


[Thread-54] [10:25:54:325] [BasicStep.configureSettings:304]  messageHandler being set=oracle.sysman.assistants.util.UIMessageHandler@6014f7
[Thread-54] [10:25:54:325] [SummarizableStep.setUpgradeStatus:620]  **++Set Upgrade status to:=1
oracle.sysman.assistants.util.step.StepExecutionException: Enterprise manager configuration failed due to the following error -
EM is not configured for this database. No EM-specific actions can be performed.
Refer to the log file at C:\OracleHomes\db10gr2\cfgtoollogs\dbua\emrep\upgrade1\emConfig.log for more details.
You can retry configuring this database with Enterprise Manager later by manually running C:\OracleHomes\db10gr2\bin\emca script.
    at oracle.sysman.assistants.util.em.EMConfigStep.executeImpl(EMConfigStep.java:141)
    at oracle.sysman.assistants.util.step.BasicStep.execute(BasicStep.java:210)
    at oracle.sysman.assistants.util.step.BasicStep.callStep(BasicStep.java:251)
    at oracle.sysman.assistants.dbma.backend.EMConfigStep.executeStepImpl(EMConfigStep.java:101)
    at oracle.sysman.assistants.dbma.backend.SummarizableStep.executeImpl(SummarizableStep.java:174)
    at oracle.sysman.assistants.util.step.BasicStep.execute(BasicStep.java:210)
    at oracle.sysman.assistants.util.step.Step.execute(Step.java:140)
    at oracle.sysman.assistants.util.step.StepContext$ModeRunner.run(StepContext.java:2468)
    at java.lang.Thread.run(Thread.java:534)
[Thread-54] [1:7:4:273] [EMConfigStep.handleNonIgnorableError:197]  EMConfig Error:=Error ignored: Could not complete the Enterprise Manager configuration.
Enterprise manager configuration failed due to the following error -
EM is not configured for this database. No EM-specific actions can be performed.
Refer to the log file at C:\OracleHomes\db10gr2\cfgtoollogs\dbua\emrep\upgrade1\emConfig.log for more details.
You can retry configuring this database with Enterprise Manager later by manually running C:\OracleHomes\db10gr2\bin\emca script.

[Thread-54] [1:7:4:273] [EMConfigStep.executeStepImpl:161]  EM Configuration Step Complete..
[Thread-54] [1:7:4:273] [SummarizableStep.executeImpl:175]  execute impl:Enterprise Manager Configuration = DONE Sucessfully
[Thread-54] [1:7:4:273] [BasicStep.configureSettings:304]  messageHandler being set=oracle.sysman.assistants.util.UIMessageHandler@6014f7

At first, i thought its over! My database is gone...but a few research down the line gave me a breather :)
I was upgrading my Oracle Enterprise Manager (OEM) Grid Control Database from 10.1.2.0 to 10.2.1.0. A Grid Control Database installation does not have a separate DBConsole installed as the Database is also managed by the OEM itself.

So in my particular case, i could simply press OK & then move ahead. The upgrade was almost complete & nothing else was required.

I just reconfigured the LSNRCTL & NET SERVICE NAME using Net Configuration Tool & rebooted my server & my new database was up & running.

However, if you are not upgrading a GC Repository Database, but an actual standalone database, this error should not be ignored as your DB Console would not be upgraded due to it.
You must followup with Oracle & find out why the upgrade for DBConsole failed.

Hope this info helps!

Keywords:
Oracle Database, Oracle 10GR1, Oracle 10GR2, Oracle Enterprise Manager, OEM, Grid Control, GC Repository Upgrade, Database Upgrade, DB Console, LISTENER.ORA, TNSNAMES.ORA, SQLNET.ORA, Listener Control, TNS Listener, DBUA, EMCA etc

Exception in thread "main" java.lang.NoClassDefFoundError: oracle/bi/analytics/management/StandardConsoleAgent

Hi Friends,

This is my newest experiment going on with OBIEE Management Pack in Oracle Enterprise Manager (OEM) Tool (Oracle Management Server (OMS)) & OBIEE Application.

OBIEE Management Pack needs to read the performance values exposed over JMX. In order to start this utility, we need to configure the runagent.sh or runagent.cmd under /$OBIEE_ROOT/systemsmanagement/

Following is the sample runagent.sh file setup for a OBIEE Setup on Unix Platform:

#!/bin/sh
# this is a template of runagent.sh to be used on Unix.
# The installer will fill in JAVA_HOME, SAROOTDIR, and SATEMPDIR

export JAVA_HOME="/usr/java/jdk1.5.0_19"
export SAROOTDIR=/$OBIEE_ROOT/
export SADATADIR=/$OBIEE_DATA/
export SATEMPDIR=/$OBIEE_DATA/tmp
export UNIXPERFDIR=${SATEMPDIR}

java_cmd="${JAVA_HOME}/bin/java -Djava.library.path=${SAROOTDIR}/server/Bin -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9980 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -classpath analytics-jmx.jar:lib/xmlparserv2.jar oracle.bi.analytics.management.StandardConsoleAgent"

${java_cmd}

Now if you carefully observe, if you invoke the script from your home directory, the classpath will fail to resolve & give error like:
Exception in thread "main" java.lang.NoClassDefFoundError: oracle/bi/analytics/management/StandardConsoleAgent

Above issue is purely due to the way classpath parameter is given in java_cmd: -classpath analytics-jmx.jar:lib/xmlparserv2.jar oracle.bi.analytics.management.StandardConsoleAgent

The analytics-jmx.jar & xmlparsev2.jar are not resolved as absolute path locations are not given here. So you need to manually change your directory till /$OBIEE_ROOT/systemsmanagement/ & then invoke runagent.sh file to start the JMX Performance counters successfully.

The JMX Performance Counters needs to be exposed constantly so that OBIEE Management Pack reads correct values. To achieve this, you need to start the runagent.sh file in background mode. e.g.-bash-3.00$ runagent.sh &  or-bash-3.00$ nohup runagent.sh & so even after you logoff your session, the utility keeps running & gathers correct values.

Enjoy!

Keywords:
OBIEE, Siebel Analytics, Business Intelligence, Oracle Enterprise Manager Grid Control, OMS, Oracle Management Server, OEM, Grid Control, JMX, JAVA, OBIEE Performance, etc