Since the upgrade to Cloud Control 12.1.0.5, we’ve been having a couple of issues, mostly around EMCLI.
Some of our databases use Service Guard, so you don’t know which node they are running on. Rather than having an agent per package, we have one on each node. One of my colleagues wrote a little script to check which node the instance is running on, and relocate it if it has moved. This is done using EMCLI and was working fine before the move to 12.1.0.5. Since the upgrade it’s been rather erratic. It would work for a while, then fail. After watching for a while I noticed a couple of things.
EMCLI calls to the OMS that sent SQL were intermittently killing in the Agent on the Cloud Control server itself. It seemed to cause Out Of Memory errors. The heap size for the agent on the CC server was set to something like 1860M. Changing it to 2048M seemed to fix that issue. The setting is in the “agent_inst/sysman/config/emd.properties” file. We now have this.
agentJavaDefines=-Xmx2048M -XX:MaxPermSize=128M
Note. The agents on the monitored servers have a tiny heap size. Nothing like this bad-boy. 🙂
After running OK for a few days, we started to get the following type of errors from EMCLI.
$ emcli sync Error: Session expired. Run emcli login to establish a session.
We fixed this by running the “setup” command again.
emcli setup -username=sysman -password="MyPassword" -url="https://myserver.example.com:7799/em"
Since then, EMCLI has seemed to behave itself.
Note. As part of the upgrade, we downloaded the latest EMCLI jar file from the server and did the setup, so this part was working fine for a while. Not sure why it started to screw up…
Anyway, all seems fine now. I’m guessing if we weren’t using EMCLI, we would never have had a problem in the first place.
Cheers
Tim…