We’ve been pushing out some Oracle 19c databases on Oracle Linux 9 (OL9) since it was certified, see here, but those databases have not been monitored or backed up by Cloud Control, because the Enterprise Manager (EM) 13.5 agent was not certified on OL9. Instead we had reverted to the bad old days of using CRON scripts to do everything.
Since we started running 19c on OL9 I had been regularly searching for the EM certification notice. Last week I noticed MOS Doc ID 2978593.1, which said, “EM Agent 13.5 with Enterprise Manager 13.5 Release Update 22 or higher is certified on RHEL9 and OL9”. Happy days. I subsequently found this post announcing update 22, which I had somehow missed. I was also nudged by someone on Twitter/X to check the certification matrix again.
EM 13.5.0.22 at Work
We have EM 13.5 running on OL8 at work. As soon as I found the agent was certified on OL9 servers I updated our installation to release update 22 and started to try and push out agents to the OL9 servers.
We hit an initial problem, which was EM uses SSH to push out the agent, and uses SHA1 to do it. Unfortunately SHA1 is not allowed on our OL9 servers, so that kind-of scuppered things. This is the error we got.
- Error: SSH connection check failed
- Cause: Connection to the SSH daemon (sshd) on the target host failed with the following error : KeyExchange signature verification failed for key type=ssh-rsa
- Recommendation: Ensure that SSH daemon (sshd) on the target host is able to respond to login requests on the provided sshd port 22.Ensure the provided user name and password or ssh keys are correct.Ensure that the property PasswordAuthentication is set to yes in the SSHD configuration file (sshd_config) on the remote host.
To resolve this the system administrators issued the following command as root on the OL9 servers.
update-crypto-policies --set DEFAULT:SHA1
Once that was done the agents pushed out with no problems.
I’m currently pushing out agents to all our 19c on OL9 servers, and replacing all the CRON scripts with EM backups and monitoring.
This brings OL9 into line with out other databases servers. Happy days.
EM 13c Installation on OL9
Although I don’t need it for work, I decided to spend the bank holiday weekend trying to do a fresh installation of 13.5 on OL9. I tried several different ways, with the main two being.
- Install and configure the base release without the patches.
- Install with the patches downloaded and applied as part of the installation and configuration.
In both cases everything looked fine until near the end of the process, where the OMS refused to start. Unfortunately I couldn’t find anything obvious in the logs. It takes a long time to run the build, so having it fail near the end is quite frustrating.
At the moment I can’t see any OL9 specific docs, so I can’t tell if I’m missing out a vital step. As mentioned in the previous section, there are definite differences between OL9 and OL8, so I would not be surprised if the documentation (or MOS note) is released that includes an obvious gotcha.
As soon as I get it working I’ll release an article and a Vagrant build.
Cheers
Tim…
Hi Tim!
Faced the same issue and found your post …. looks like we need to have a little patience
and waiting for EL9-documentation.
regards,
Rainer
Hi Tim,
I was also strugeling to get a fresh installation of EM 13.5.0.0.22 working on OL9 and ran into the same issue, that OHS refused to start by the end of the installation.
But I guess that I have at least found the source of the problem and possibly also the workaround. In my case I have performed a “software-only” installation first, then applied
the 13.5.0.0.22 RU (with all required prerequisite patches in advance). After that I ran the configuration with “ConfigureGC.sh”, which refuses to start OHS at the end of the process.
By searching the logs it turned out that the following library is missing: “libclntshcore.so.12.1” (which is truly not available within the /lib directory).
The workaround seems to be to just copy the existing libclntshcore.so.12.1 file from the /agent_/instantclient/ to the /lib directory before(!) starting the configuration with “ConfigureGC.sh”.
At least I was able to find Doc ID 2790890.1 which confirms that. Infact, I have done it a little bit different (by installing files from Patch 35775632 into /lib/stubs/ and run “middleware/bin/genclntsh”, which in turned creates the required “libclntshcore.so.12.1” – but the result seems to be the same).
After that, the configuration smoothly finished and I got EM 13.5.0.0.22 up and running.
The second issue regarding the ssh-rsa keys still using SHA1 causing problems was also on our agenda. But unfortunately our Linux Admins did not allow us to set “update-crypto-policies –set DEFAULT:SHA1”.
Therefor I just tried it by using different key types, which seems to work:
When using EM 13.5.0.0.22 on OL9 as well as rolling out agents to OL9 DB-Hosts it is finally working by creating ECDSA keys (instead of RSA keys) by issuing: “ssh-keygen -t ecdsa -m PEM” on the EM13c/OMS-Host (honestly I don’t know if “-m PEM” is really required), but it works).
Those keys are then accepted when creating a named credential and the initial rollout/push of the agent via ssh keys is working.
I still have the impression that even if EM 13.5.0.0.22 is finally certified under OL9 there are still many bugs around and Oracle’s software quality has not really improved…
Hope this helps!
Best regards,
Alexander Almer, DBA, Vienna
Yes. Those points all look familiar. My Vagrant build has these lines.
# Fix broken web tier by copying libclntshcore.so.12.1 file.
cp /u01/app/oracle/agent/agent_13.5.0.0.0/instantclient/libclntshcore.so.12.1 /u01/app/oracle/middleware/ohs/lib/
${MW_HOME}/bin/emctl start oms
I don’t think I would consider running 13.5 on OL9 for real. We run it on OL8, and push agents to OL9, so what we have so far is fine.
Cheers
Tim…!