Cloud Control : It’s production upgrade day…


cloudI mentioned a couple of months ago I was planning to upgrade our production Enterprise Manager Cloud Control installation from to Well, today was the day. I held back a while because I knew I would be out of the country for a while on the Latin America tour and I didn’t want to make a big change before I ran away. :)

So today I pretty much did exactly what was in my upgrade article and everything went well. I upgraded the OMS and the local agent and I’ll run like that for a couple of days before I start pushing out the agent updates to the monitored hosts.

Happy days!

If you are interested, you can see some of my Cloud Control articles here.



Upgrade Cloud Control 12cR4 to 12cR5


em-12cA couple of weeks ago I wrote a post about doing a Cloud Control 12cR5 installation and said I would be testing the upgrade from 12cR4. I’ve now done that.

The upgrade documentation is quite extensive and the prerequisites are going to be different depending on the database and cloud control versions you are starting with, so this is no way a “recommended” way to do the upgrade. Each one will need to be approached on a case-by-case basis. It’s just meant to give a flavour of what you have to do.

Suffice to say, it worked fine for me. :)



Oracle Enterprise Manager Cloud Control 12c Release 5 ( : My first two installations


em-12cI’ve done a couple of play installations of EM12c, just to get a feel for it. You can see the result of that here.

From an installation perspective, everything was pretty similar to the previous releases. I tried the installation on both OL5 and OL6, in both cases using 12c as the database repository. No dramas there.

A couple of things of note.

  1. The 12c repository template database is a Non-CDB architecture.
  2. The Weblogic installation uses Java6.


The next step is to try some upgrades from EM (on DB to EM, which is what I’ll need for my upgrades at work. The testing is quite time consuming and boring, but it’s got to be done before I can unleash this on the company. :)



PS. Remember to download from (in a couple of days) for your production installations. Apparently there is a difference to the license agreement.

EM Cloud Control 12c : The 24 Hour DBA


I think I’ve lived through all the ages of Enterprise Manager. I used the Java console version back in the days when admitting you used it got you excommunicated from the church of DBA. I lived through the difficult birth of the web-based Grid Control. I’ve been there since the start of Cloud Control. I’ll no doubt be there when it is renamed to Big Data Cloud Pixie Dust Manager (As A Service).

I was walking from the pool to work this morning, checking my emails on my phone and it struck me (not for the first time) that I’m pretty much a 24 hour DBA these days. I’m not paid to be on call, I’m just a 9-5 guy, but all my Cloud Control notifications come through to my phone and tablet. I know when backups have completed (or failed). I know when a Tnsping takes too long. I know when we have storage issues. I know all this because Cloud Control tells me.

Now you might look on this as a bad thing, but being the control freak I am, I prefer to get a message on a Sunday telling me something is broken, hop on the computer and fix it there and then, rather than coming in on Monday to a complete sh*t-storm of users complaining. I’m not paid to do it, but that’s the way I roll.

While walking down memory lane I was thinking about all the scripting I used to do to check all this stuff. Endless amounts of shell scripts to check services and backups etc. I don’t do hardly any of that these days. Cloud Control handles all that.

We are a pretty small Oracle shop, but I think life would be a whole lot more difficult without Cloud Control. I’ve mentioned this a number of times, but it’s worth saying again… If you have more than a handful of Oracle databases, you really should be using Cloud Control these days. It’s as simple as that.

Just in case you are wondering, this is how our infrastructure looks this morning… :)




Enterprise Manager Cloud Control 12cR4 Production Upgrade


I’ve already written about the 12cR3 to 12cR4 upgrade here. I did a few run through’s at home to practice it and it all seemed good.

Setting The Scene

Just to set the scene, for our production environment we run Cloud Control in a VMware virtual machine, using Oracle Linux 6.5 as the guest OS. With that setup, we can use a simple installation (DB and OMS on the same VM) and use VMware to provide our failover, rather than having to worry about multiple OMS installations and any DB failover technology etc. If there’s one thing I’ve learned about Cloud Control, it’s Keep It Simple Stupid (KISS)! As far as our managed servers go, most of our databases and all our middle tier stuff runs on VMware and Oracle Linux too. We have a handful of things still hanging around on HP-UX and Solaris, which will hopefully be migrated soon…

Upgrade Attempt 1 : Non-Starter

Yesterday I started the upgrade of our production system. Pretty much straight out of the blocks I hit a road block. It didn’t like the agents running on our HP-UX servers. The upgrades of the HP-UX agents are so painful. Every time so far I’ve had to reinstall them. As a result, I didn’t bother to upgrade them last time and kept running with the previous version of the agents. The upgrade wouldn’t have anything to do with that, so I forgot about the Cloud Control upgrade and I spent yesterday attempting to upgrade the HP-UX agents to 12cR3, before I could attempt the 12cR4 Cloud Control upgrade.

As usual, the upgrade of the agents on HP-UX involved me uninstalling, removing all the targets, installing, discovering all the targets and setting up the backups etc. Not all of it is scripted yet, so it is an annoying and painful process. I’m not sure if other HP-UX users suffer this, but it seems pretty consistently bad for us. The sooner we get rid of these straggling HP-UX servers the better!

So this wasn’t so much a failure of the upgrade. It was really down to me being lazy and not bothering to upgrade some agents.

Fast forward to this morning and I was actually ready to start the upgrade. :)

Upgrade Attempt 2 : Success

With the 12cR3 agents in place on HP-UX, the upgrade ran past that step with no problems and on to the main body of the installation. The install and upgrade were textbook.

I’ve upgraded the agent on the cloud control server, but I’m not going to upgrade any of the other agents until I know things are working fine.

Happy days!



Enterprise Manager Cloud Control 12c Release 4 (12cR4) Articles


I’ve started to play around with Enterprise Manager Cloud Control 12R4. The clean installations on Oracle Linux 5 and Oracle Linux 6 were really easy. You can see how I did them here.

I used as the database repository. I’ll probably have a go with a 12c database in future, which is now supported, but my main focus this time was to check out something similar to what I have at work. Forgive my caution, but I’ll not be using 12c database for my EM repository for a while yet. Cloud Control is too important to risk…

I wasn’t going to bother with an upgrade article because  wrote a good blog post about the upgrade here. Whilst going through the upgrade, there were a few things I needed to make extra notes on, since I have an terrible memory, so it ended up as a separate article.

Ultimately, you are going to need to read the upgrade docs because there are lots of caveats to think about, depending on what options you use.

Previous upgrades at work were a pain, but we had a more complicated installation with multiple management servers and a separate database. More recently we switched to running a simple configuration (DB and OMS on the same server) on an Oracle Linux VM. That makes doing a trial run of the upgrade at home a more realistic test. Whether it’s an improved Cloud Control upgrade process or the fact this is a single machine setup, the upgrade was really easy. Now that I’ve practiced the upgrade at home, I’m feeling relatively confident doing the upgrade in production. I’ll probably leave it until I get back from Bulgaria (BGOUG). It’s not really fair to change everything and then leave the country… :)



Cloud Control 12cR3, Oracle Linux and VMware


I mentioned some time ago that I was pushing my current company to move much of their gear on to VMware, mostly because of poor resource utilization on many of the servers. That process is still under way.

One thing I wanted to mention specifically was our use of Cloud Control 12cR3. Up until recently, we were using physical kit for this. We had an 11.2 database on HP-UX, With HA provided by HP Service Guard. We had two management servers on physical kit running RHEL5 pointing at this Service Guard package to give us some resiliency in of the OMS. It worked, but it was over complicated and I was never really happy with it for a number of reasons:

  • HP-UX for the databases : I know some of you guys love it, but I don’t.
  • Two management servers : Seems like a waste of kit to me. We either have them on their own boxes and waste lots of resources, or have multiple installations on those boxes, which adds to complexity and management of the kit.
  • RHEL : Why pay for RHEL when we can use Oracle Linux and decide for ourselves if we want to pay for the extra features support gives us, or just use it for free?

So what are we running now? We have one VMware VM, running Oracle Linux 6. That has both the Oracle 11.2 database for the repository and the Cloud Control 12cR3 OMS running on it. We use VMware functionality for the HA of this system.

Why do I like this situation?

  • Cloud Control is a complicated beast and I am a big fan of KISS (Keep It Simple Stupid). Having everything on a single VM is about as simple as it gets.
  • If I am using Cloud Control in this way I pay nothing for the database repository. As soon as you start thinking about RAC or Data Guard to protect your repository you have to pay for Enterprise Edition licenses.
  • Using VMware HA functionality gives us good enough HA for our purposes. We can failover or live migrate between hosts in the data centre, or between data centres.
  • We can clone the whole installation in a few minutes and use that as a base for upgrades. If something goes wrong, we just flip back.

While I was at Oracle OpenWorld I discussed this a number of times and it seems it is a very common approach.

Another thing that came out of those discussions is many people still misunderstand what Oracle Linux is and the support status of Oracle Linux, and more specifically UEK, on VMware. Suffice to say, it’s all supported, as discussed in my Oracle Linux : Frequently Asked Questions article.

If you are struggling to decide how best to run Cloud Control in your organization, I would recommend using a virtual environment (Oracle VM or VMware) and run it on Oracle Linux 6.



Cloud Control 12c Database Backup Jobs (Continued)


I’ve been rather critical of the way Cloud Control handles database backup jobs, as can be seen in these two previous posts.

Yesterday I found out I schedule database backups in Cloud Control the “wrong way”…

So typically, when I am sorting out a new database, I do something like this:

  • If the host doesn’t already have an agent, push one out the to server.
  • Discover the targets. For existing hosts, this may just be discovering a new database.
  • Tell Cloud Control to used the RMAN catalog for backups. (Oracle Database > Availability > Backup & Recovery > Recovery Catalog Settings)
  • Set the default location for the backups and any other stuff. (Oracle Database > Availability > Backup & Recovery > Backups Settings)
  • Finally schedule the backup. (Oracle Database > Availability > Backup & Recovery > Schedule Backups…)

Notice how all the backup-related things come from the same location (Oracle Database > Availability > Backup & Recovery). Obvious right? Wrong!

So it turns out, if you schedule the backup using the (Oracle Database > Availability > Backup & Recovery > Schedule Backups…) screen, you do schedule a backup job, it works and it is visible in the (Enterprise > Job > Activity) screen, but it is not a “real” Cloud Control job. It is actually created using the job type of “Backup”, which is not a supported job type, rather than the job type “RMANScript”.

If you schedule a backup in this way, you can’t:

  • use the job library.
  • do a create-like of an existing job.
  • edit most of the details about the job after it’s created, including the RMAN script.
  • describe the job using EMCLI.

So what is the solution? Well, you do all the prep in the same way, but you don’t actually schedule the job using the (Oracle Database > Availability > Backup & Recovery > Schedule Backups…) screen. You do it directly from the (Enterprise > Job > Activity) screen. If you do it from here, then you get to pick the “RMANScript” job type and suddenly everything works much better and things seem a lot more consistent. Basically, you’ve created a “real” Cloud Control job.

So in my latest round of [enchancement requests | bug reports] I got the answer back from development that I was doing it wrong, which is good to know, and they want to close the SR. My response to that was, why do you let me do it the “wrong way”? Why does the (Oracle Database > Availability > Backup & Recovery > Schedule Backups…) screen not schedule a “real” Cloud Control job? It seems odd to me that you build a product that lets you do things inconsistently. It’s probably a throw back to the old DBConsole stuff, but that doesn’t make it any less annoying.

I’m going to recreate all my backup jobs to make them use the “RMANScript” job type, which will solve my EMCLI issues, but I really think this mess should get cleaned up. I wonder how many other people out there are not creating “real” Cloud Control jobs for their database backups?

I’m going add this blog post to my SR and see what the response is…



Oracle Cloud Control EMCLI : First Thoughts


Whilst waiting for a new version of Cloud Control, which will hopefully include my job scheduler enhancement requests, I decided to see if I could solve my problem using the command line interface (EMCLI). That spawned this very basic article.

I was initially really excited by the EMCLI, planning to pretty much replace my job creation with EMCLI scripts, but there seem to be a bunch of discrepancies with the EMCLI compared to the regular Cloud Control interface. Now I admit I’m a newbie at this, but here’s what I’ve seen so far.

The built in help is fine, sometimes prompting you with other commands to get more information, but sometimes it leaves quite important stuff out. For example, when you do a “describe_job_type” for the job type “RMANScript”, it does not include the possible schedule information, so it appears you can only schedule one-off jobs. A little digging and you find the scheduling is possible, and seems to work OK. It’s a little annoying this isn’t included in the property file template though.

It would seem like the easiest way to figure out how to schedule a job using EMCLI would be to create a DB backup job manually, the use the “describe_job” verb to figure out what the job properties are. The problem is, you can’t describe a job with the job type of “Backup”.

C:\emcli>emcli describe_job -name=test_db_backup
job type [Backup] not supported in EMCLI.


But wait a minute. When you create a DB backup from EMCLI it is created with the job type of “RMANScript”, which works with “describe_job” just fine. So why do the two methods for creating a job to do the same thing use a different job type? Even if that were sensible for some reason, why does the job type of “Backup” not work with the “describe_job” verb?

My next issue was I can create and delete a one-off job using “create_job” and “delete_job” respectively, but as soon as I create a scheduled job with a repeating schedule, the “delete_job” verb seems to be useless, because as long as there is a scheduled run in the future, you get the following message.

C:\emcli>emcli delete_job -name=test_db_backup
Error: Cannot delete a job with active executions. Before deletion, the job status must be one of STOPPED,COMPLETED,ABORTED, or FAILED.


Remember, this is not a running job, just a job that is scheduled to run in the future. If you try to delete the job from the Cloud Control interface it works fine…

As I said before, I’m a total newbie at this EMCLI stuff, so it may be my (lack of) understanding that is wrong, but there seems to be a number of inconsistencies that make it really annoying. I’m raising enhancement requests as I find these issues, so hopefully things will be resolved in future releases, or the responses will explain why my issues are due to be being a dumb-ass. :)

Moaning aside, I think the EMCLI stuff has got a lot of potential. Personally, I would like to script a lot of the configuration, giving the option of clean installs of new versions as well as upgrades.



Oracle Cloud Control 12c Release 3 Installation (EM12cR3)…


With all the excitement about Oracle Database 12c being released, you may have missed the release of Cloud Control 12c Release 3. It’s available for download from OTN. All the usual ports are available. This is the version you are going to need if you want to monitor 12c databases in your organisation.

I did a quick run through the Cloud Control 12c Release 3 installation yesterday on both OL5 and OL6. As I suspected, there is pretty much no change in the installation process compared to the previous release.

I haven’t tried an upgrade yet, so I don’t know how easy that is. Perhaps someone else can comment? :)

Note. Oracle Database 12c is not a supported repository database version, so you are still going to need Oracle 11g for this installation.