Enterprise Manager Cloud Control 12cR4 Production Upgrade

I’ve already written about the 12cR3 to 12cR4 upgrade here. I did a few run through’s at home to practice it and it all seemed good.

Setting The Scene

Just to set the scene, for our production environment we run Cloud Control in a VMware virtual machine, using Oracle Linux 6.5 as the guest OS. With that setup, we can use a simple installation (DB and OMS on the same VM) and use VMware to provide our failover, rather than having to worry about multiple OMS installations and any DB failover technology etc. If there’s one thing I’ve learned about Cloud Control, it’s Keep It Simple Stupid (KISS)! As far as our managed servers go, most of our databases and all our middle tier stuff runs on VMware and Oracle Linux too. We have a handful of things still hanging around on HP-UX and Solaris, which will hopefully be migrated soon…

Upgrade Attempt 1 : Non-Starter

Yesterday I started the upgrade of our production system. Pretty much straight out of the blocks I hit a road block. It didn’t like the agents running on our HP-UX servers. The upgrades of the HP-UX agents are so painful. Every time so far I’ve had to reinstall them. As a result, I didn’t bother to upgrade them last time and kept running with the previous version of the agents. The upgrade wouldn’t have anything to do with that, so I forgot about the Cloud Control upgrade and I spent yesterday attempting to upgrade the HP-UX agents to 12cR3, before I could attempt the 12cR4 Cloud Control upgrade.

As usual, the upgrade of the agents on HP-UX involved me uninstalling, removing all the targets, installing, discovering all the targets and setting up the backups etc. Not all of it is scripted yet, so it is an annoying and painful process. I’m not sure if other HP-UX users suffer this, but it seems pretty consistently bad for us. The sooner we get rid of these straggling HP-UX servers the better!

So this wasn’t so much a failure of the upgrade. It was really down to me being lazy and not bothering to upgrade some agents.

Fast forward to this morning and I was actually ready to start the upgrade. 🙂

Upgrade Attempt 2 : Success

With the 12cR3 agents in place on HP-UX, the upgrade ran past that step with no problems and on to the main body of the installation. The install and upgrade were textbook.

I’ve upgraded the agent on the cloud control server, but I’m not going to upgrade any of the other agents until I know things are working fine.

Happy days!

Cheers

Tim…

Enterprise Manager Cloud Control 12c Release 4 (12cR4) Articles

I’ve started to play around with Enterprise Manager Cloud Control 12R4. The clean installations on Oracle Linux 5 and Oracle Linux 6 were really easy. You can see how I did them here.

I used 11.2.0.4 as the database repository. I’ll probably have a go with a 12c database in future, which is now supported, but my main focus this time was to check out something similar to what I have at work. Forgive my caution, but I’ll not be using 12c database for my EM repository for a while yet. Cloud Control is too important to risk…

I wasn’t going to bother with an upgrade article because  wrote a good blog post about the upgrade here. Whilst going through the upgrade, there were a few things I needed to make extra notes on, since I have an terrible memory, so it ended up as a separate article.

Ultimately, you are going to need to read the upgrade docs because there are lots of caveats to think about, depending on what options you use.

Previous upgrades at work were a pain, but we had a more complicated installation with multiple management servers and a separate database. More recently we switched to running a simple configuration (DB and OMS on the same server) on an Oracle Linux VM. That makes doing a trial run of the upgrade at home a more realistic test. Whether it’s an improved Cloud Control upgrade process or the fact this is a single machine setup, the upgrade was really easy. Now that I’ve practiced the upgrade at home, I’m feeling relatively confident doing the upgrade in production. I’ll probably leave it until I get back from Bulgaria (BGOUG). It’s not really fair to change everything and then leave the country… 🙂

Cheers

Tim…

Cloud Control 12cR3, Oracle Linux and VMware

I mentioned some time ago that I was pushing my current company to move much of their gear on to VMware, mostly because of poor resource utilization on many of the servers. That process is still under way.

One thing I wanted to mention specifically was our use of Cloud Control 12cR3. Up until recently, we were using physical kit for this. We had an 11.2 database on HP-UX, With HA provided by HP Service Guard. We had two management servers on physical kit running RHEL5 pointing at this Service Guard package to give us some resiliency in of the OMS. It worked, but it was over complicated and I was never really happy with it for a number of reasons:

  • HP-UX for the databases : I know some of you guys love it, but I don’t.
  • Two management servers : Seems like a waste of kit to me. We either have them on their own boxes and waste lots of resources, or have multiple installations on those boxes, which adds to complexity and management of the kit.
  • RHEL : Why pay for RHEL when we can use Oracle Linux and decide for ourselves if we want to pay for the extra features support gives us, or just use it for free?

So what are we running now? We have one VMware VM, running Oracle Linux 6. That has both the Oracle 11.2 database for the repository and the Cloud Control 12cR3 OMS running on it. We use VMware functionality for the HA of this system.

Why do I like this situation?

  • Cloud Control is a complicated beast and I am a big fan of KISS (Keep It Simple Stupid). Having everything on a single VM is about as simple as it gets.
  • If I am using Cloud Control in this way I pay nothing for the database repository. As soon as you start thinking about RAC or Data Guard to protect your repository you have to pay for Enterprise Edition licenses.
  • Using VMware HA functionality gives us good enough HA for our purposes. We can failover or live migrate between hosts in the data centre, or between data centres.
  • We can clone the whole installation in a few minutes and use that as a base for upgrades. If something goes wrong, we just flip back.

While I was at Oracle OpenWorld I discussed this a number of times and it seems it is a very common approach.

Another thing that came out of those discussions is many people still misunderstand what Oracle Linux is and the support status of Oracle Linux, and more specifically UEK, on VMware. Suffice to say, it’s all supported, as discussed in my Oracle Linux : Frequently Asked Questions article.

If you are struggling to decide how best to run Cloud Control in your organization, I would recommend using a virtual environment (Oracle VM or VMware) and run it on Oracle Linux 6.

Cheers

Tim…

Cloud Control 12c Database Backup Jobs (Continued)

I’ve been rather critical of the way Cloud Control handles database backup jobs, as can be seen in these two previous posts.

Yesterday I found out I schedule database backups in Cloud Control the “wrong way”…

So typically, when I am sorting out a new database, I do something like this:

  • If the host doesn’t already have an agent, push one out the to server.
  • Discover the targets. For existing hosts, this may just be discovering a new database.
  • Tell Cloud Control to used the RMAN catalog for backups. (Oracle Database > Availability > Backup & Recovery > Recovery Catalog Settings)
  • Set the default location for the backups and any other stuff. (Oracle Database > Availability > Backup & Recovery > Backups Settings)
  • Finally schedule the backup. (Oracle Database > Availability > Backup & Recovery > Schedule Backups…)

Notice how all the backup-related things come from the same location (Oracle Database > Availability > Backup & Recovery). Obvious right? Wrong!

So it turns out, if you schedule the backup using the (Oracle Database > Availability > Backup & Recovery > Schedule Backups…) screen, you do schedule a backup job, it works and it is visible in the (Enterprise > Job > Activity) screen, but it is not a “real” Cloud Control job. It is actually created using the job type of “Backup”, which is not a supported job type, rather than the job type “RMANScript”.

If you schedule a backup in this way, you can’t:

  • use the job library.
  • do a create-like of an existing job.
  • edit most of the details about the job after it’s created, including the RMAN script.
  • describe the job using EMCLI.

So what is the solution? Well, you do all the prep in the same way, but you don’t actually schedule the job using the (Oracle Database > Availability > Backup & Recovery > Schedule Backups…) screen. You do it directly from the (Enterprise > Job > Activity) screen. If you do it from here, then you get to pick the “RMANScript” job type and suddenly everything works much better and things seem a lot more consistent. Basically, you’ve created a “real” Cloud Control job.

So in my latest round of [enchancement requests | bug reports] I got the answer back from development that I was doing it wrong, which is good to know, and they want to close the SR. My response to that was, why do you let me do it the “wrong way”? Why does the (Oracle Database > Availability > Backup & Recovery > Schedule Backups…) screen not schedule a “real” Cloud Control job? It seems odd to me that you build a product that lets you do things inconsistently. It’s probably a throw back to the old DBConsole stuff, but that doesn’t make it any less annoying.

I’m going to recreate all my backup jobs to make them use the “RMANScript” job type, which will solve my EMCLI issues, but I really think this mess should get cleaned up. I wonder how many other people out there are not creating “real” Cloud Control jobs for their database backups?

I’m going add this blog post to my SR and see what the response is…

Cheers

Tim…

Oracle Cloud Control EMCLI : First Thoughts

Whilst waiting for a new version of Cloud Control, which will hopefully include my job scheduler enhancement requests, I decided to see if I could solve my problem using the command line interface (EMCLI). That spawned this very basic article.

I was initially really excited by the EMCLI, planning to pretty much replace my job creation with EMCLI scripts, but there seem to be a bunch of discrepancies with the EMCLI compared to the regular Cloud Control interface. Now I admit I’m a newbie at this, but here’s what I’ve seen so far.

The built in help is fine, sometimes prompting you with other commands to get more information, but sometimes it leaves quite important stuff out. For example, when you do a “describe_job_type” for the job type “RMANScript”, it does not include the possible schedule information, so it appears you can only schedule one-off jobs. A little digging and you find the scheduling is possible, and seems to work OK. It’s a little annoying this isn’t included in the property file template though.

It would seem like the easiest way to figure out how to schedule a job using EMCLI would be to create a DB backup job manually, the use the “describe_job” verb to figure out what the job properties are. The problem is, you can’t describe a job with the job type of “Backup”.

C:\emcli>emcli describe_job -name=test_db_backup
job type [Backup] not supported in EMCLI.

C:\emcli>

But wait a minute. When you create a DB backup from EMCLI it is created with the job type of “RMANScript”, which works with “describe_job” just fine. So why do the two methods for creating a job to do the same thing use a different job type? Even if that were sensible for some reason, why does the job type of “Backup” not work with the “describe_job” verb?

My next issue was I can create and delete a one-off job using “create_job” and “delete_job” respectively, but as soon as I create a scheduled job with a repeating schedule, the “delete_job” verb seems to be useless, because as long as there is a scheduled run in the future, you get the following message.

C:\emcli>emcli delete_job -name=test_db_backup
Error: Cannot delete a job with active executions. Before deletion, the job status must be one of STOPPED,COMPLETED,ABORTED, or FAILED.

C:\emcli>

Remember, this is not a running job, just a job that is scheduled to run in the future. If you try to delete the job from the Cloud Control interface it works fine…

As I said before, I’m a total newbie at this EMCLI stuff, so it may be my (lack of) understanding that is wrong, but there seems to be a number of inconsistencies that make it really annoying. I’m raising enhancement requests as I find these issues, so hopefully things will be resolved in future releases, or the responses will explain why my issues are due to be being a dumb-ass. 🙂

Moaning aside, I think the EMCLI stuff has got a lot of potential. Personally, I would like to script a lot of the configuration, giving the option of clean installs of new versions as well as upgrades.

Cheers

Tim…

Oracle Cloud Control 12c Release 3 Installation (EM12cR3)…

With all the excitement about Oracle Database 12c being released, you may have missed the release of Cloud Control 12c Release 3. It’s available for download from OTN. All the usual ports are available. This is the version you are going to need if you want to monitor 12c databases in your organisation.

I did a quick run through the Cloud Control 12c Release 3 installation yesterday on both OL5 and OL6. As I suspected, there is pretty much no change in the installation process compared to the previous release.

I haven’t tried an upgrade yet, so I don’t know how easy that is. Perhaps someone else can comment? 🙂

Note. Oracle Database 12c is not a supported repository database version, so you are still going to need Oracle 11g for this installation.

Cheers

Tim…

I love Cloud Control 12c, but the job management is really annoying!

Cloud Control 12c is a great product. Yes, it is suffering from bloat, but generally it is a really great tool. I’m always encouraging people to ditch the DBControl and switch to Cloud Control!

Having said that, one area that annoys the hell out of me is the job management, which feels really clumsy. I started to write this post, then felt a bit guilty because I hadn’t actually bothered to raise an enhancement request, so that’s what I’ve just done!

My main gripes are the following.

  • Editing Jobs: You can edit some parameters of jobs, but not all. This is really frustrating and often means you have to delete and recreate a job in order to make a minor change.
  • Create Like: There is still no support for “Create Like” of a database backup job. Most of our jobs are database backup jobs, so this omission is quite annoying.
  • Run Now: There is no “Run Now” type of functionality on the scheduled jobs. You either have to create a new one-off job (with out Create Like), or alter the schedule, then remember to alter it back once the job has finished. A “Run Now” option that doesn’t affect the normal schedule would be really handy.
  • Backup Destination: If you use “CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT ‘/path/to/backup/%U’;” from RMAN, or use the Backup Settings screen in Cloud Control, which does the same thing, the path used for the backup works fine. If on the other hand you use the “Over Default Settings” option while creating the database backup job, the settings seem to be completely ignored. This should either work, or be removed as an option because it catches me out all the time!

OK, the last one is a bug, not an enhancement request, but I thought I would throw that into the mix anyway. 🙂

We are in the process of upgrading and moving databases at the moment, so amending the backup jobs is a big thing for us. Yesterday afternoon, evening and night was littered with a chorus of expletives in relation to Cloud Control job management, which gave me the activation energy to post this rant. 🙂

Note. Even as I rant, I am still convinced centralising your jobs in Cloud Control is a good thing!

Cheers

Tim…

Update: My enhancement request is being split into three separate SRs and they are being put forward to the development team as formal enhancement requests. Fingers crossed they will get implemented.

The fourth point is a “problem between keyboard and chair”. I misunderstood what this feature was supposed to do. My bad. 🙂

Cloud Control 12cR2 : Is it really release 2?

Oracle Cloud Control 12cR2 is installed and merrily monitoring one of the test 11gR2 databases running on HP-UX. I’ll probably leave it like that until I come back from Oracle OpenWorld. I don’t want to change the entire administration and monitoring infrastructure just as I leave for a couple of weeks.

As I’m re-familiarizing myself with the 12c way of doing things, I’ve been wondering if this really is a full “Release 2” product, or just 12cR1 with Bundle Patch 2. Not surprisingly, one of my readers asked the same question, pointing out the version 12.1.0.2 does not look consistent with a “Release 2” product, which would typically be 12.2.0.1.

I take Oracle version numbers with a pinch of salt. I’m currently using WebLogic version 10.3.5, which is 11g WebLogic. 🙂 At least WebLogic 12c has been versioned appropriately. 🙂

So why the 12cR2 branding, when 12cR1 of the database hasn’t been released yet? My guess is this is a marketing move for one very specific reason. One of the big marketing messages around Cloud Control was its ability manage clouds and allow you to charge customers based on their resource usage. While speaking recently to a representative of a large oracle customer/partner, I found out this functionality plain didn’t work, at least not with their selection of (latest version) Oracle products.

Assuming Cloud Control 12cR2 is now actually capable of delivering on this promise, that represents quite a big change that’s probably worthy of a re-brand, even if the version number doesn’t warrant it.

Of course, this is all speculation on my part. I’m not using it for managing clouds or charging customers. I’m just a regular DBA who likes to watch the performance page every few minutes while doing my administration in SQL*Plus. 🙂

Cheers

Tim…

Update: See Hans’ comment about the version number. He’s quite correct that this version falls in line with a new release in the GC/CC universe.

Enterprise Manager Cloud Control 12c Release 2 Installation…

I did an EM Cloud Control 12cR2 installation at work yesterday. The database repository was 11.2.0.3 on HP-UX and the middle tier was installed on RHEL 5.8. The installation was pretty much the same as the 12cR1 version. Over the next few days I’ll be testing out some of the features to decide if we can move across to it permanently.

Today I did two run throughs of single server installations on Oracle Linux 5.8 and 6.3. There are a couple of minor differences, but nothing to worry about. You can see what I did here:

The installations are a little small, so they are not too fast, but it’s good enough to test things out.

Cheers

Tim…

Update: It’s been a while since I used the 12c version, so I’ve had to relearn a few simple things. I thought I might as well write the down in case it helps others.

Enterprise Manager Cloud Control 12c Release 2…

Enterprise Manager Cloud Control 12c Release 2 was released a couple of days ago for all the major platforms. That in itself is not news any more, but the fact we are going to trial it at work as a replacement for our 11g Grid Control installation is.

It’s in my low priority task list, so I’m not sure I’ll get it all sorted before OOW12, but it is something to look forward too. I know it’s tragic, but I’m quite excited. 🙂

Cheers

Tim…