Upgrade Cloud Control 12cR4 to 12cR5

 

em-12cA couple of weeks ago I wrote a post about doing a Cloud Control 12cR5 installation and said I would be testing the upgrade from 12cR4. I’ve now done that.

The upgrade documentation is quite extensive and the prerequisites are going to be different depending on the database and cloud control versions you are starting with, so this is no way a “recommended” way to do the upgrade. Each one will need to be approached on a case-by-case basis. It’s just meant to give a flavour of what you have to do.

Suffice to say, it worked fine for me. :)

Cheers

Tim…

Oracle Enterprise Manager Cloud Control 12c Release 5 (12.1.0.5) : My first two installations

 

em-12cI’ve done a couple of play installations of EM12c 12.1.0.5, just to get a feel for it. You can see the result of that here.

From an installation perspective, everything was pretty similar to the previous releases. I tried the installation on both OL5 and OL6, in both cases using 12c as the database repository. No dramas there.

A couple of things of note.

  1. The 12c repository template database is a Non-CDB architecture.
  2. The Weblogic installation uses Java6.

Interesting…

The next step is to try some upgrades from EM 12.1.0.4 (on DB 11.2.0.4) to EM 12.1.0.5, which is what I’ll need for my upgrades at work. The testing is quite time consuming and boring, but it’s got to be done before I can unleash this on the company. :)

Cheers

Tim…

PS. Remember to download from edelivery.oracle.com (in a couple of days) for your production installations. Apparently there is a difference to the license agreement.

EM Cloud Control 12c : The 24 Hour DBA

 

I think I’ve lived through all the ages of Enterprise Manager. I used the Java console version back in the days when admitting you used it got you excommunicated from the church of DBA. I lived through the difficult birth of the web-based Grid Control. I’ve been there since the start of Cloud Control. I’ll no doubt be there when it is renamed to Big Data Cloud Pixie Dust Manager (As A Service).

I was walking from the pool to work this morning, checking my emails on my phone and it struck me (not for the first time) that I’m pretty much a 24 hour DBA these days. I’m not paid to be on call, I’m just a 9-5 guy, but all my Cloud Control notifications come through to my phone and tablet. I know when backups have completed (or failed). I know when a Tnsping takes too long. I know when we have storage issues. I know all this because Cloud Control tells me.

Now you might look on this as a bad thing, but being the control freak I am, I prefer to get a message on a Sunday telling me something is broken, hop on the computer and fix it there and then, rather than coming in on Monday to a complete sh*t-storm of users complaining. I’m not paid to do it, but that’s the way I roll.

While walking down memory lane I was thinking about all the scripting I used to do to check all this stuff. Endless amounts of shell scripts to check services and backups etc. I don’t do hardly any of that these days. Cloud Control handles all that.

We are a pretty small Oracle shop, but I think life would be a whole lot more difficult without Cloud Control. I’ve mentioned this a number of times, but it’s worth saying again… If you have more than a handful of Oracle databases, you really should be using Cloud Control these days. It’s as simple as that.

Just in case you are wondering, this is how our infrastructure looks this morning… :)

em-all-green

Cheers

Tim…

Enterprise Manager Cloud Control 12cR4 Production Upgrade

 

I’ve already written about the 12cR3 to 12cR4 upgrade here. I did a few run through’s at home to practice it and it all seemed good.

Setting The Scene

Just to set the scene, for our production environment we run Cloud Control in a VMware virtual machine, using Oracle Linux 6.5 as the guest OS. With that setup, we can use a simple installation (DB and OMS on the same VM) and use VMware to provide our failover, rather than having to worry about multiple OMS installations and any DB failover technology etc. If there’s one thing I’ve learned about Cloud Control, it’s Keep It Simple Stupid (KISS)! As far as our managed servers go, most of our databases and all our middle tier stuff runs on VMware and Oracle Linux too. We have a handful of things still hanging around on HP-UX and Solaris, which will hopefully be migrated soon…

Upgrade Attempt 1 : Non-Starter

Yesterday I started the upgrade of our production system. Pretty much straight out of the blocks I hit a road block. It didn’t like the agents running on our HP-UX servers. The upgrades of the HP-UX agents are so painful. Every time so far I’ve had to reinstall them. As a result, I didn’t bother to upgrade them last time and kept running with the previous version of the agents. The upgrade wouldn’t have anything to do with that, so I forgot about the Cloud Control upgrade and I spent yesterday attempting to upgrade the HP-UX agents to 12cR3, before I could attempt the 12cR4 Cloud Control upgrade.

As usual, the upgrade of the agents on HP-UX involved me uninstalling, removing all the targets, installing, discovering all the targets and setting up the backups etc. Not all of it is scripted yet, so it is an annoying and painful process. I’m not sure if other HP-UX users suffer this, but it seems pretty consistently bad for us. The sooner we get rid of these straggling HP-UX servers the better!

So this wasn’t so much a failure of the upgrade. It was really down to me being lazy and not bothering to upgrade some agents.

Fast forward to this morning and I was actually ready to start the upgrade. :)

Upgrade Attempt 2 : Success

With the 12cR3 agents in place on HP-UX, the upgrade ran past that step with no problems and on to the main body of the installation. The install and upgrade were textbook.

I’ve upgraded the agent on the cloud control server, but I’m not going to upgrade any of the other agents until I know things are working fine.

Happy days!

Cheers

Tim…

Enterprise Manager Cloud Control 12c Release 4 (12cR4) Articles

 

I’ve started to play around with Enterprise Manager Cloud Control 12R4. The clean installations on Oracle Linux 5 and Oracle Linux 6 were really easy. You can see how I did them here.

I used 11.2.0.4 as the database repository. I’ll probably have a go with a 12c database in future, which is now supported, but my main focus this time was to check out something similar to what I have at work. Forgive my caution, but I’ll not be using 12c database for my EM repository for a while yet. Cloud Control is too important to risk…

I wasn’t going to bother with an upgrade article because  wrote a good blog post about the upgrade here. Whilst going through the upgrade, there were a few things I needed to make extra notes on, since I have an terrible memory, so it ended up as a separate article.

Ultimately, you are going to need to read the upgrade docs because there are lots of caveats to think about, depending on what options you use.

Previous upgrades at work were a pain, but we had a more complicated installation with multiple management servers and a separate database. More recently we switched to running a simple configuration (DB and OMS on the same server) on an Oracle Linux VM. That makes doing a trial run of the upgrade at home a more realistic test. Whether it’s an improved Cloud Control upgrade process or the fact this is a single machine setup, the upgrade was really easy. Now that I’ve practiced the upgrade at home, I’m feeling relatively confident doing the upgrade in production. I’ll probably leave it until I get back from Bulgaria (BGOUG). It’s not really fair to change everything and then leave the country… :)

Cheers

Tim…

Cloud Control 12cR3, Oracle Linux and VMware

 

I mentioned some time ago that I was pushing my current company to move much of their gear on to VMware, mostly because of poor resource utilization on many of the servers. That process is still under way.

One thing I wanted to mention specifically was our use of Cloud Control 12cR3. Up until recently, we were using physical kit for this. We had an 11.2 database on HP-UX, With HA provided by HP Service Guard. We had two management servers on physical kit running RHEL5 pointing at this Service Guard package to give us some resiliency in of the OMS. It worked, but it was over complicated and I was never really happy with it for a number of reasons:

  • HP-UX for the databases : I know some of you guys love it, but I don’t.
  • Two management servers : Seems like a waste of kit to me. We either have them on their own boxes and waste lots of resources, or have multiple installations on those boxes, which adds to complexity and management of the kit.
  • RHEL : Why pay for RHEL when we can use Oracle Linux and decide for ourselves if we want to pay for the extra features support gives us, or just use it for free?

So what are we running now? We have one VMware VM, running Oracle Linux 6. That has both the Oracle 11.2 database for the repository and the Cloud Control 12cR3 OMS running on it. We use VMware functionality for the HA of this system.

Why do I like this situation?

  • Cloud Control is a complicated beast and I am a big fan of KISS (Keep It Simple Stupid). Having everything on a single VM is about as simple as it gets.
  • If I am using Cloud Control in this way I pay nothing for the database repository. As soon as you start thinking about RAC or Data Guard to protect your repository you have to pay for Enterprise Edition licenses.
  • Using VMware HA functionality gives us good enough HA for our purposes. We can failover or live migrate between hosts in the data centre, or between data centres.
  • We can clone the whole installation in a few minutes and use that as a base for upgrades. If something goes wrong, we just flip back.

While I was at Oracle OpenWorld I discussed this a number of times and it seems it is a very common approach.

Another thing that came out of those discussions is many people still misunderstand what Oracle Linux is and the support status of Oracle Linux, and more specifically UEK, on VMware. Suffice to say, it’s all supported, as discussed in my Oracle Linux : Frequently Asked Questions article.

If you are struggling to decide how best to run Cloud Control in your organization, I would recommend using a virtual environment (Oracle VM or VMware) and run it on Oracle Linux 6.

Cheers

Tim…

Cloud Control 12c Database Backup Jobs (Continued)

 

I’ve been rather critical of the way Cloud Control handles database backup jobs, as can be seen in these two previous posts.

Yesterday I found out I schedule database backups in Cloud Control the “wrong way”…

So typically, when I am sorting out a new database, I do something like this:

  • If the host doesn’t already have an agent, push one out the to server.
  • Discover the targets. For existing hosts, this may just be discovering a new database.
  • Tell Cloud Control to used the RMAN catalog for backups. (Oracle Database > Availability > Backup & Recovery > Recovery Catalog Settings)
  • Set the default location for the backups and any other stuff. (Oracle Database > Availability > Backup & Recovery > Backups Settings)
  • Finally schedule the backup. (Oracle Database > Availability > Backup & Recovery > Schedule Backups…)

Notice how all the backup-related things come from the same location (Oracle Database > Availability > Backup & Recovery). Obvious right? Wrong!

So it turns out, if you schedule the backup using the (Oracle Database > Availability > Backup & Recovery > Schedule Backups…) screen, you do schedule a backup job, it works and it is visible in the (Enterprise > Job > Activity) screen, but it is not a “real” Cloud Control job. It is actually created using the job type of “Backup”, which is not a supported job type, rather than the job type “RMANScript”.

If you schedule a backup in this way, you can’t:

  • use the job library.
  • do a create-like of an existing job.
  • edit most of the details about the job after it’s created, including the RMAN script.
  • describe the job using EMCLI.

So what is the solution? Well, you do all the prep in the same way, but you don’t actually schedule the job using the (Oracle Database > Availability > Backup & Recovery > Schedule Backups…) screen. You do it directly from the (Enterprise > Job > Activity) screen. If you do it from here, then you get to pick the “RMANScript” job type and suddenly everything works much better and things seem a lot more consistent. Basically, you’ve created a “real” Cloud Control job.

So in my latest round of [enchancement requests | bug reports] I got the answer back from development that I was doing it wrong, which is good to know, and they want to close the SR. My response to that was, why do you let me do it the “wrong way”? Why does the (Oracle Database > Availability > Backup & Recovery > Schedule Backups…) screen not schedule a “real” Cloud Control job? It seems odd to me that you build a product that lets you do things inconsistently. It’s probably a throw back to the old DBConsole stuff, but that doesn’t make it any less annoying.

I’m going to recreate all my backup jobs to make them use the “RMANScript” job type, which will solve my EMCLI issues, but I really think this mess should get cleaned up. I wonder how many other people out there are not creating “real” Cloud Control jobs for their database backups?

I’m going add this blog post to my SR and see what the response is…

Cheers

Tim…

Oracle Cloud Control EMCLI : First Thoughts

 

Whilst waiting for a new version of Cloud Control, which will hopefully include my job scheduler enhancement requests, I decided to see if I could solve my problem using the command line interface (EMCLI). That spawned this very basic article.

I was initially really excited by the EMCLI, planning to pretty much replace my job creation with EMCLI scripts, but there seem to be a bunch of discrepancies with the EMCLI compared to the regular Cloud Control interface. Now I admit I’m a newbie at this, but here’s what I’ve seen so far.

The built in help is fine, sometimes prompting you with other commands to get more information, but sometimes it leaves quite important stuff out. For example, when you do a “describe_job_type” for the job type “RMANScript”, it does not include the possible schedule information, so it appears you can only schedule one-off jobs. A little digging and you find the scheduling is possible, and seems to work OK. It’s a little annoying this isn’t included in the property file template though.

It would seem like the easiest way to figure out how to schedule a job using EMCLI would be to create a DB backup job manually, the use the “describe_job” verb to figure out what the job properties are. The problem is, you can’t describe a job with the job type of “Backup”.

C:\emcli>emcli describe_job -name=test_db_backup
job type [Backup] not supported in EMCLI.

C:\emcli>

But wait a minute. When you create a DB backup from EMCLI it is created with the job type of “RMANScript”, which works with “describe_job” just fine. So why do the two methods for creating a job to do the same thing use a different job type? Even if that were sensible for some reason, why does the job type of “Backup” not work with the “describe_job” verb?

My next issue was I can create and delete a one-off job using “create_job” and “delete_job” respectively, but as soon as I create a scheduled job with a repeating schedule, the “delete_job” verb seems to be useless, because as long as there is a scheduled run in the future, you get the following message.

C:\emcli>emcli delete_job -name=test_db_backup
Error: Cannot delete a job with active executions. Before deletion, the job status must be one of STOPPED,COMPLETED,ABORTED, or FAILED.

C:\emcli>

Remember, this is not a running job, just a job that is scheduled to run in the future. If you try to delete the job from the Cloud Control interface it works fine…

As I said before, I’m a total newbie at this EMCLI stuff, so it may be my (lack of) understanding that is wrong, but there seems to be a number of inconsistencies that make it really annoying. I’m raising enhancement requests as I find these issues, so hopefully things will be resolved in future releases, or the responses will explain why my issues are due to be being a dumb-ass. :)

Moaning aside, I think the EMCLI stuff has got a lot of potential. Personally, I would like to script a lot of the configuration, giving the option of clean installs of new versions as well as upgrades.

Cheers

Tim…

Oracle Cloud Control 12c Release 3 Installation (EM12cR3)…

 

With all the excitement about Oracle Database 12c being released, you may have missed the release of Cloud Control 12c Release 3. It’s available for download from OTN. All the usual ports are available. This is the version you are going to need if you want to monitor 12c databases in your organisation.

I did a quick run through the Cloud Control 12c Release 3 installation yesterday on both OL5 and OL6. As I suspected, there is pretty much no change in the installation process compared to the previous release.

I haven’t tried an upgrade yet, so I don’t know how easy that is. Perhaps someone else can comment? :)

Note. Oracle Database 12c is not a supported repository database version, so you are still going to need Oracle 11g for this installation.

Cheers

Tim…

I love Cloud Control 12c, but the job management is really annoying!

 

Cloud Control 12c is a great product. Yes, it is suffering from bloat, but generally it is a really great tool. I’m always encouraging people to ditch the DBControl and switch to Cloud Control!

Having said that, one area that annoys the hell out of me is the job management, which feels really clumsy. I started to write this post, then felt a bit guilty because I hadn’t actually bothered to raise an enhancement request, so that’s what I’ve just done!

My main gripes are the following.

  • Editing Jobs: You can edit some parameters of jobs, but not all. This is really frustrating and often means you have to delete and recreate a job in order to make a minor change.
  • Create Like: There is still no support for “Create Like” of a database backup job. Most of our jobs are database backup jobs, so this omission is quite annoying.
  • Run Now: There is no “Run Now” type of functionality on the scheduled jobs. You either have to create a new one-off job (with out Create Like), or alter the schedule, then remember to alter it back once the job has finished. A “Run Now” option that doesn’t affect the normal schedule would be really handy.
  • Backup Destination: If you use “CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT ‘/path/to/backup/%U’;” from RMAN, or use the Backup Settings screen in Cloud Control, which does the same thing, the path used for the backup works fine. If on the other hand you use the “Over Default Settings” option while creating the database backup job, the settings seem to be completely ignored. This should either work, or be removed as an option because it catches me out all the time!

OK, the last one is a bug, not an enhancement request, but I thought I would throw that into the mix anyway. :)

We are in the process of upgrading and moving databases at the moment, so amending the backup jobs is a big thing for us. Yesterday afternoon, evening and night was littered with a chorus of expletives in relation to Cloud Control job management, which gave me the activation energy to post this rant. :)

Note. Even as I rant, I am still convinced centralising your jobs in Cloud Control is a good thing!

Cheers

Tim…

Update: My enhancement request is being split into three separate SRs and they are being put forward to the development team as formal enhancement requests. Fingers crossed they will get implemented.

The fourth point is a “problem between keyboard and chair”. I misunderstood what this feature was supposed to do. My bad. :)