Enterprise Manager Cloud Control 13.3 Silent Upgrade

A few days ago I put out a post called Enterprise Manager Cloud Control 13.3 Vagrant Build. In a comment on that post Dinesh said,

“Would like to see the silent upgrade from oem12c to oem13c upgrade post from you”

I normally try to keep on top of upgrades, so I’ve never done a jump bigger than one version, but I was checking through the documentation, and assuming it’s a supported start version, there isn’t much difference between the upgrades from most versions, so I thought I would give it a go. As I mentioned in the last post, I already had a GUI upgrade article.

I didn’t have a silent upgrade though. I do now. πŸ™‚

In order to try this out I did a Vagrant build of Cloud Control 12cR5, to give me an easy way to get a clean starting point. You can see that here. If you want to practice the upgrade, it’s a really easy way to do it.

I’ve left the agent upgrade using the Cloud Control interface, but there is an example of how to do it silently using WLST here. It’s pretty simple.

So not only did I not expect to be writing that post a few days ago, but I certainly didn’t expect to be writing this one. πŸ™‚



Why Automation Matters : Reducing the Cost of Failure

Recently I watched a video called The Future of Faster Enterprises by AWS Enterprise Strategist, Miriam McLemore. I think its a really good video, even if you don’t care about AWS or cloud in general. There is a wider message there.

One of the points Miriam raised was “Reducing the cost of failure”, which sparked a conversation between myself and a colleague. When you’re trying to improve the way you work, you are going to have to try new things. Not all of those things are going to work out. The important point is you try them, see if they work for you. If they do great. If they don’t, you throw them away and move on. Reducing the cost of failure is a really important part of encouraging the culture of experimentation needed for continuous improvement.

Recently I wrote a post called you have to keep working just to stand still. Now add to that the work required to move your company forward and I think you’ll see why any barrier to progress is a problem.

So what factors affect the cost of failure? Here are a few.

  • Lack of automation. If humans are involved in providing infrastructure, it’s going to increase the time it takes to set things up (see lost time), and they will get disgruntled when you ask them to throw it away 2 hours after you’ve got it. You need to be able to build and burn kit rapidly to have any hope of experimenting. This is why the focus on the automation part of flow in DevOps is so important, for both business as usual and experimentation.
  • Bloated waterfall process. If your company expects a detailed plan of action before you so much as fart, you are going to fail. You have to be agile. I’m not using the term agile in the, “I’m too lazy to plan”, sense. I mean proper agile.
  • Time. Your company has to value progress and be willing to allocate time to it. You can’t rely on the fact Beryl and Bert go home every night and no-life their way through learning something new, so the business can reap the benefit of it for free. Yes that happens, but companies that rely on it will fail.
  • Be accepting of failure. I’m not talking about being happy to be rubbish. I’m not talking about being accepting of failure in well defined business as usual (BAU) work. I’m talking about being accepting of failure during experimentation. Not everything will work. Not everything will be the right solution for you or your company. You have to be willing to try and fail or you will fall at the first hurdle.

Check out the rest of the series here.



Technology : You have to keep working just to stand still!

This is a topic of conversation that has come up a lot recently, both at work and whilst updating the computers for family members, so I thought I would write something down.

I don’t think people realise how much work it takes to stand still in the technology world.

When we think about the pieces that make up your typical on-prem web application running on virtual machines, what does standing still mean? To me it means the following.

  • Database : A regular patching cycle for the database and the operating system it sits on. Also, upgrades/rebuilds as required.
  • Application Server : See database.
  • Web Layer : See database.
  • Load Balancer : Regular patching of the appliance, and replacements/upgrades as required.

Now multiply all that up for all the projects you are working on. That represents a substantial amount of work, whether you call is Business As Usual (BAU) or Internal Projects. What’s worse, there is no “visible benefit” from this work. Most users won’t have a clue it is happening, as they won’t get a new screen or a new widget to play with. It’s pretty much invisible, but it has to happen, just to remain static.

At this point I can hear people saying, “But standing still is literally doing nothing, so what are you talking about?” Well, if today I have a fully patched system using supported versions of all software, to stand still I have to remain on a fully patched system using supported versions of all software. If that means upgrades or rebuilds of kit, so be it.

Remember, if I do nothing at all, I’m no longer standing still, I’m moving backwards!

Think about that for a second. To stand still I’ve got to learn all the new stuff so I can upgrade to 19c and get long term support for my databases, even if I didn’t want to. Same goes for other products. Even if I use none of the new features. There is a big investment needed by a company, and for you personally, just to stand still. Now breaking new ground, well that’s a whole different ball game… πŸ™‚

So what are the solutions:

  • It helps if you recognise the problem in the first place. Far too many people think doing nothing is standing still, when it’s not.
  • Automation will help you stay on top of things. Reliable and repeatable processes make keeping things up to date a lot simpler. Automated testing is the icing on the cake here.
  • Cloud? Platform as a Service (PaaS), when it is done right, can help you keep on top of things. Having a service where you don’t have to worry about OS, DB and app server patches, because it’s all handled by the platform is a big bonus.



Some related posts:

Email Problems : A Quick Update…

I just wanted to write a quick post about an email problem I’ve discovered recently.

My website runs on AWS, but emails to my normal email address get directed by DNS to a mailbox hosted by another company. I use Gmail as my mail client, so it picks up the posts from that mailbox. Emails that are sent directly work fine, but I recently noticed those sent from my web server were failing. There are several parts of the site that send emails so I know what is going on and can respond. None of those have been working for some time. That issue is now fixed.

Some of the things that generate these emails (like comments) get picked up in my daily/weekly workflow, so I didn’t really notice a dramatic change, but some only get to me via these emails from the site, so they were hidden in the mail queue of doom.

By the time I had discovered the problem I had several thousand emails sitting in the queue. I started to work through them, but realised it was too big a job. I picked a random sample of mails and could see there were a mixture of topics including questions, messages of support and offers I simply could not refuse. πŸ™‚ I decided the only way to move forward was to delete the lot. It would have taken me weeks to get through the backlog.

So in this post I would just like to say a few things.

  • To those people that wrote to send their support for what I do on the site, thank you very much, and I’m sorry I’ve not been able to respond personally. I normally reply to these messages with a quick thank you, so I hope you don’t think I’m an arrogant prick and just ignoring you.
  • To the people who sent questions, I’m sorry your question was never answered. Please remember, I have a full time job and I do this for fun in my spare time. There is one of me, and literally thousands of you. I closed my forum because the workload was too big to cope with. If I have to choose between spending time producing new content, or answering your questions, I’m going to pick producing new content. Sorry. πŸ™‚
  • To those people offering me business opportunities. No.
  • To those people offering me various services of a questionable nature. No.

I hope there was nothing super important. Once again, sorry!



PS. Before you ask, I am going to check my mail queue from time to time in future, just in case something like this happens again… πŸ™‚

Cloud : Who are the gatekeepers now?

There’s something you might consider sinister lurking in the cloud, and it might cause a big disruption in who are considered the gatekeepers of your company’s services. I’ve mentioned governance in passing before, but maybe it’s time for me to do some thinking out loud to get this straight in my own head.

In the on-prem world the IT departments tend to be the gatekeepers, because they are responsible for provisioning, developing and maintaining the systems. If you want some new infrastructure or a new application, you have to go and ask IT, so it’s pretty easy for them to keep a handle on what is going on and stay in control.

Infrastructure as a Service (IaaS)

The initial move to the cloud didn’t really change this. Most people who proudly proclaimed they had moved to the cloud were using Infrastructure as a Service (IaaS), and were really just using the cloud provider as a basic hosting company. I’ve never really considered this cloud. Yes, you get some flexibility in resource allocation, but it’s pretty much what we’ve always done with hosting companies. It’s just “other people’s servers”. As far as IaaS goes, the gatekeepers are still the same, because you need all/most of the same skills to plan, setup and maintain such systems.

Platform as a Service (PaaS)

When we start talking about Platform as a Service (PaaS), things start to get a little bit trickier. The early days of PaaS weren’t a great deal different to IaaS, as some of the PaaS services weren’t what I would call platforms. They were glorified IaaS, with pre-installed software you had to manage yourself. With the emergence of proper platforms, which automate much of the day-to-day drudgery, things started to shift. A developer could request a database without having to speak to the DBAs, sysadmins, virtualisation and network folks. You can of course question the logic of that, but it’s an option and there is the beginning of a power shift.

When we start talking about IoT and Serverless platforms things change big-time. The chances are the gatekeeper will be the budget holder, since you will be charged on a per request basis, and probably have to set a maximum spend per unit time to keep things under control. Depending on how your company manages departmental budgets, the gatekeeper could be whoever has some spare cash this quarter…

Software as a Service (SaaS)

Software as a Service (SaaS) possibly presents the biggest challenge for traditional on-prem IT departments, as the business can literally go out and pick the product they want, without so much of a thought for what IT think about it. Once they’ve spent the money, they will probably come to IT and expect them to magic up all the data integrations to make things work as expected. Also, once that money has been spent, good luck trying to persuade people they backed the wrong horse. SaaS puts the business users directly in the driving seat.


It would be naive to think any movement to the cloud (IaaS, PaaS or SaaS) could be done independently of an existing IT department, but the tide is turning.

The IT world has changed. The traditional power bases are eroding, and you’ve got to adapt to survive. Every time you say “No”, without offering an alternative solution, you’re helping to make yourself redundant. Every time you say, “We will need to investigate it”, as a delaying tactic, you’re helping to make yourself redundant. Every time you ignore new development and delivery pipelines and platforms, you are sending yourself to an early retirement. I’m not saying jump on every bandwagon, but you need to be aware of them, and why they may or may not be useful to you and your company.

Recently I heard someone utter the phrase, “you’re not the only hotel in town”. I love that, and it should be a wake-up call for any traditional IT departments and clouds deniers.

It’s natural selection baby! Adapt or die!



The problem with Googling for solutions

I started to write a post, then realised I’ve already written it several times before, with the most coherent of them here.

So instead I’m going to change it up a little and tell a story.

I’m a generalist, and as you will know it’s really hard to be good at everything, so clearly there are some things I’m “not so good at”. Like most people, I use Google a lot, and my Google-fu is strong.

A couple of weeks ago we did a security scan of an existing system, which revealed some security flaws. It was a non-Oracle product, so I didn’t have a recipe to follow and I started Googling for solutions. The product in question is very popular, and there were lots of responses to my Google search, with most of the top results coming from Stack Exchange (Stack Overflow). Happy days I thought, as the Stack Exchange sites is effectively peer-reviewed, in that the correct answers are usually up-voted.

I looked at the first few different threads and people were saying the same thing. The highest up-voted answer on each thread gave a very direct and simple parameter value to solve the issue I had, so I was happy…

I followed the advice, set the parameter, restarted the service and tested. It didn’t do what everyone claimed it would. Armed with the parameter name, I searched the product documentation, and clearly the parameter didn’t do what the Stack Exchange answers said it did.

That seemed very odd, so I assumed these must be answers that were correct for an old version of the product. I checked the docs for previous versions. Same result. After reading the docs I found the real answer, implemented it, tested it and it worked.

What is really worrying about this is the answers on several threads on Stack Exchange were wrong. Those incorrect answers had been up-voted by lots of people, which suggests they agreed with the answer, even though these solutions could *never* have worked. So this seems to indicate one of two things to me.

  • People read the answer, it sounded plausible, which it did, so they up-voted it without trying it.
  • People had actually used this solution, thought it was the right solution and up-voted it, but clearly never tested their system or they would have seen it didn’t work and they still have the same security flaw.

One of the things I say in that post linked above is.

“Remember, even when you have built up a list of trusted sources, you should still constantly test what they say. Everyone can make mistakes.”

That’s really important because the internet is full of great information, but it’s also full of bullshit. Being able to tell the difference is really important, and the only way to do that is to test it, or do further research if it’s something you can’t test for yourself…



WordPress 5.2 “Jaco”

WordPress 5.2 “Jaco” was released yesterday.

For the most part these updates pass me by as I’m not too interested in WordPress features. I just write stuff and publish it. Simple as that. So often I just apply them and forget about them.

One thing that did catch my eye was the mentioned improvement to the Site Health feature, available from “Tools > Site Health”.

After upgrading 5 different WordPress installations, I checked the Site Health on this blog and there were a few things flagged. It turned out I wasn’t on the latest version of PHP, I was on an older version of PHP7, and I had one mandatory and two optional modules missing. I fixed all that with the following.

wget http://rpms.remirepo.net/enterprise/remi-release-7.rpm
rpm -Uvh remi-release-7.rpm
yum-config-manager --enable remi-php73
yum update -y
yum install -y php-gd php-bcmath php-pecl-imagick

If you were on the website or blog this morning, you may have had a bit of a funky experience as I restarted Apache a few times. πŸ™‚

All the Site Health tests are passed now. Happy days.

The WordPress update itself went smoothly. I’m guessing by the time you read this, your site may have auto-upgraded anyway.



Oracle Database 19c (19.3) : Installations, RAC, Data Guard etc.

A few weeks ago I put out a post about 19c installations and all that good stuff. That post was using the 19.2 release, which was not the official on-prem release of the product. Now Oracle 19c (19.3) has dropped and is available from here, and here, this post is just to say all those builds have been updated to use this 19.3 release. I also noticed the 19c preinstall package is available from yum.oracle.com.

Not surprisingly, I took the Vagrant and Docker builds I did for 19.2 and just changed the environment variables holding the software zip names, and everything worked just fine. Here are the associated articles, with those minor edits to reflect this version change.

I’ve committed a whole bunch of stuff to GitHub.

  • Vagrant build of 19c on OL7 with APEX and ORDS (here).
  • Vagrant build of 19c on Fedora 29 (here).
  • Vagrant hands-off build of 19c RAC on OL7 (here).
  • Vagrant hands-off build of 19c Data Guard on OL7 (here).
  • Docker 19c on OL7 build (here).
  • Docker 19c RPM on OL7 build (here).
  • Docker compose (here) and swarm (here) stacks.

Automation is awesome! πŸ™‚



Java and Tomcat Updates : Vagrant and Docker

Yesterday was another update frenzy.

I already mentioned the update to VirtualBox 6.0.6 in yesterday’s post.

At the same time we got the quarterly updates to Java and I noticed a new version of Tomcat, so I downloaded OpenJDK 12.0.1 and Tomcat 9.0.19 and added them to my Vagrant and Docker builds.

If you are interested in this stuff, you can check it out here.

Remember, this is just my playground stuff. If you find it useful, that’s great. If not, there are plenty of other people messing about with this stuff. πŸ™‚



The Art of Miscommunication : About that thing from last week…

One of the things I get all the time is an apparent continuation of a conversation I was in some time ago, and I’m meant to pick up the thread without any explanation or context.

Depending on the people you are dealing with, this can be quite an easy trap to fall into. As a DBA that looks after many instances of multiple database engines, middle tier technologies and load balancers, I tend to get pulled into loads of conversations. For the person in question, let’s say a developer working on one project, they see this interaction as a one-to-one relationship between them and me, but for me it’s a one-to-many, as I’m dealing with many such conversations at the same time.

When I’m in that position I’ve stopped trying to figure it out, and now the conversation goes something like this.

  • Person : You know that thing we were talking about last week?
  • Me : No.
  • Person : You know, that thing…
  • Me : I literally have no idea. Please explain…

In some situations, it turns out it wasn’t even me they had the conversation with, or I wasn’t copied into the emails. A fact that only becomes evident when they take the time to order their thoughts…

I’m generally pretty happy to help people out, but I’m not going to go through hypnotherapy to pull back distant memories in order to continue a conversation you think I should remember. As you saw from my previous post on this subject, I expect each interaction to be self-contained. If there is any context necessary, it should be in the interaction itself. I shouldn’t need the skills of Professor X to pull it out of your head…

So before you pick up the phone, start typing on chat, or begin an email, take a second to plan the conversation in your own mind.

  • What introduction is necessary to get people up to speed?
  • Is there any prior knowledge I’m assuming, that I probably shouldn’t?
  • What is the main purpose of this interaction?
  • What are the outcomes I’m looking for?

It will only take a couple of seconds to figure this out. I’m not asking you to spend an hour preparing for a five minute chat, but don’t just launch into a stream of consciousness and expect everyone else to jump back in at exactly the same spot they left.