Autonomous Database : “Hand-tuning doesn’t scale”

I was at a talk by Chris Thalinger at Oracle Code One called “Performance tuning Twitter services with Graal and machine learning”. One of the things he said was, “Hand-tuning doesn’t scale”, and it brought into focus some of the things that have been going on in the Autonomous Database, which is closer to my world. 🙂

In my post called It’s not all about you! I discussed the reaction to a new feature mentioned in the ACE Director briefing. It has been spoken about publicly now, so I guess I’m allowed to mention it by name. The feature in question was Automatic Index Tuning that (insert Safe Harbour slide) might be in Oracle 19c, or in an autonomous database cloud service in the future. Once this feature was mentioned, the list of questions started to pile up, before we even knew what it was or how it was implemented. I mentioned my own reaction to this specific feature, but let’s look at this in the broader sense of autonomous services generally.

As I mentioned, watching Chris’ session brought all this into focus for me. Sorry if I’m stating the obvious, but here goes.

  • Even if I were capable of doing a better job than an automatic performance tuning feature, and I’m not sure I can, that is just me. Is everyone else I work with at my level of understanding or better? Is everyone else who works with the database across the world at my level of understanding or better? If the answer to that is no, then there is a need for feature X, whatever it is.
  • Let’s say I have a group of really skilled people that can do better than automatic feature X. Are they constantly looking at the system, trying to get the best performance possible, or are they working on hundreds or thousands of different targets, and actually spending very little time on each? As their workload grows, which it invariably will, will they be able to spend more or less time looking at each specific feature?

I know there are some consultants that get to go in and solve specific problems on specific systems, and maybe those folks will look down on automatic performance tuning features, but I have to look after loads of disparate systems and I get 30 seconds to get something done before I have to move on. I like to think I’m pretty good at Oracle database stuff, but I need all the help I can get if I want to keep things running smoothly.

When a new automatic feature is announced we always get super intense about it, which usually results in a lot of wailing and gnashing of teeth. Sometimes this is for very good reason, as the early incarnations of some features have been problematic, but over time they often become the norm. Think about the following, and what life would be like without them…

For some people reading this, they may never have experienced life without these features. Believe me, it wasn’t pretty! 🙂

Whether it’s a specific automatic feature, like Automatic Index Tuning, or a grander vision, like the Autonomous Database family of cloud services, this is part of the natural evolution of the database. At *some point* in the future I can see all my databases running on the cloud and all of them being some form of autonomous service, regardless of which cloud provider is running them.

Check out the rest of the series here.

Cheers

Tim…

PS. I hope people understand the spirit of what I’m saying, but I feel the need to include a few statements, as some people on Twitter seemed to get the wrong end of the stick.

  • I’m not saying you can do a rubbish job and leave it up to an automatic tuning feature to fix your crap application. Bad software always runs badly, no matter what you do with it. You might be able to mask some of the problems, but you don’t fix them.
  • I’m not suggesting the development process shouldn’t include proper testing, including unit, integration, UAT and performance testing. See previous point.
  • The more you know about your platform, the better job you can do, even if you have automatic features to help you.

Oracle Enterprise Manager Cloud Control 13c Release 3 (13.3.0.0) Upgrade

A few months ago I wrote about the installation and upgrade Oracle Enterprise Manager Cloud Control 13c Release 3 (13.3.0.0).

At the time I did a clean install and an example upgrade from 13.2 to 13.3. The idea behind the upgrade was basically to practice what I needed to do at work.

Just before I left for OpenWorld I got our virtualization folks to give me a clone of the production Cloud Control VM and I ran a practice upgrade on that. It’s important to do a “real” run through, as sometimes you hit issues you don’t see when upgrading from a clean installation of the previous version. In the past the upgrade of the clean installation of the previous version has worked fine, but the real upgrade failed the prerequisite checks as some of the agents or plugins were too old. The latest test on the clone worked fine, so we had the green light to do the production upgrade.

Post OOW18, my first job on returning to work was to get Cloud Control upgraded. I repeated the process I had done on the clone and it went fine.

In a funny coincidence, while I was doing the upgrade someone retweeted the blog post from a few months ago. Weird.

As a reminder, here are the 13.3 articles.

Cheers

Tim…

Oracle OpenWorld and Code One 2018 : It’s a Wrap!

Here are some top-level thoughts about what happened over the week at Oracle OpenWorld and Oracle Code One.

  • Oracle Cloud Infrastructure (OCI) has come of age. I spoke to a bunch of non-Oracle folks who are using OCI for real workloads and the general perception was that it delivers. Of course the Oracle folks are going to say this, which I why I didn’t ask them. 🙂 It’s taken some time for Oracle to get to this point, but they finally seem to have the infrastructure to move forward with the rest of their services.
  • There was a continued focus on automation and the autonomous services. I understand some people seem conflicted about this, but this is a continuation of what’s been happening over the last 20 years. As I’ve said before, what we have now is not the destination. It’s the start (of this part) of the journey.
  • All the base (on-prem) products continue to evolve. As has been the case in recent years, the evolution of Oracle products seems to be based on the features Oracle themselves need to improve their cloud services, but that is fine as it’s making the products better for us on-prem customers too.
  • Oracle’s support of Cloud Native Computing Foundation (CNCF) is interesting. Allowing people to do the same thing on-prem and in the cloud is good for a couple of reasons. It helps people in the migration from on-prem to cloud. It also stops people feeling trapped on a cloud service. The former is great for cloud providers from an adoption perspective, but the latter is a little scary I guess. It’s important cloud providers don’t give people a reason to want to move off their services!
  • Speaking to non-Oracle folks, there is a perception that Oracle still lags behind on the customer service side of things. I wrote about this a couple of years ago in a post called Oracle: Tech Company or Service Company? I hope Oracle focus on this. There is no point having great tech if people don’t feel confident about using it because of the customer service side of things.
  • I’m a little confused by the re-branding of  “Oracle Groundbreakers” and the “Oracle Groundbreaker Ambassadors”. I miss Oracle Technology Network (OTN). 🙂

Here are the posts related to this trip.

Thanks to the Oracle ACE Program and the Oracle Groundbreaker Ambassadors Program for making this trip possible for me. Let’s see what the coming year brings…

Cheers

Tim…

Oracle OpenWorld and Code One 2018 : The Journey Home

I had the morning in the hotel, trying to catch up on things I missed during this trip and I did a quick visit to the gym. At about 14:00 I checked out of the hotel and got the Bart to the airport.

My boarding pass looked like it said terminal “1”, which sounded a bit odd, but I went there to check anyway. It turned out it was terminal “I”, for “international”, so I got the monorail back to the original place I had started. I got in the queue for bag drop and Chris Thalinger was at the opposite bag drop grinning at me. He was late for his flight. We had a bit of a chat while moving through security, then he went off to catch get his flight. I was still in plenty of time, so I walked down to by boarding gate, to find my flight had been delayed by an hour, so then I was really early. 🙁

The flight from San Francisco to Dublin took about 9.5 hours, which was nearly an hour quicker than it was listed. They were trying to make up for lost time, and we had a medical emergency on the plane, and this time it wasn’t me. After we landed we had to sit on the plane while the medics did their thing. We ended up about 30+ minutes late. I did a little head-nodding on the plane, but not really something I would call sleep. I couldn’t really watch films as there was so little room, my face was pretty much against the screen in front of me. I turned it off and listened to music instead, and stood at the back of the plane, getting in the way a lot.

I originally had a 5-ish hour layover in Dublin, but because of the delay to the first flight that was cut to a bit over 4 hours. Dublin airport is OK, but hanging around at any airport for more than a couple of hours is soul destroying. I could feel myself getting progressively more jittery as time passed. 🙂

The flight from Dublin to Birmingham was less than an hour, but felt like an eternity. I had a window seat on the plane, but someone was sitting in it, so got the aisle instead, which was a good thing.

After a short taxi ride I was home. I put on my first load of washing, and got in the bath (sorry for the mental image)… By the time I got out it was time for the second load of washing, then bed. I woke up in the morning, put on  the third load of washing, and cut my hair. It was only at this point that I started to feel remotely clean.

That’s another OpenWorld done. I’ll write a wrap up post once everything has distilled…

Cheers

Tim…

Oracle OpenWorld and Code One 2018 : Day 4 – Thursday

As is the case with a trip to San Francisco, I get into the time zone the day before I have to leave, which only serves to make the trip home and subsequent recovery from it harder. 🙂

I woke up at a more normal local time, did my blog for the previous day and headed down to the conference…

First up for me was Joel Kallman speaking about “Oracle Application Express 18 New Features”. As followers of the blog know, I’m the worst Application Express (APEX) developer that has ever existed, even though I’ve been using it since before it got named APEX. APEX is becoming a bigger thing for us at work, in no small part because I keep pushing out small applications using it. I wrote down a list of things I need to take a look at, so fingers crossed I will get a chance to do that soon.

Next up was Colm Divilly with “Kicking mod_plsql to the Curb for a Modern REST-Driven Application”. We use Oracle REST Data Services (ORDS) at work, but we also have a bunch of things lingering on mod_plsql, partly due to time/testing constraints and partly due to the use of the mod_plsql document upload/download functionality. Yes, we could have rewritten that, but… Well ORDS 18.3 has us covered, due to the inclusion of the “owa.docTable” parameter. I’m hoping I can try this out when I get back, and maybe rid myself of the mod_plsql dependencies once and for all. He talked about a bunch of other stuff, including database authentication, but the upload/download stuff will have the biggest impact on me personally.

Next up it was me with “DBA Does Docker“. Overall the talk went OK, but I got my timing kind-of wrong and spent too long on the demos, so I ran out of time at the end. Another 5-10 minutes would have been nice, but them’s the breaks. 🙂 I got a lot of questions at the end, so I took them outside so as not to annoy the following speaker too much. By the time I finished answering questions I had missed all but the last 10 minutes of the final session, so I hung around until that finished and met up with Connor and Roel to walk back to the hotel and then grab some food.

I was under the impression the final day was going to be extended, but as it turned out everything pretty much ended at 14:00. I spent the rest of the day in my hotel room in a vegetative state, as the post-conference crash began… 🙂

Tomorrow (Friday) I start the journey back to the UK, finally getting home on Saturday evening UK time. I’ll write a wrap-up post when I get back with my impressions on the conference as a whole.

Cheers

Tim…

Oracle OpenWorld and Code One 2018 : Day 3 – Wednesday

Wednesday was the start of the burnout! After two days of all-day briefings and a couple of days of OpenWorld and Code One madness, mixed with a touch of jet-lag, I felt totally beat up. I woke up early, went to the gym and caught up on some blogging. The only thing I definitely had to do was my 14:00-16:00 shift on the Blockchain Beer Demo in the Groundbreakers Hub.

Photo by Mike Donovan

When I got to the conference the first thing I did was head down to the exhibition hall and demo grounds to try to catch up with a few people. I visited the folks at the Dbvisit stand where I met all the usual suspects, plus a couple of folks I’d never met before. I spent quite a bit of time chatting with Mike and Vit about containers for the database layer.  I know I say it every year, but I really like the Dbvisit crew. They are a good crowd and it’s nice to see good people doing well!

From there I saw a bunch of shady characters hanging around the APEX stand at the demo grounds, so I went to bug them for a bit, and pretend I know what APEX is. After that I grabbed some food and then it was time for my shift at the Blockchain Beer Demo…

I was an abject failure at the Blockchain Beer Demo. I know what blockchain is, and I know what beer is, but I can’t really talk in detail about either of them. Luckily we had a blockchain and a beer expert (Neil) on hand, so I mostly handed out stickers and pointing people in the right direction, depending on what they were interested in, which was mostly beer. Despite my incompetence, I did have a good time chatting to people. We were next to the “Meet the Experts” stand, so I provided waiter service to them. 🙂

Photo by Brendan Tierney

From there I went across to tail end of the chocolate tasting, then we got kicked out of the exhibition hall as everyone was starting the breakdown. On the way out Brendan and I got quick photo with the ACE and Groundbreakers frames. I also got one with Liron, but he’s not posted his. 🙂

Most people were heading out for the CloudFest event in the evening, but I gave my ticket away, so I grabbed some food and headed back to the hotel to crash.

Tomorrow (Thursday) I have my DBA Does Docker talk at 12:00, then the event is over for me.

Cheers

Tim…

Even ACEs Make Mistakes : You never forget the first time you drop the production database!

At Oracle OpenWorld 2018 a few of the EMEA Oracle ACEs got together to do a lighthearted joint presentation, where we had four minutes each to speak about a mistake we had made and what we learned from it. Not surprisingly the session was called, “Even the ACEs make Mistakes – What did they learn?” Some of us said we would blog our stories, so here’s mine…

I had performed some maintenance on a production database, moving some objects into new tablespaces and purging a lot of old data. It meant that some tablespaces now contained a lot of free space, that would never be reused, so I decided to do a manual tablespace reorganisation to reclaim the unused disk space.

Having moved all the objects out of one tablespace I preceded to drop it, but forgot to add the “including contents and datafiles” clause. That meant I had to manually delete the datafiles from the file system. The only problem was I deleted the wrong ones! Here’s a graphical representation of what happened.

Luckily Oracle has a really good backup/recovery tool called Recovery Manager (RMAN), and I had backups of the database from before I messed up, so I restored and recovered the missing datafiles and everything was working again.

What did I learn?

  • Always have a backup! Luckily I did.
  • If you are going to drop a tablespace, make sure it is the correct one (I did) and make sure you add the “including contents and datafiles” clause (I didn’t), so Oracle will delete the datafiles for you. That way you won’t delete the wrong files by accident.
  • Don’t be me. I’m a dick! 🙂

Cheers

Tim…

Oracle OpenWorld and Code One 2018 : Day 2 – Tuesday

I got up early and finished off my blog post for the previous day. Once I had posted it I opened my phone to see I was 15 minutes late for my first appointment for the day. Panic!

I got a quick shower, got dressed and scuttled off to the Women in Technology (#WIT) breakfast. I was “fashionably late”. Sorry! I’m part of a WIT group at work, so it’s good to get to a few of these events and get some ideas and feedback. It was nice to see so many Oracle ACEs, Groundbreakers and Java Champions in the room.

From there I went on a battery hunt with Michelle Malcher, which took us to Target. That’s when I knew I was in the USA. 🙂 From there we headed off to the Groundbreakers Hub, without batteries, where I noticed I was in second place on the Charity Augmented Reality Game leader board. I spent a few minutes getting to the top of the list, so I could get a picture. I’m sure I’ll be nowhere near the top by the end of the week, but for a time I was #1! 🙂

Please go and play it, as it raises money for charity!

Pretty soon it was time for my session called Cool New Features for Developers in 18c and 12c. As the name suggests, this was a list of things I think are cool in 12c and 18c database. Some are cool for geek value. Some are cool because they do things for me, so I can do less work and get all the glory. 🙂 I normally do an article to back all my presentations, but since this was really a series of live demos, the link goes to a simple link page, pointing to all the separate articles explaining the features I’m demoing.

I sat in another talk, which I’m not going to mention. It wasn’t that the talk itself was bad. It’s just the title didn’t match the content, so it was a bit frustrating. If the name implies it’s a deep dive, it shouldn’t be a 101 talk. If the name implies it’s a beginner talk, it really should be. If a member of the audience doesn’t understand their own ability and picks the wrong talk, that’s not your fault, but pitching a talk as one thing and delivering another is annoying. Rather than walk out I just opened the laptop and played catch-up on a few things.

Next up it was “Even the ACEs make Mistakes – What did they learn?”, where I had a 4 minute slot along with a bunch of other speakers. Four minutes is a surprisingly long time. I had a story, which I will blog about separately, but because I knew I had a four minute limit I rushed, missed out half of what I was going to say, and ended up finishing 2 minutes early. Go figure! A number of others had a similar experience, but luckily some had the opposite problem. 🙂 The session was closed by Michelle Malcher, who was not originally on the list of speakers, but summed everything up really well, which I will paraphrase now. It’s really important to let your kids (and other people) know about your failures. It teaches them it’s OK to fail, so they are not scared of trying new stuff. It’s sad if people miss out because they are scared to fail. I thought this session was fun and lighthearted. I hope the audience liked it.

Pretty soon it was time for the Oracle Code keynote that was scheduled for 17:00-19:00. Instantly that sets alarm bells ringing in my head. A two hour keynote? It was a very mixed bag. There was a lot of the corporate stuff I guess you expect from a keynote, specifically what services Oracle offer. There was an interesting talk by Dee Kumar about Cloud Native Computing Foundation (CNCF), of which Oracle is a member. There was a really long talk about blockchain, that could probably have been compressed into 5 minutes.  After about 90 minutes the room was looking pretty empty and we finally got to the bit people had originally come for…

The fireside chat with Neha Narkhede, Guido van Rossum, Doug Cutting, Graeme Rocher, and Charles Nutter was really good, but way to short. You’ve got a bunch of really smart people to turn up and you give them 30 minutes of a 2 hour keynote to speak about their experiences of open source. I’m sorry, but that is such a waste of a great opportunity. What’s worse, by the time they came out on stage the room had emptied, which was not what these folks deserved.

Once the fireside chat was over, a group of people who came top of the speaker evaluations from last years Java One conference were invited on stage as “Oracle Code One Stars”. See if you can see Connor McDonald‘s leg in this photo. It’s the only bit of him showing. 🙂

From there is was back to the hotel and bed.

Cheers

Tim…

Oracle OpenWorld and Code One 2018 : Day 1 – Monday

Day 1 started stupidly early as always. I was playing around with 18c Express Edition (XE), seeing if I could run my demos on it for the talk I *thought* I had later in the day. I actually didn’t get it sorted, because I found a weird little issue that I dived into instead. 🙂

After much coffee and diet coke it was time to leave for my 09:00 Code One session. When I walked into Moscone West there were a lot of people queuing for registration, so I started to think I might be presenting to zero people.

I got there about 30 minutes early and started to set up, then noticed it wasn’t the session I thought I was doing. It was the session on Analytic Functions instead. I’m using the jet-lag excuse again… 🙂 The demos for this session were using SQLcl on my laptop, connecting to Autonomous Transaction Processing on Oracle Cloud, but of course this could have been pretty much any Oracle database from 8i onward. 🙂 I think the session went well.

From here on my timeline gets a little hazy, so some of these things may get reported out of sequence. Such is the state of my brain at the moment. 🙂 I didn’t have a plan for anything really. I just bounced between things. I apologise in advance for the terrible photography. These people are not blurred in real life. 🙂

I headed down to the “Groundbreakers Hub” in the exhibition hall of Moscone West. I did a little scan of the surrounding stands with Neil Chandler. We picked up some free t-shirts, which I’m pretty sure won’t fit me. 🙂

Next up I went to see Wim Coekaerts with “Oracle Linux : State of the Penguin”. I walked in and Wim said, “My biggest fan has arrived”. Insert manga-style schoolgirl giggle from me. 🙂 There’s a lot going on in the Oracle Linux world, specifically in the surrounding platform support. Wim covered a bunch of things, but there were some standouts for me. Oracle supporting/tracking Cloud Native Computing Foundation (CNCF), giving you the ability to deploy on cloud and on-prem. Oracle Linux preview of Kata Containers, giving you the feel of Docker, but the security of a lightweight VM. He also mentioned the Oracle Instant Client is now in the  repositories, so just a regular “yum install” to get the client with no additional click-through license. That seemed like a pretty popular move judging by the retweets I got. 🙂

From there I went across to see some of the lightning talks at OakTable World. I’m here on Oracle’s dollar, so I feel kind-of weird going to a non-Oracle event, which is why I tend not to spend too much time at OakTable World. When I got there I was pleasantly surprised to see Dad! I thought he would be off somewhere on a yacht, or riding through the desert on a Harley.

First up was the mighty John Beresniewicz with “NoSQL is anti-relational”, then “Average Active Sessions – Deeper Meaning”, where he gave a history of how the Average Active Sessions concept came into being. JB presents with such clarity. He’s awesome. That was followed by “Think Data, Not Database” by Arup Nanda, where he encouraged us to think of data in a wider sense (databases, spreadsheets and data in queues like Kafka) when we are considering performance and processing of data. The last lightening session I saw was Tim Gorman (the real Tim) speaking about “Data Masking”. He made some great points about the attack surface of non-prod (dev, test, UAT etc.) systems being much bigger than production, which of course leads to the question of how do you protect the data in them if you are using production data. Anonymisation and masking being part of the process.

I really like the lightning talk approach. I wish Oracle would bring this kind of thing into OOW. It would give more people a chance to speak, and you get a lot of good content in a short time. I’m mostly after pointers, not essays. 🙂

I tried to get food, but they were out of everything except chicken, so I headed back to the hub and got an icecream. 🙂 While I was there I bumped into Chris Thalinger and went along to see his session on “Performance tuning Twitter services with Graal and machine learning”. If you’ve not heard of it, Twitter is like Facebook, but with less functionality… 🙂 A lot of the technical stuff went over my head, but the message was pretty clear. Graal is cool and will improve the performance many applications running on the JVM, just by enabling it. In addition, Twitter were able to improve the performance of their workloads by using “autotune” to automate the tuning of the JVM parameters. Autonomous JVM anyone?

From there I went back down to the Groundbreakers Hub and basically hung around for the rest of the day, chatted to folks and did some of the demos. I disturbed my Son, who was trying to finish his slides. He gave me a quick run through, so I didn’t have to go to his session. I’m a terrible Dad!

From there is was back to the hotel to crash. I’m sure there were other things I could do, but it’s Monday and I already feel beat up.

Tomorrow (today by the time I post this) I have a session on “Cool New Features for Developers in Oracle Database 18c and Oracle Database 12c”, a five minute slot in “Even the ACEs make Mistakes – What did they learn?” and I think I have to go on stage at the Code One keynote between 17:00-19:00 to get a prize? Not sure I’ll still be awake by then. 🙂

Cheers

Tim…

It’s not all about you!

I want to start with a couple of examples.

A few years ago Oracle released the Exadata Express Cloud Service and I really didn’t get it. I mean I knew what it was, a managed cloud service based on a PDB on an Exadata, but I just couldn’t see it being of any use to me. Of course, big mouth that I am, I said as much. Then other people in the room that weren’t DBAs started showing some interest in the service and I thought to myself, “Oh, it’s not all about you!” 🙂

I had a similar experience about something mentioned during the ACE briefing yesterday. A specific feature that may or may not be discussed at OOW18 was presented to us. Many of the DBAs in the room got super intense about it, and I could feel myself making lists of possible problems and questions I needed to ask, then Simon sitting next to me said something like, that sounds really powerful. Once again I had brought all my baggage with me and couldn’t give things a fair hearing. I was already making judgements before I had even heard all the facts and seen it in action. What’s more, even if I decided it wasn’t for me, that doesn’t mean it’s true for everyone else.

As you may know, Oracle Database 18c XE has been released. The news was greeted by a number of different reactions. Some were excited about the release, while some were concerned about some of the “missing bits”. I understand where they are coming from, because I am often in this position also, but I wrote a tweet that said,

“People who use XE:

– People who want a smallish and functional Oracle DB for free in production.

– People learning, teaching, training. Options would be nice, but not essential.

– Professionals who have access to EE+Options, and want XE to have everything. :)”

I hope people didn’t take offence to that, because as I’ve explained before I also fall into this trap too. I agree there will be use cases that are affected by what is, and is not in this edition, but maybe those are not applicable to everyone?

You will see a bunch of stuff announced at OOW18 and Code One this week. Before you go off the deep end, ask yourself if it is actually intended for a user like you, and if you think there is a section of the market that will welcome it, even if you don’t?

Having said all that, I reserve the right to fly off the handle at stuff and completely ignore my own advice… Do as I say, not as I do. 🙂

Cheers

Tim…

PS. It is all about me really! 🙂