Month: March 2022

Why Automation Matters : Taking a step backward to move forward!

There was a small thread on Twitter today about unit testing, which I’m going to extrapolate to automation generally.

There can be a certain reluctance towards writing unit tests. I guess the thinking goes, I’m here to write code and solve problems, not waste time writing test code. Unit test quality and code coverage varies, but it’s not unusual to hear people say their unit tests have more lines of code than the code they are testing. I guess that adds to the reluctance. What some people fail to see is once the unit tests are written, they may never have to manually test that code again. If the code changes over time, the unit test may only need a few small tweaks to bring them up to date. Over the lifetime of the project, that initial investment can represent a massive saving…

This is true of many aspects of automation. Yes, you can create a new database in a few minutes by clicking some buttons on a GUI, and that’s fine when you get one request a week. What happens when developers want a new database for every test they run? Your button presses don’t scale. If instead you automated the process, you would never have to manually create a database again, and developers could build and burn databases to their heart’s content.

You could be the gatekeeper who runs scripts in the production environment, but what happens when you’re on holiday? What happens when the rate of production deployments increase? You become the bottleneck doing meaningless work. If you had helped build a deployment pipeline, those production deployments could happen automatically, with the correct governance of course.

I’ve said it before in this series, but I’ll say it again. Working in the tech industry is like swimming upriver. You can’t just stop swimming, because that means you are moving backwards. If more water is added to the river, the flow rate increases and you are overwhelmed. You have to keep trying to improve your efficiency to protect yourself against what is coming round the next bend…

I know it can be hard when you have a pressing deadline, but you really are taking a step backward to move many steps forward!

Check out the rest of the series here.

Cheers

Tim…

Video : Ansible : Installation and Configuration

In today’s video we’ll cover the installation and configuration of Ansible.

The video is based on the following article, and will be the first in a series of videos.

You might find some useful stuff here.

The star of today’s video is Neil Chandler, who took time out of his busy bathroom schedule to record this clip.

Cheers

Tim…

Death of the DBA… Again…

If you’ve followed the blog for a while you will know I’ve revisited this topic several times since I first wrote about it in 2007. Here are some links if you want to go down memory lane.

I think I’ve been pretty consistent with my opinion on this subject, but I still get people misunderstanding the argument, so I’m going to try again, without revisiting the contents of all those previous articles.

What is a DBA?

This means something different in each company and each person I speak to.

In some companies it can mean a basic operations job. Install, patch, check backups and run scripts that other people send to you. All of these tasks are easily automated on-prem, and essentially don’t exist on the cloud. If this is the role you do, and this is all you do, you are going to have a bad time unless you gain some new skills. In other companies it can mean something completely different. My official title is “Lead DBA”. What do I do? Just off the top of my head:

  • DBA work with Oracle, SQL Server, MySQL.
  • Administration of middle tier stuff like WebLogic, Tomcat, NGINX etc.
  • I look after the load balancer configuration for a big chunk of the back-end business systems, including writing iRules in TCL.
  • Support for a few proprietary 3rd party apps.
  • Docker/containers.
  • APEX and ORDS. I’m the worlds worst APEX developer (see here), but I have to look after it, and support the APEX developers.
  • I don’t do much traditional development in this company, but I provide support when people have SQL and PL/SQL questions, because I’ve done that for a minute. 🙂
  • I’m increasingly being drawn into automation using shell scripts, Ansible, Terraform, Liquibase etc.

I’ve already got rid of some of the operational aspects of my job. Over time I’m hoping more will go, but that mostly depends on external constraints holding me back. Even if my involvement with databases stopped completely, I can still remain busy. Am I saying my role is what a DBA should be? No. I think my position is a little more extreme than some. I’m saying DBA is a title that various people and companies use, but it doesn’t really mean anything now. It can be anything from basic operations to “Do Bloody Anything”.

When is it going to happen?

For some companies it already has. Some companies will hang on to the old ways until the bitter end. It won’t happen overnight, but if it is not happening already in your company, what you are likely to see is a gradual drop in operational tasks as they get automated. This allows either the same number of people to cover more systems, or less people to do all the current work.

If you are seeing pieces of your role disappearing, you have to do something else to add value!

But person X disagrees with you!

That’s fine. This is only my opinion and you don’t have to agree, but please check the context of what people are saying. Often the responses to my comments include mentions of performance tuning and diagnosing architectural issues. I have consistently said the “operations DBA” role would go. The job that focuses on the basic grunt work. There will be a continued need for people who can diagnose and solve performance problems, both in the databases and in the application architecture. Moving to the cloud won’t magically cure all performance issues, and some would say they will increase the number of issues. You can still deliver architecturally crap solutions on the cloud. You can still do bad database design and write bad applications. Good people will always find a home, provided they don’t stick rigidly to the past.

You also have to look at the person making the comments. If someone is a performance consultant to the stars, but identifies as a DBA, they are probably going to take these comments personally and hear me saying they are redundant, which I am not. If someone runs a DBA as a service company, they won’t like the sound of this and will go into defensive mode.

I’ve been a regular DBA for most of my working life, and I’ve watched the job evolve. You may have a different experience, and that is fine, but I speak to a lot of people, and I think my opinion on this subject tracks pretty well with what is happening out there.

You are really talking about the cloud aren’t you?

Not. Automation is the thing that is removing the need for the operational DBA and basic system admin roles. Even if your company is in cloud denial, it doesn’t mean they won’t want to take advantage of automation. The cloud makes some things easier from an automation perspective, because the cloud providers have done some of the leg-work for you, but automation existed long before the cloud…

What should I do?

When you know, please let me know so I can do it too. 🙂 Seriously though, keep your mind open to new opportunities, and if you get a chance to try something new, give it a go. Nothing is ever wasted. Some people will gravitate to the data and analytics side of things. Some to development. Some to the architectural end of things. In all cases, there is a lot to learn and the less you know when you start, the harder the journey will be, so get learning whatever you like the look of…

Cheers

Tim…

Update 1: In social media comments, some people have mentioned the term “Data Engineer”. To me a data engineer needs to understand data, and requires and understanding of the development process. I’ve met some operational DBAs that can barely string together a SQL statement. These types of operational DBAs have a lot to learn before they can consider themselves a data engineer. A DBA can become a data engineer, but being a DBA does not mean you are a data engineer.

Update 2: Don’t get distracted by the name DBA. I don’t care about the name of the role. I’m talking about the function the person is performing. If they are performing basic operational tasks, whatever they are called, they need to start getting some new skills.

AZ-900 Microsoft Azure Fundamentals (my thoughts)

I recently had to sit the AZ-900 Microsoft Azure Fundamentals exam for work. Here are some thoughts about it.

Preparation

There are lots of ways to prepare for this exam. Here are the things I used…

  • Microsoft Certified: Azure Fundamentals – Learn : My initial plan was to just work through these notes and sit the exam. After I had completed two of the six modules I was going crazy. I found the material really dull. Even though I switched to using video tutorials, I still came back to this material, and used the knowledge check questions. There are also some things in this material that was not covered elsewhere.
  • AZ-900 | Microsoft Azure Fundamentals Full Course, Free Practice Tests, Website and Study Guides : One of my colleagues suggested this YouTube playlist. It made the process of covering the material a lot easier. The guy who put this together included linked materials, which was handy. I used the free practice questions he provided to check my knowledge. This playlist was my main preparation.
  • Microsoft Azure Fundamentals Certification Course (AZ-900) – Pass the exam in 3 hours! : Someone on Twitter recommended this video. I only watched about 20 minutes of it, but it seemed OK. A number of people said it was good. You might want to give it a go.
  • Microsoft Virtual Training Days : This is a Microsoft provided 4 hour training session, spread over two days. It’s free, and if you attend it you get an exam voucher for free. They claim it is all you need to pass the exam, but I don’t think that is true. I did all my prep using the YouTube playlist mentioned previously, and attended this course as a revision session before sitting the exam. It is pre-recorded material, but there are people available to answer questions during the course. There are 2-3 sessions per month, so it’s easy to register for.

Remember, it’s a fundamentals exam, so it doesn’t go into any great depth, and it doesn’t qualify you for anything, but it visits a wide variety of features available in Azure. Despite the lack of technical content, it takes quite a lot of time to prepare to pass the exam.

The Exam

My company uses Azure, and we are encouraged to take this cert. As mentioned above, if you attend the free Microsoft Virtual Training Days for the “Fundamentals” training, you get an exam voucher that will allow you to sit the exam for free.

The exam is multiple choice. There are traditional multiple choice questions, drag and drop questions, and sentence completion questions. There are between 40 and 60 questions and you need to score at least 700 out of 1000 points to pass. Some questions are simple True/False answers. Some are more complex, with multiple sub-questions, and are worth more points. The number of questions you get asked varies depending on how many of the complex multi-point questions you get asked. I read someone saying they had 30 questions, but the official site says 40-60. I had 37 questions in 45 minutes. 🙂

There are questions that are not covered in any of the training materials I used, but you can hopefully guess at them, knowing what you do know. I believe the Microsoft Learn, Virtual Training Days and exam are due to be updated in April 2022. We were told the exam would track what is in the current training materials. That was not true. There were questions on subjects not in the current Microsoft Learn training materials, or the Virtual Training Days. Sneaky.

I’m sure some people will attend the Virtual Training Days course and pass the exam if they have great memory retention and get lucky with the questions that come up. For most people, I think just attending the training days course would result in a failure in the exam. I would not have passed with just the Virtual Training Days course.

BTW: During the Virtual Training Days course someone asked what happens if you fail the exam. The tutor said you can get another free exam voucher if you attend the training course again. Free is a good deal. 🙂

Thoughts

I had a bit of a rant on Twitter while I was preparing for the exam, because I found the content really dull. I had a look through the equivalent exams for AWS and Oracle Cloud and I think I would have a similar experience with them also. The problem as I see it is cloud providers support a huge number of services, and most of them are of little interest to me. Despite that you have to visit loads of them for the fundamentals/foundations courses. The result is:

  • For things you are interested in, you don’t get any useful information.
  • You have to waste time learning stuff you are not interested in.

But it’s a fundamentals exam stupid! Yeah I know, but it doesn’t stop it being annoying.

In my company I will never get to define identity, governance, networking, pricing etc. At best I will be given a resource group and be told to work in there. As a result I couldn’t help but feel most of this was pointless.

As I said on Twitter,

“The vast majority of the cloud is amazingly boring, drab and shite! The fun thing is building solutions with it, not the boring fuckin’ paper work…”

Me, 2022

So I’m not a fan of these types of cloud certifications. I don’t want to do any more cloud certifications, but that’s just my opinion. I guess as long as they are free, you can decide for yourself. There is no way I would pay for this…

Cheers

Tim…

PS. I passed it.

PPS. I know I’m a grumpy old git, but it wasn’t my choice to do this certification, so that kind-of coloured my whole view of it.

PPPS. I’ve suggested to other people they should attend the Virtual Training Days course, and then decide if they want to revise for the exam. I think the course gives you a good idea of what you can do on Azure. Whether you want to spend the time to prepare for the exam is another matter.

How are you provisioning your databases on-prem and in the cloud? Poll results discussed.

Following on from my previous post, I wanted to discuss the results of the polls regarding database provisioning.

This was the first question I asked.

How are you provisioning your databases on-prem and in the cloud?

A couple of years ago I stopped putting GUI installation articles on my website. They look pretty and seem to get a lot of views, but I thought posting them was wrong because I never use GUI installations. Posting them felt like I was sending the wrong message. I wrote about that here. This was one of the reasons I lead with this question. I was pleased with the results of the poll.

  • Using GUI: I understand some people don’t want to take a step backwards to move forwards, but at nearly 20%, this number is still too high IMHO. You can’t check button clicks into version control. I’m sure some smart arse will tell me you can if you use Robotic Process Automation (RPA) to click them. 🙂
  • Shell scripts: If I’m honest I thought 34% was on the low side. I was expecting more people to be using silent installations using shell scripts, but the number is lower because of the next option. If people have made a big investment into writing robust shell scripts with good error handling, I can understand the reluctance to move away from them. Ansible and Terraform are nice, but they are not magic. 🙂
  • Ansible/Terraform/Other: This was actually the surprise of bunch. I wasn’t expecting this number to be so high, but I was pleasantly surprised. The previous post showed lots of people running their databases in the cloud, which has no doubt helped to drive the uptake of automation tools like Ansible and Terraform. Happy days!

Spurred on by a question from Jasmin Fluri, I asked the following question to drill down a little more.

For people using Ansible and/or Terraform, how automated is your process?

This was also a pleasant surprise.

  • We run it manually: I was expecting this to be way ahead of the pack, but at nearly 38% I was wrong, which made me happy. I have no problem if people are running Ansible or Terraform manually. A pipeline is just a bunch of building blocks threaded together. The fact people have taken the time to build these blocks is great. Threading them together is nice, but it’s not the “be all and end all”. The important bit is the definitions of the systems are in code!
  • Automated pipeline: Over 33% made me happy. My assumption was this would be lower, and I wrong and I’m glad.
  • Terrahawks was a cartoon: The people who picked this were wrong! Terrahawks was a kids TV show using puppets, not animation. I’m really surprised nobody noticed this. The community let me down! 😉 If we discount this response from the mix, it makes the other two responses close to 50:50, which is cool.

On a bit of a tangent, I wanted to know how dominant Git was these days.

What Version Control System (VCS) are you suing for your database scripts and code?
  • Git – On Prem: I knew Git would dominate, but I wasn’t sure if people would be hosting their repositories on-prem or in the cloud. With a response of over 30%, that means nearly half of the Git users were hosting their repositories on-prem, which was higher than I expected.
  • Git – Cloud-based: I expected this to dominate, so 37% was a little lower than I expected. Only a little over half of the Git users were using cloud-based repositories. We use cloud-based Git repositories, but we always keep a backup on-prem. Just in case.
  • Other VCS – Not Git: I expected this to have a reasonable showing as VCS software like Subversion used to be really popular, so I knew things would linger. Nearly 19% isn’t bad. I don’t think there is anything wrong with using something other than Git, but Git has become so pervasive in tooling it probably makes sense to take the plunge.
  • VCS is for wimps: I’m hoping nearly 13% of the respondents were picking this to wind me up, but I suspect they weren’t. If you are not currently using version control, please start!

Version control is at the heart of automation, DevOps, Infrastructure as Code (IaC) and all that funky business, so if people can just get that right they have taken the first step on the journey.

So overall this makes very pleasant reading. Lots of people are provisioning databases using some form of scripting, rather than GUIs, and a bunch of people are automating that provisioning. This is what I wanted to hear.

Cheers

Tim…

PS. You know the caveats. These are small sample sizes. My audience has an Oracle bias. I’m no expert at automation, DevOps and the cloud. Just a keen dabbler.

Are you running production databases on the cloud? Poll results discussed.

It can be quite difficult to know if your impression of technology usage is skewed. Your opinion is probably going to depend on a number of factors including what you read, who you follow, and the type of company you work for. For this reason I asked some questions on Twitter the other day, just to gauge the response.

Let me start by saying, this is a small sample size, and most of my followers come from the Oracle community, including a number of Oracle staff. This may skew the results compared to other database engines, and technology stacks. I’m commenting on the results as if this were a representative sample, but you can decide if think it is…

So this was the first question I asked.

Is your company running production relational databases in the cloud?

We can see there was a fairly even spread of answers.

  • All prod DBs in cloud: A response of nearly 19% picking this option kind-of surprised me. I speak to a lot of people, and there always seems to be something they’ve got that doesn’t fit well in the cloud for them. Having this many people saying they’ve managed to make everything fit is interesting.
  • Some prod DBs in cloud: I expected this response to be high and with over 27% it was. When we add this to the previous category, we can see that over 46% of companies have got some or all of their production relational databases in the cloud. That’s a lot.
  • Not yet, but planned: At over 24%, when added to the previous categories, it would seem that over 70% of companies see some perceived value in running their databases in the cloud. Making that initial step can be difficult. I would suggest people try with a greenfield project, so they can test the water.
  • Over my dead body: At 29%, this is a lot of people that have no intention of moving their databases to the cloud at this moment in time. We might get some answers about why from the next question.

This was my second question.

What’s stopping you from moving your databases to the cloud?

Once again, we get a fairly even spread of responses.

  • Legal/Compliance: Over 17% of respondents have hit this brick wall. Depending on your industry and your country, cloud may not be an option for you yet. Cloud providers are constantly opening up data centres around the world, but there are still countries and regions that are not well represented. Added to that, some organisations can’t use public cloud. Most cloud providers have special regions for government or defence systems, but they tend to be focused in certain geographical regions. This is a show stopper, until the appropriate services become available, or some hybrid solution becomes acceptable.
  • Company Culture: At over 30%, this is a road block to lots of things. Any sort of technology disruption involves a change in company culture, and that’s one of the hardest things to achieve. It’s very hard to push this message from the bottom up. Ultimately it needs senior management who understand the need for change and *really* want to make that change. I say *really* because I get the feeling most management like to talk the talk, but very few can walk the walk.
  • Cloud is Expensive: At nearly 29%, this is an interesting one. The answer to the question, “is cloud more expensive?”, is yes and no. 🙂 If you are only looking at headline figures for services, then it can seem quite expensive, but the cloud gives us a number of ways to reduce costs. Reserved instances reduce the cost of compute power. Selecting the correct shape and tier of the service can change costs a lot. Spinning down non-production services when they are not used, and down-scaling production services during off-peak hours can save a lot of money, and these are not things that necessarily result in a saving on-prem. I also get the impression many companies don’t work out their total cost of ownership (TCO) properly. They forget that their on-prem kit requires space, power, lighting, cooling, networking, staffing etc. When they check the price of a service on the cloud, it includes all that, but if you don’t take that into consideration, you are not making a fair comparison. Some things will definitely be cheaper on the cloud. Some things, not so much. 🙂
  • Cloud Sucks: At nearly 23%, this is a big chunk of people. It’s hard to know if they have valid reasons for this sentiment or not. Let’s take it on face value and assume they do. If this were a reflection of the whole industry, it’s going to be interesting to see how these people will be won over by the cloud providers.

The comments resulted in a few interesting things. I’ve responded to some of them here.

  • “Lack of cloud skills.” We all have to start somewhere. I would suggest starting with small proof of concept (POC) projects to test the water.
  • “Unreasonable Oracle licencing restrictions.” In case you don’t know, the core factor doesn’t apply to clouds other than Oracle Cloud, which makes Oracle licensing more expensive on non-Oracle clouds. Of course, everything can be negotiated.
  • “Lack of availability of Cloud experts to assist/advise.” I’m sure there are lots of people that claim they would be able to help, but how many with a proven track record is questionable. 🙂
  • “We have a massive legacy estate to consider.” Certainly, not everything is easy to move the the cloud, and the bigger your estate, the more daunting it is. I’m sure most cloud providers would love to help. 🙂
  • “Latency with fat client applications.” I had this conversation myself when discussing moving some of our SQL Server databases to Azure. It can be a problem!
  • “Seasonal businesses with uncertain money flow may not able to meet the deadlines for subscription payments.” Scaling services correctly could help with this. Scale down services during low periods, and scale up during high periods.
  • “The prime fear is being pulled off from the grid. Undependable internet connections.” Sure. Not every place has dependable networking.
  • “Bandwidth requirements & limited customization possibilities.” Ingress and egress costs vary with cloud providers. It may be intelligent design of your processes can reduce the amount of data being pushed outside the cloud provider. The cloud is very customisable, so I’m not sure what the issue is here, but I’m sure there are some things that will be problematic to some people.

Overall I think this was an interesting exercise. Even five years ago I would have expected the responses to skew more in favour of on-prem. Barring some huge change in mindset, I would expect the answers to be even more in favour of cloud in another 5 years.

Regardless of your stance, it seems clear that familiarity with cloud services should be on your radar, if it’s not already. Your current company may not be fans of the cloud, but if you change jobs the cloud may be a high priority for your new company.

Cheers

Tim…

PS. I’ve been running my website on AWS since 2016 . I started to write about some services on AWS and Azure in 2015. I’ve been playing with Oracle Cloud since its inception in 2016 (I think). Despite all this, I consider myself a dabbler, rather than an expert.

Video : SecureFiles Shrink in Oracle Database 21c

In today’s video we’ll demonstrate the SecureFiles Shrink feature, introduced in Oracle database 21c.

The video is based on this article.

You might also find these useful.

The star of today’s video is Arman Sharma. This clip was from Sangam15. To think that was seven years ago is mind-blowing!

Cheers

Tim…