Tag: vaulted

Video : Ansible Playbooks : Introduction

In today’s video we’ll introduce Ansible Playbooks.

There’s quite a lot to cover, so this is only one of several videos covering playbooks. The video is based on the following article.

You might find some useful stuff here.

The star of today’s video is Bailey, who has been known to associate with Connor McDonald at times…

Cheers

Tim…

Why Automation Matters : Taking a step backward to move forward!

There was a small thread on Twitter today about unit testing, which I’m going to extrapolate to automation generally.

There can be a certain reluctance towards writing unit tests. I guess the thinking goes, I’m here to write code and solve problems, not waste time writing test code. Unit test quality and code coverage varies, but it’s not unusual to hear people say their unit tests have more lines of code than the code they are testing. I guess that adds to the reluctance. What some people fail to see is once the unit tests are written, they may never have to manually test that code again. If the code changes over time, the unit test may only need a few small tweaks to bring them up to date. Over the lifetime of the project, that initial investment can represent a massive saving…

This is true of many aspects of automation. Yes, you can create a new database in a few minutes by clicking some buttons on a GUI, and that’s fine when you get one request a week. What happens when developers want a new database for every test they run? Your button presses don’t scale. If instead you automated the process, you would never have to manually create a database again, and developers could build and burn databases to their heart’s content.

You could be the gatekeeper who runs scripts in the production environment, but what happens when you’re on holiday? What happens when the rate of production deployments increase? You become the bottleneck doing meaningless work. If you had helped build a deployment pipeline, those production deployments could happen automatically, with the correct governance of course.

I’ve said it before in this series, but I’ll say it again. Working in the tech industry is like swimming upriver. You can’t just stop swimming, because that means you are moving backwards. If more water is added to the river, the flow rate increases and you are overwhelmed. You have to keep trying to improve your efficiency to protect yourself against what is coming round the next bend…

I know it can be hard when you have a pressing deadline, but you really are taking a step backward to move many steps forward!

Check out the rest of the series here.

Cheers

Tim…

Video : Ansible : Installation and Configuration

In today’s video we’ll cover the installation and configuration of Ansible.

The video is based on the following article, and will be the first in a series of videos.

You might find some useful stuff here.

The star of today’s video is Neil Chandler, who took time out of his busy bathroom schedule to record this clip.

Cheers

Tim…

Death of the DBA… Again…

If you’ve followed the blog for a while you will know I’ve revisited this topic several times since I first wrote about it in 2007. Here are some links if you want to go down memory lane.

I think I’ve been pretty consistent with my opinion on this subject, but I still get people misunderstanding the argument, so I’m going to try again, without revisiting the contents of all those previous articles.

What is a DBA?

This means something different in each company and each person I speak to.

In some companies it can mean a basic operations job. Install, patch, check backups and run scripts that other people send to you. All of these tasks are easily automated on-prem, and essentially don’t exist on the cloud. If this is the role you do, and this is all you do, you are going to have a bad time unless you gain some new skills. In other companies it can mean something completely different. My official title is “Lead DBA”. What do I do? Just off the top of my head:

  • DBA work with Oracle, SQL Server, MySQL.
  • Administration of middle tier stuff like WebLogic, Tomcat, NGINX etc.
  • I look after the load balancer configuration for a big chunk of the back-end business systems, including writing iRules in TCL.
  • Support for a few proprietary 3rd party apps.
  • Docker/containers.
  • APEX and ORDS. I’m the worlds worst APEX developer (see here), but I have to look after it, and support the APEX developers.
  • I don’t do much traditional development in this company, but I provide support when people have SQL and PL/SQL questions, because I’ve done that for a minute. 🙂
  • I’m increasingly being drawn into automation using shell scripts, Ansible, Terraform, Liquibase etc.

I’ve already got rid of some of the operational aspects of my job. Over time I’m hoping more will go, but that mostly depends on external constraints holding me back. Even if my involvement with databases stopped completely, I can still remain busy. Am I saying my role is what a DBA should be? No. I think my position is a little more extreme than some. I’m saying DBA is a title that various people and companies use, but it doesn’t really mean anything now. It can be anything from basic operations to “Do Bloody Anything”.

When is it going to happen?

For some companies it already has. Some companies will hang on to the old ways until the bitter end. It won’t happen overnight, but if it is not happening already in your company, what you are likely to see is a gradual drop in operational tasks as they get automated. This allows either the same number of people to cover more systems, or less people to do all the current work.

If you are seeing pieces of your role disappearing, you have to do something else to add value!

But person X disagrees with you!

That’s fine. This is only my opinion and you don’t have to agree, but please check the context of what people are saying. Often the responses to my comments include mentions of performance tuning and diagnosing architectural issues. I have consistently said the “operations DBA” role would go. The job that focuses on the basic grunt work. There will be a continued need for people who can diagnose and solve performance problems, both in the databases and in the application architecture. Moving to the cloud won’t magically cure all performance issues, and some would say they will increase the number of issues. You can still deliver architecturally crap solutions on the cloud. You can still do bad database design and write bad applications. Good people will always find a home, provided they don’t stick rigidly to the past.

You also have to look at the person making the comments. If someone is a performance consultant to the stars, but identifies as a DBA, they are probably going to take these comments personally and hear me saying they are redundant, which I am not. If someone runs a DBA as a service company, they won’t like the sound of this and will go into defensive mode.

I’ve been a regular DBA for most of my working life, and I’ve watched the job evolve. You may have a different experience, and that is fine, but I speak to a lot of people, and I think my opinion on this subject tracks pretty well with what is happening out there.

You are really talking about the cloud aren’t you?

Not. Automation is the thing that is removing the need for the operational DBA and basic system admin roles. Even if your company is in cloud denial, it doesn’t mean they won’t want to take advantage of automation. The cloud makes some things easier from an automation perspective, because the cloud providers have done some of the leg-work for you, but automation existed long before the cloud…

What should I do?

When you know, please let me know so I can do it too. 🙂 Seriously though, keep your mind open to new opportunities, and if you get a chance to try something new, give it a go. Nothing is ever wasted. Some people will gravitate to the data and analytics side of things. Some to development. Some to the architectural end of things. In all cases, there is a lot to learn and the less you know when you start, the harder the journey will be, so get learning whatever you like the look of…

Cheers

Tim…

Update 1: In social media comments, some people have mentioned the term “Data Engineer”. To me a data engineer needs to understand data, and requires and understanding of the development process. I’ve met some operational DBAs that can barely string together a SQL statement. These types of operational DBAs have a lot to learn before they can consider themselves a data engineer. A DBA can become a data engineer, but being a DBA does not mean you are a data engineer.

Update 2: Don’t get distracted by the name DBA. I don’t care about the name of the role. I’m talking about the function the person is performing. If they are performing basic operational tasks, whatever they are called, they need to start getting some new skills.

How are you provisioning your databases on-prem and in the cloud? Poll results discussed.

Following on from my previous post, I wanted to discuss the results of the polls regarding database provisioning.

This was the first question I asked.

How are you provisioning your databases on-prem and in the cloud?

A couple of years ago I stopped putting GUI installation articles on my website. They look pretty and seem to get a lot of views, but I thought posting them was wrong because I never use GUI installations. Posting them felt like I was sending the wrong message. I wrote about that here. This was one of the reasons I lead with this question. I was pleased with the results of the poll.

  • Using GUI: I understand some people don’t want to take a step backwards to move forwards, but at nearly 20%, this number is still too high IMHO. You can’t check button clicks into version control. I’m sure some smart arse will tell me you can if you use Robotic Process Automation (RPA) to click them. 🙂
  • Shell scripts: If I’m honest I thought 34% was on the low side. I was expecting more people to be using silent installations using shell scripts, but the number is lower because of the next option. If people have made a big investment into writing robust shell scripts with good error handling, I can understand the reluctance to move away from them. Ansible and Terraform are nice, but they are not magic. 🙂
  • Ansible/Terraform/Other: This was actually the surprise of bunch. I wasn’t expecting this number to be so high, but I was pleasantly surprised. The previous post showed lots of people running their databases in the cloud, which has no doubt helped to drive the uptake of automation tools like Ansible and Terraform. Happy days!

Spurred on by a question from Jasmin Fluri, I asked the following question to drill down a little more.

For people using Ansible and/or Terraform, how automated is your process?

This was also a pleasant surprise.

  • We run it manually: I was expecting this to be way ahead of the pack, but at nearly 38% I was wrong, which made me happy. I have no problem if people are running Ansible or Terraform manually. A pipeline is just a bunch of building blocks threaded together. The fact people have taken the time to build these blocks is great. Threading them together is nice, but it’s not the “be all and end all”. The important bit is the definitions of the systems are in code!
  • Automated pipeline: Over 33% made me happy. My assumption was this would be lower, and I wrong and I’m glad.
  • Terrahawks was a cartoon: The people who picked this were wrong! Terrahawks was a kids TV show using puppets, not animation. I’m really surprised nobody noticed this. The community let me down! 😉 If we discount this response from the mix, it makes the other two responses close to 50:50, which is cool.

On a bit of a tangent, I wanted to know how dominant Git was these days.

What Version Control System (VCS) are you suing for your database scripts and code?
  • Git – On Prem: I knew Git would dominate, but I wasn’t sure if people would be hosting their repositories on-prem or in the cloud. With a response of over 30%, that means nearly half of the Git users were hosting their repositories on-prem, which was higher than I expected.
  • Git – Cloud-based: I expected this to dominate, so 37% was a little lower than I expected. Only a little over half of the Git users were using cloud-based repositories. We use cloud-based Git repositories, but we always keep a backup on-prem. Just in case.
  • Other VCS – Not Git: I expected this to have a reasonable showing as VCS software like Subversion used to be really popular, so I knew things would linger. Nearly 19% isn’t bad. I don’t think there is anything wrong with using something other than Git, but Git has become so pervasive in tooling it probably makes sense to take the plunge.
  • VCS is for wimps: I’m hoping nearly 13% of the respondents were picking this to wind me up, but I suspect they weren’t. If you are not currently using version control, please start!

Version control is at the heart of automation, DevOps, Infrastructure as Code (IaC) and all that funky business, so if people can just get that right they have taken the first step on the journey.

So overall this makes very pleasant reading. Lots of people are provisioning databases using some form of scripting, rather than GUIs, and a bunch of people are automating that provisioning. This is what I wanted to hear.

Cheers

Tim…

PS. You know the caveats. These are small sample sizes. My audience has an Oracle bias. I’m no expert at automation, DevOps and the cloud. Just a keen dabbler.

Are you running production databases on the cloud? Poll results discussed.

It can be quite difficult to know if your impression of technology usage is skewed. Your opinion is probably going to depend on a number of factors including what you read, who you follow, and the type of company you work for. For this reason I asked some questions on Twitter the other day, just to gauge the response.

Let me start by saying, this is a small sample size, and most of my followers come from the Oracle community, including a number of Oracle staff. This may skew the results compared to other database engines, and technology stacks. I’m commenting on the results as if this were a representative sample, but you can decide if think it is…

So this was the first question I asked.

Is your company running production relational databases in the cloud?

We can see there was a fairly even spread of answers.

  • All prod DBs in cloud: A response of nearly 19% picking this option kind-of surprised me. I speak to a lot of people, and there always seems to be something they’ve got that doesn’t fit well in the cloud for them. Having this many people saying they’ve managed to make everything fit is interesting.
  • Some prod DBs in cloud: I expected this response to be high and with over 27% it was. When we add this to the previous category, we can see that over 46% of companies have got some or all of their production relational databases in the cloud. That’s a lot.
  • Not yet, but planned: At over 24%, when added to the previous categories, it would seem that over 70% of companies see some perceived value in running their databases in the cloud. Making that initial step can be difficult. I would suggest people try with a greenfield project, so they can test the water.
  • Over my dead body: At 29%, this is a lot of people that have no intention of moving their databases to the cloud at this moment in time. We might get some answers about why from the next question.

This was my second question.

What’s stopping you from moving your databases to the cloud?

Once again, we get a fairly even spread of responses.

  • Legal/Compliance: Over 17% of respondents have hit this brick wall. Depending on your industry and your country, cloud may not be an option for you yet. Cloud providers are constantly opening up data centres around the world, but there are still countries and regions that are not well represented. Added to that, some organisations can’t use public cloud. Most cloud providers have special regions for government or defence systems, but they tend to be focused in certain geographical regions. This is a show stopper, until the appropriate services become available, or some hybrid solution becomes acceptable.
  • Company Culture: At over 30%, this is a road block to lots of things. Any sort of technology disruption involves a change in company culture, and that’s one of the hardest things to achieve. It’s very hard to push this message from the bottom up. Ultimately it needs senior management who understand the need for change and *really* want to make that change. I say *really* because I get the feeling most management like to talk the talk, but very few can walk the walk.
  • Cloud is Expensive: At nearly 29%, this is an interesting one. The answer to the question, “is cloud more expensive?”, is yes and no. 🙂 If you are only looking at headline figures for services, then it can seem quite expensive, but the cloud gives us a number of ways to reduce costs. Reserved instances reduce the cost of compute power. Selecting the correct shape and tier of the service can change costs a lot. Spinning down non-production services when they are not used, and down-scaling production services during off-peak hours can save a lot of money, and these are not things that necessarily result in a saving on-prem. I also get the impression many companies don’t work out their total cost of ownership (TCO) properly. They forget that their on-prem kit requires space, power, lighting, cooling, networking, staffing etc. When they check the price of a service on the cloud, it includes all that, but if you don’t take that into consideration, you are not making a fair comparison. Some things will definitely be cheaper on the cloud. Some things, not so much. 🙂
  • Cloud Sucks: At nearly 23%, this is a big chunk of people. It’s hard to know if they have valid reasons for this sentiment or not. Let’s take it on face value and assume they do. If this were a reflection of the whole industry, it’s going to be interesting to see how these people will be won over by the cloud providers.

The comments resulted in a few interesting things. I’ve responded to some of them here.

  • “Lack of cloud skills.” We all have to start somewhere. I would suggest starting with small proof of concept (POC) projects to test the water.
  • “Unreasonable Oracle licencing restrictions.” In case you don’t know, the core factor doesn’t apply to clouds other than Oracle Cloud, which makes Oracle licensing more expensive on non-Oracle clouds. Of course, everything can be negotiated.
  • “Lack of availability of Cloud experts to assist/advise.” I’m sure there are lots of people that claim they would be able to help, but how many with a proven track record is questionable. 🙂
  • “We have a massive legacy estate to consider.” Certainly, not everything is easy to move the the cloud, and the bigger your estate, the more daunting it is. I’m sure most cloud providers would love to help. 🙂
  • “Latency with fat client applications.” I had this conversation myself when discussing moving some of our SQL Server databases to Azure. It can be a problem!
  • “Seasonal businesses with uncertain money flow may not able to meet the deadlines for subscription payments.” Scaling services correctly could help with this. Scale down services during low periods, and scale up during high periods.
  • “The prime fear is being pulled off from the grid. Undependable internet connections.” Sure. Not every place has dependable networking.
  • “Bandwidth requirements & limited customization possibilities.” Ingress and egress costs vary with cloud providers. It may be intelligent design of your processes can reduce the amount of data being pushed outside the cloud provider. The cloud is very customisable, so I’m not sure what the issue is here, but I’m sure there are some things that will be problematic to some people.

Overall I think this was an interesting exercise. Even five years ago I would have expected the responses to skew more in favour of on-prem. Barring some huge change in mindset, I would expect the answers to be even more in favour of cloud in another 5 years.

Regardless of your stance, it seems clear that familiarity with cloud services should be on your radar, if it’s not already. Your current company may not be fans of the cloud, but if you change jobs the cloud may be a high priority for your new company.

Cheers

Tim…

PS. I’ve been running my website on AWS since 2016 . I started to write about some services on AWS and Azure in 2015. I’ve been playing with Oracle Cloud since its inception in 2016 (I think). Despite all this, I consider myself a dabbler, rather than an expert.

Video : SecureFiles Shrink in Oracle Database 21c

In today’s video we’ll demonstrate the SecureFiles Shrink feature, introduced in Oracle database 21c.

The video is based on this article.

You might also find these useful.

The star of today’s video is Arman Sharma. This clip was from Sangam15. To think that was seven years ago is mind-blowing!

Cheers

Tim…

Video : The PREPARE DATABASE FOR DATA GUARD Command in Oracle Database 21c

In today’s video we demonstrate the PREPARE DATABASE FOR DATA GUARD command, introduced in Oracle database 21c.

The video is based on this article.

The star of today’s video is Osama Mustafa, who was looking like a Teenage Mutant Ninja Turtle at that OpenWorld. 🙂

Cheers

Tim…