You’ll have heard me barking on about automation, but one subject that’s been conspicuous by its absence is the automation of SQL and PL/SQL deployments…
I had heard of some products that might work for me, like Flyway and Liquibase, but couldn’t really make up my mind or find the time to start learning them. Next thing I knew, SQLcl got Liquibase built in, so I figured that was probably the decision made for me in terms of product. This also coincided with discussions about making a deployment pipeline for APEX applications, which kind-of focused me. It’s sometimes hard to find the time to learn something when there is not a pressing demand for it…
Despite thinking I would probably be using the SQLcl implentation, I started playing with the regular Liquibase client first. Kind of like starting at grass roots. If you are working in a mixed environment, you might prefer to use the regular client, as it will work with multiple engines.
Both these articles were written more than 3 months ago, but I was holding them back on publishing them for a couple of reasons.
I’m pretty new to this, and I realise some of the ways I’m suggesting to use them do not fall in line with the way I guess many Liquibase users would want to use them. I’m not trying to make out I know better, but I do know what will suit me. I don’t like defining all objects as XML and the Formatted SQL Changelogs don’t look like a natural way to work. I want the developer to do their job in their normal way as much as possible. That means using DDL, DML and PL/SQL scripts.
I thought there was a bug in one aspect of the SQLcl implementation, but thanks to Jeff Smith, I found out it was a problem between my keyboard and seat. 🙂
With a little cajoling from Jeff, I finally released them last night, then found a bunch of typos that quickly got corrected. Why are those never visible until you hit the publish button? 🙂
The biggest shock for most people will probably be that it’s not magic! I’m semi-joking there, but I figure a lot of people assume these products solve your problems, but they don’t. Both Flyway and Liquibase provide a tool set to help you, but ultimately you are going to need to modify the way you work. If you are doing random-ass stuff with no discipline, automation is never going to work for you, regardless of products. If you are prepared to work with some discipline, then tools like Liquibase can help you build the type of automated deployment pipelines you see all the time with other languages and tech stacks.
The ultimate goal is to be able to progress code through environments in a sensible way, making sure all environments are left in the same state, and allow someone to do that promotion of code without having to give them loads of passwords etc. You would probably want a commit in a branch of your source control to trigger this.
So looking back to the APEX deployments, we might think of something like this.
A developer finishes their work and exports the current application using APEXExport. It doesn’t have to be that tool, but humans have a way of screwing things up, so having a guaranteed export mechanism makes sense.
Code gets checked into your source control. This includes any DDL, DML, packages, and of course the APEX application script.
A new changelog is created for the work which includes any necessary scripts, including DDL and DML, as well as the APEX script, all included in the correct order. That new changelog for this piece of work is included in the master changelog, and these are committed to source control.
That commit of the changelog, or more likely a merge into a branch triggers the deployment automation.
A build agent pulls down the latest source, which will probably include stuff from multiple repositories, then applies it with Liquibase, using the changelog to tell it what to do.
That sounds pretty simple, but depending on your company and how you work, that might be kind-of hard.
The master changelog effectively serialises the application of changes to the database. That has to be managed carefully. If stuff is done out of order, or clashes with another developer, that has to be managed. It’s not always a simple process.
You will need something to react to commits and merges in source control. In my company we use TeamCity, and I’ve also used GitLab Pipelines to do this type of thing, but if you don’t have any background in these automation tools, then that part of the automation is going to be a learning curve.
We also have to consider how we handle actions from privileged accounts. Not all changes in the database are done using the same user.
Probably the biggest factor is the level of commitment you need as a team. It’s a culture change and everyone has to be on board with this. One person manually pushing their stuff into an environment can break all your hard work.
I’m toying with the idea of doing a series of posts to demonstrate such a pipeline, but it’s kind-of difficult to know how to pitch it without making it too specific, or too long and boring. 🙂
First up was Matt Fooks speaking about “Microsoft Cloud App Security”. Matt covered a number of use cases, including shadow IT detection, log collection, checking compliance of applications and using policies to protect systems. He demoed a few of these. The flexibility of the cloud is great, but it also allows you to create a security nightmare as your cloud estate grows. MCAS gives you visibility and control over that. I guess the value of this will depend how far down the cloud journey you are. If you’ve got a bit of IaaS that’s being managed centrally, this isn’t going to sounds too interesting. Once you open the gates and let other people/teams get involved in provisioning services, you are going to need something like this to keep some level of control over the sprawl.
I heard one of the attendees mention Snowflake, so I collared him during the break to discuss it. I’m not so interested in the headline stuff. I care more about the boring day-to-day stuff, as people tend not to talk about it. It was a really interesting. Networking is great.
Next up was Richard Harrison with “The Journey from being a DBA Guy for 20 years to becoming an Azure Guy”. Richard was a fellow Oracle ACE before he moved in a new direction, so it was good to see him again. We spent some time chatting before his session, and I kept him for ages after the session chatting about a bunch of cloud related stuff. As the name of the session suggests, this was about his transition. What made him decide to make the move. His first opening into the world of the cloud. Some of the steps along the way and the amount of work involved. Richard is a smart guy, so when he says it’s hard work to keep on top of things due to the rate of change in the cloud, that should be a warning sign for people who are focused on the status quo.
There were some pieces that related nicely to the first session. For example, he discussed the control/governance aspect. To paraphrase, some services like security, budget management and databases are kept under central control, because of the critical nature of them, but people are given pretty much a free reign with platforms within their resource group. Why? Because there are loads of services with loads of features and trying to manage them centrally is practically impossible. I think of this as a move to a high-trust culture. Of course, you have tools to monitor what’s going on to stop people doing crazy stuff, but ultimately you have to start releasing control (where it’s appropriate) or people will look elsewhere. 🙂
I’m hoping Richard will come back again and tell us some more about what he is doing, and I know he’s got some really interesting Oracle-related content that will work really well at Oracle events, so hopefully he’ll throw his hat into that arena too. I don’t want to say anything more because I don’t want to steal his thunder.
Thanks to everyone who turned up to support the event, the speakers, and the sponsor Pure Technology Group. See you at the next event.
Good API design is not as simple as you might think, and making sure you pass back relevant information, like URLs for navigating through the services and maybe even service documentation links can make things a lot clearer.
The star of today’s video is James Morle. One of the OGs of performance!
You can check out my ORDS articles and YouTube ORDS playlist here.
I spent the holidays playing around with ORDS quite a bit, so I came back to work today and pushed it out across all Dev and Test installations.
As I’ve mentioned before, at work we run ORDS on Tomcat inside Docker containers. The build we use is very similar to this one I put on GitHub, but with some extra work-related bits added.
What did I have to do for this update?
Build a new version of our ORDS Docker image with version 19.4 of the ORDS and SQLcl software.
Remove all the containers based on this image and fire up new containers.
How long did it take to deploy this to all Dev and Test instances?
The build of the new Docker image took about 5 minutes. It’s mostly just unzipping the software. This can be done before we touch any running containers, so there is no downtime associated with this.
The removal and creation of all the containers took about 5 minutes as well. Each container is created in a second, but the first run with a new version of ORDS has to do the ORDS upgrade in the database, which takes a few minutes sometimes. If there were no ORDS upgrade, the containers start really quickly.
So effectively, in 5 minutes we replaced all the “kit” and ran the ORDS upgrade across everything. I could have done production in that same 5 minute span too, but I’m not allowed to yet. 🙂
Why am I talking about this?
It’s just another example of why containers make more sense than conventional app servers for this type of stuff.
To throw away kit and rebuild it from scratch takes an eternity here. I can do the equivalent with containers in seconds.
Once I’ve tested a new image and proved it works, I can roll that same image out across everything with no worries. If it works against one database, it will work against all the others. That’s the great thing about standardising the approach you take!
And another thing!
I’ve enabled SQL Developer Web on every Dev/Test installation too. Now all I’ve got to do is wait for the right opportunity to use it to save the day when someone is waiting for a firewall change, and act all casual like it’s no big thing! 🙂
So in summary
Containers good! ORDS good!
If you are interested in playing with Docker, you can find more information here.
I spotted it this morning, downloaded it and installed in straight away. I had no installation dramas on Windows 10, macoS Catalina and Oracle Linux 7 hosts.
The problem *for me* was the current version of Vagrant (2.2.6) doesn’t support VirtualBox 6.1 yet. I can’t live without Vagrant these days, so I installed VirtualBOx 6.0.14 again and normal life resumed. See Update.
I’m sure there will be a new release of Vagrant soon that supports VirtualBox 6.1, but for now if you use Vagrant, don’t upgrade to VirtualBox 6.1 yet. I’m sure you won’t have to wait long… See Update.
Update 1 : A couple of people Peter Wahl and Andrea Cremonesi pointed me at this post by Simon Coter, which contains config changes to allow Vagrant 2.2.6 to run with VirtualBox 6.1.
Update 2 : I’ve followed Simon’s post and it worked fine. If you are using Windows 10 as the host and have done a default installation of Vagrant, the files he’s discussing are in these directories.
Well, it seems 2019 was another slightly bizarre year for me.
I just looked back on last year’s review (here) and I’m guessing I had “resting bitch face” while I was reading it…
One of the things I mentioned last year was I was taking a year off presenting. I had two events I had already committed to at the start of the year, then dropped off the face of the earth for the rest of the year. I came out of retirement for OpenWorld. I was originally going to pull out of that also, but the wife persuaded me I should go. I’m glad I did.
The reason for not presenting was really to give myself a break. I had been having a lot of trouble travelling, which I suspect is mostly down to being so fat. It was good not to have the hassle of travelling, but I did miss seeing folks. I’ve done some presentations at work during the year, so it’s not like I’ve done nothing…
The problem with not doing conferences was that work was very busy and I didn’t take many holidays, so I was basically swapping one set of stress for another.
I’m thinking I might do a few things this year. I’m not going to go mad and try and speak at loads of events, but I’ve got to get back on the horse. Of course, the first thing is to think of something to present…
If you’ve followed the blog, you’ll know I’ve been going to a few local meetups for Docker, DevOps and Azure. I’m a tourist, rather than a speaker. It’s good to do something different!
Last year I said I wanted to start doing some videos on my YouTube Channel again. I was “on a break” for a while before that. Well I started at the end of January and I think I’ve done about 44 videos this year. There are a few new playlists, and some additions to existing ones listed here.
I like doing the videos, and it alleviates my guilt at not presenting, but they do take up a lot of time. I know some people have asked for longer, more in-depth videos, but I would estimate that for every minute of the final video, it takes me about 60 minutes of work, so a 5 minute video takes me about 5 hours. With that in mind, the chances of me doing a long-form video are remote.
I know some people can just record themselves talking, but presenting takes a lot of prep for me. For a live presentation I work for days/weeks so I can look casual. For videos it’s a little easier as you can edit stuff and trim it down, but it still takes time…
If you’ve watched my videos, you’ll know I put in little cameos of people saying “.com” to finish my introduction line. I went to put out the latest collection of them (Volume 3) and noticed I had never uploaded Volume 2, so today I posted two short videos. Thanks to everyone who has helped.
Work continues to be problematic. I could blame the company, but ultimately it comes down to me. I’ve got to learn to walk away. I keep doing stuff in an attempt to make a difference, but things progress at a glacial pace and I get frustrated and think if I do more I can get things moving faster. I can’t. You can lead a horse to water, but that doesn’t make it a duck.
I’m going to try and stop moaning about work, and focus on fixing me. It’s too easy to get a victim mentality about this. I don’t need the job. I know the problems there are not my fault. Fuck ’em!
The website it ticking along as usual. The numbers are similar to last year. I did over 130 blog posts and a hundred articles. I’ve got a bunch of other stuff waiting in the wings, but I can’t publish them yet. In a couple of cases I’ve got multiple related articles, but they are kind-of blocked until some bugs get fixed. It’s a little frustrating, but…
Maintenance of the website takes more time each year. The de-support of the non-CDB architecture in Oracle 20c has got people into a PDB frame of mind, and a lot more people have started to read the multitenant articles on my website. I’ve had to go through them a few times making changes to make things a little clearer, or put in some updates where a feature has changed slightly. Today Patrick Jolliffe sent me a message about a change, so I had to work through it, check it and amend the article. It’s a dirty job, but someone’s got to do it!
Similar to last year, quite a lot of effort has been going into my Docker and Vagrant stuff on GitHub. It’s something I find odd, as it takes up a lot of time and doesn’t really result in anything for my website. Sure, I use them to test stuff and I get a lot of good feedback about them, but I’ve spent nearly 20 years putting stuff on the website, and it feels odd to “produce content” for something that’s not my website.
Nephew #1 has just turned 18 and nephew #2 has turned 15. December is a busy month. It’s hard to believe how time flies. I bought my house when the couple I shared a house with got pregnant with their first child. That child is now at Uni doing dentistry, and I’ve not got round to buying furniture or curtains yet. I’m not even going to pretend I’m going to do something about it by next year.
Physically I’m a bit of a wreck. I’ve really got to get my shit together, especially if I’m going to start travelling again!
Everything changes, and yet nothing changes… Let’s see what 2020 brings. I hope it’s good for everyone!
Having complained about how the female lead character was used in The Unicorn Project, I figured it was time to read this book and see it done properly again.
I’m not going to include any full-on spoilers, but some things might give the game away, so don’t read this if you’ve not already read the book.
During the first sequence in the book I got a pretty good idea what was coming and I was like, “No. No. Nooooooooo!” I’m not going to say what it was, but it was a major kick in the gonads…
Tiffany Aching is a great character. It would be so easy for a lesser writer to make her a Mary Sue, but Pratchett keeps her multi-dimensional. Sometimes strong, clever, and at times ruthless. Sometimes self-doubting and almost naive.
As you would expect for this part of the disc (world), there are a number of familiar characters. It’s wrong to say any character in Discworld is “my favourite”, as it changes with each book, and sometime several times in a single book. This book contained several of my favourite characters. Some old and some new. 🙂 There was also a brief appearance by Horace, a Blue Lancre cheese made by Tiffany, who was known to eat mice, and as it turns out is capable of fighting alongside the Nac Mac Feegle. I’m ashamed to admit I had forgotten about him until he was mentioned in a scene.
This was quite an emotional roller coaster ride of a story. Partly because of the story itself and the characters involved. Partly because it was the last of a 41 book series, which I loved. Partly because of the reason for why it was the last book.
Oh well. Happy days!
For those that don’t know, here’s a little bit of history…
Despite having a degree and a PhD, I had somehow managed to remain pretty terrible at reading. I suspect a mild case of dyslexia maybe. In 2006, at the tender age of 37, I decided to force myself to start reading in an attempt to improve.
I started by reading The Vampire Chronicles by Anne Rice. In 2007 I started to read the Discworld series and got hooked. I read all of the books (very slowly) that existed until I was done, then kept reading each new one as they arrived. I’ve dipped in and out of other things since then, but by far the majority of the books I’ve read in my life were part of this series. We are fast nearing the end of 2019, and that part of my reading life is now over. 🙁
For those that care, I am substantially better at reading now. Part of that is practice of course. Part of it is not beating myself up about being crap at reading anymore. Part of it is the additional confidence public speaking has given me. I’m still pretty terrible at reading out loud, but I’m less bothered by the mistakes now. 🙂
So for the part you played in that process, thanks Terry!
I loved The Phoenix Project, but absolutely hated The DevOps Handbook, so I was a little reluctant to start reading The Unicorn Project, as I was really worried I would hate it, and it would tarnish the memory of The Phoenix Project.
Overall it was fine, but IMHO it was nowhere near as good as The Phoenix Project.
I’m not going to talk details here, but instead talk about my feelings about the book. You don’t have to agree. 🙂
Let’s talk about Maxine
In The Phoenix Project, Bill was an experienced manager, but he was asked to take over a role that was totally out of his comfort zone. He was mentored by Eric, who helped to develop him in his new role using “The Three Ways” and giving him an understanding of the 4 types of work, and how they affect productivity. Bill was gradually introduced to the three ways, one at a time, and we as the reader went on that journey with him and his colleagues. There was a definite story arc and a clear development of his character.
In The Unicorn Project, Maxine is a super God-mode developer/architect that has done pretty much everything before, and is totally amazing at almost everything. The first two chapters tell us this repeatedly. At the start of the book she is already the finished article. As a result of this there seems little in the way of character development here. Where do you go from amazing? When she does interact with Eric, he basically brain-dumps “The Five Ideals” in one shot to Maxine and friends, and they pretty much run with it. The story arc and development of Maxine as a character, and most of the other characters also, is weak in comparison to Bill’s story from the first book.
You see this problem in reality TV competitions. If a person’s first audition is “too good”, they won’t win the show. The show is built around the journey. The alternative is to craft a back story that fakes a journey, which is why they didn’t mention Kelly Clarkson had already recorded demos and turned down 2 record contracts before she auditioned for American Idol. The character arc is built around the journey from waitress to diva.
The lack of “progress” of the lead character is the biggest problem with this book from my perspective. I know it’s a DevOps book, but it’s not a reference book. It’s meant to be like The Phoenix Project, which uses a story to convey the message.
I like the fact there is a female lead character, and I certainly understand the problem with making her a newbie, mentored by an old white guy :), but I think this situation caused a problem with the story line, and ultimately how “The Five Ideals” were presented to us as the reader. The concepts and meaning of them should have been drip fed to us, like they were in the first book. Almost making us feel like we’ve discovered them for ourselves.
As far as I see it, one of the following would have solved this problem.
Leave Maxine as a super God-mode developer, and have her introduce the rest of the characters in the development teams to “The Five Ideals”. Maybe she learnt “The Three Ways” from Eric in the past, and developed them further, giving them a more development focused slant? I guess how you play this depends on how wedded you are to the idea that Eric is involved in this part of the company transformation. She could have just figured this stuff out for herself.
Make Maxine less capable, and have a female mentor that teachers her about “The Five Ideals”, allowing her story arc to be similar to Bill’s, and allowing us to go on the journey with her. If you need a link back to Eric, you can always make this mentor one of Eric’s proteges or a former colleague. Hell, make her the person who taught Eric.
Make Maxine less capable and let Eric mentor her, like he did with Bill. Yes, people will criticise the fact you showed her as less capable, but the development of her character would have been more interesting.
So the problem isn’t the fact the lead character is a woman. There is no journey!
Why do you care about the story?
The great thing about The Phoenix Project was we learned about The Three Ways and the types of work as part of the story. It was not a technical reference book, but allowed someone new to the concepts to understand them, and see why they were important. I think a lot of people from a less technical background could pick up the book and see exactly why this stuff made sense. It was in itself a good tool to promote change.
The main purpose of The Unicorn Project is to tell us about “The Five Ideals” and how they relate to the three horizons framework. I don’t think it did a good job of that. When I finished the book I found myself asking, “Did they really explain any of this stuff properly?” I Googled some of the concepts to make sure, and learned more in a couple of pages than I did in the whole book.
I feel like you would get more value out of reading The Phoenix Project, then following it up with this interview with Gene Kim.
It’s interesting that in the interview Gene Kim says, “Maxine who is a very talented architect, knows the five ideal patterns.” From where? Just from what Eric said? OK. Is she meant to be relating what Eric tells her to what she already knew? If so, how did she find this stuff for herself? I feel like that journey is a lot more interesting, and likely more informative than what we actually got. After all, that journey is the one we saw Bill make in the first story.
After reading the interview I can see what was actually meant to be happening here was Maxine was just observing what was going wrong, whilst already knowing the solution. Eric was effectively irrelevant to her progress. In fact, her progress was not the point. It was the progress of the project that was the point. I don’t think I got that from the book at all, and if that were the case, why not go with my first suggestion and make her the “Eric” of this book? I think that would have worked a lot better! Maxine is Yoda. That works for me.
As it stands, I don’t think you can give this book to someone and say, “This is why DevOps matters”, in the same way you could with The Phoenix Project. If that’s what it was meant to be, then I think it has probably failed. If it’s a rallying call for people who already know about DevOps, then it’s probably not too bad, but could have made most of the important points in a blog post.
I see some very gushy comments about how “amazing” the book is. That is not the book I read, although you will read the word “amazing” all the time in it…
What really matters I guess
The Five Ideals are pretty much a different take on The Three Ways. The Phoenix Project and The Three Ways probably feel more infrastructure focused, while The Unicorn Project and The Five Ideals are more development focused. At least, that’s what my boss tells me. 🙂 That might make The Five Ideals more attractive to a section of the audience, as they may feel more relevant. Being someone that bridges the infrastructure and development gap, I think they both apply equally well really, but I can see how they might have a different appeal. They are different ways of stating the same thing.
There are some great messages in this book, and I agree with the vast majority of them. If you can focus on “the message”, I think you will enjoy it a lot more than me. Having said that, it feels really badly written compared to The Phoenix Project. To the point where I can hardly believe these two books are by the same author. Better editing, and maybe reducing many chapters to two thirds or half their current size, could have given this more punch and made it much better. Better than The Phoenix Project? I don’t know, but as it stands it feels like a pale imitation of it. I keep wanting to say the word “clumsy”, and everything about it feels that way. Having said all that, there were a couple of chapters towards the end which were quite exciting, so it’s not all bad.
Despite all this, I think it will be a valuable source of quotes, or paraphrased statements, and similar to The Phoenix Project, it will be used to help effect change in stubborn organisations. For that alone I guess we should be grateful.
I wonder how I would have felt if I had read this book first?
I’m interested to know what others felt. Maybe I was just expecting too much, having been such a big fan of The Phoenix Project, which I guess you already figured out. 🙂