Birmingham Digital & DevOps Meetup : August 2019

Yesterday evening I went along to the Birmingham Digital & DevOps Meetup for the first time. It followed the usual meetup format of quick intro, talk, break, talk then home.

First up was Elton Stoneman from Docker with “Just What Is A โ€œService Meshโ€, And If I Get One Will It Make Everything OK?” The session started by describing the problems associated with communication between the building blocks of a system, and how a service mesh can alleviate some of them. It then moved on to some service mesh demos using Istio. These included examples of altering the routing of traffic to do canary testing and targeting specific groups etc.

Elton was really honest about the learning curve, issues and overhead associated with this sort of setup. One comment I really liked was when he showed a slide containing the following, saying that often people assume there is a progression from left to right.

Meaning people assume you learn Docker, then you need some form of orchestration so you learn Swarm. From there you naturally progress to Kubernetes and once you understand that, you will inevitably move on to a service mesh using something like Istio. Elton’s point was you don’t *have to* continue on this progression. You can step off at any point once you’ve achieved the functionality you need. I think this is a really important point and I can see it reflected in what I do with Docker. We’ve got some things that stop at just using Docker containers, with no orchestration at all. I work on a project that requires some orchestration, so we use Swarm, which is really easy to use. So far I’ve had no reason to go beyond Swarm, and even considering a service mesh is so far down the line for us. I’m not discounting the relevance of these for everyone, but they don’t make sense for me at this point.

It was a really good session and I learned a lot. You can check out Elton’s blog here.

After the break it was James Relph with “Container Security Fundamentals”. This started of with a basic introduction to containers, using that as an entry point to explain how containers can be problematic from a security perspective, and what you can do to reduce the impact. He covered a lot of stuff, some of which I already do, some I know about and some stuff that was new to me. This is not an exhaustive list.

  • Don’t automatically trust images from Docker hub. Do your due diligence, even when they are from a reputable source.
  • Use your own image repository. He mentioned ECR amongst others. This can be used for your own images, but also base images from Docker Hub, which you have verified.
  • Don’t use “latest”, but use specific tagged versions. Latest gives you all the latest fixes, but all the latest bugs too. You should test and verify before you let images out into your infrastructure.
  • Multi-stage builds to reduce the size of containers and minimise the attack surface. Basically, copy out what you need and leave the crap behind.
  • Using sidecar containers to provide specific services, allowing your application images to remain more focused. The sidecar images can be maintained by feature experts to make sure they are as secure as possible.
  • Scanning images using Clair, amongst other things, to check for dodgy software. One of the audience mentioned Anchore.
  • Using microVMs like Firecracker to provide additional isolation, whilst retaining the ease of use of containers. I’ve not played with this, but I have tried Kata Containers, which seems to do pretty much the same.

There was a lot in there!

I was a bit nervous going into the event thinking it would all go over my head, and some of it probably did, but it was cool. I got to speak to a few people before the event, during the break and at the end. It seemed like there were quite a mix of people there from beginners in these areas upward, so I didn’t feel out of place.

A few times I found myself thinking, that’s great, but what do I do about my 3rd party applications? I’ve written before (here) about how 3rd party apps screw everything up. ๐Ÿ™‚

Thanks to Elton Stoneman and James Relph for taking the time to come and speak to us. Thanks to the folks from BrumDigitalDevOps for organising the event, and to Capgemini UK for sponsoring the event.

Cheers

Tim…

Video : Vagrant : Oracle Database Build (19c on OL8)

Today’s video is an example of using Vagrant to perform an Oracle database build.

In this example I was using Oracle 19c on Oracle Linux 8. It also installs APEX 19.1, ORDS 19.2, SQLcl 19.2, with ORDS running on Tomcat 9 and OpenJDK 12.

If you’re new to Vagrant, there is an introduction video here. There’s also an article if you prefer to read that.

If you want to play around with some of my other Vagrant builds, you can find them here.

If you want to read about some of the individual pieces that make up this build, you can find them here.

The star of today’s video is Noel Portugal. It’s been far too long since I’ve seen you dude!

Cheers

Tim…

Walk the walk : Delivering is more important than talking about delivering!

This post could be about so many things, but I’m going to try and keep it technology related.

I like to talk and clearly after 19+ years of doing the website and more than a decade of presenting it’s clear I’m not afraid of making my opinion known. I like to think it’s an educated opinion, based on experience and the ability to actually deliver on it, or be part of a group that delivers.

What I find increasingly difficult is dealing with people who can talk the talk, but can’t walk the walk. It’s really easy in our industry to say the right things and be seen as someone that is making a difference, when actually it’s all just hot air. All that really matters to me is that you can deliver. It’s nice if I like the people I work with, because it makes the workplace more pleasant, but that’s a bonus. I don’t have to like you to work with you. I just want you to deliver on what you promise.

The internet is both the greatest and the worst thing rolled into one. Everyone is a self-proclaimed expert about things before they’ve even finished reading the book. Everyone presents their “hello world” app as proof they’ve mastered the tech. Everyone is a certified rocket surgeon, but has never seen a rocket.

So what do you do when you encounter people like this? You ask questions! Most of these types break down under questioning. Drill into the subject and ask for examples of projects etc. If they know their stuff they should be able to answer, and will probably get a kick out of telling you about it. If not, just their manner will often give the game away, even if you know nothing about the subject yourself. As the saying goes, “You don’t need to know how to swim to recognise when someone is drowning!” It’s usually pretty easy to spot when someone is full of ๐Ÿ’ฉ. I know who you are!

Cheers

Tim…

PS. Don’t even get me started on idiots who can neither talk the talk, or walk the walk, but still seem to get promoted to their level of incompetence.

PPS. I work with several. Can you tell I’m pissed off?

I’m 2% DevOps, 3% agile and 4% automated because of 3rd party apps…

I was having a discussion with my boss about the impact of 3rd party apps on the way we work, and how difficult things are when you have to deal with 3rd party apps, as opposed to just writing your own software.

It’s easier to do things well when you are in control of all the pieces. Most of the examples you see are people writing their own software, typically on new projects. That’s very different to dealing with old projects and 3rd party apps. I’ll give you some examples, without trashing the companies responsible for this.

Example 1

Our student system is provided by a 3rd party. The company in question has a really antiquated way of delivering applications. In recent years they’ve tried to resolve this by writing their own delivery mechanism, made up of some custom software and Jenkins. The problem is, this is just a wrapper over the old process, so it is not the most reliable tool in the world. Someone like me would describe it as putting lipstick on a pig.

In addition to that, you have to use a GUI to perform the operations. At this point there is no API to allow you to script operations, which makes building them into a bigger process really problematic. We have internal development which is gradually moving to something resembling CI/CD, but it will never truly meet that goal, because we have to include manual management of things because of the limitations of the 3rd party software.

I’m sure long-term customers see the new delivery mechanism as a great improvement, but it’s not something you would deliver for a new product. It’s less painful than it was, but not really good.

Example 2

We have a publishing system that is written in Java and runs on Tomcat. It is so close to being hands-off, but there are a couple of problems.

  • When you deploy a new version, it starts in maintenance mode and you need manual interaction to click an OK button a few times on a web-based maintenance screen. I’ve never “not clicked” the OK button, so I just want a “just do it” option, so I can let it get on with it.
  • When some features are enabled by the power users, the next restart of the application flips you into maintenance mode. We’ve had P1 incidents because a host failure has caused the VM to start on a new host, and because a user has enabled a new feature in the app, the automatic startup stalls, waiting for me to click the OK button a few times.

There are some other annoyances, which impact on availability and possible topology, as well. There is no way to resolve these because of limitations in the application. All we can do is raise enhancement requests with the vendor.

I could go on with more examples, but I think you get the message.

So what do you do?

It can be quite disheartening when you want to do things well, but you have to keep compromising because of factors outside your control. You have to try not to give up, and just keep plugging away.

  • Don’t make unrealistic comparisons between your environment and others. There’s no point comparing your mixed environment to a software house. I’ve worked in both. They are very different. Take what works. Ditch what doesn’t.
  • Semi-automated processes are better than processes that are 100% manual. Maybe you can use Robotic Process Automation (RPA) to automate what is essentially a manual process.
  • Try to make sure these considerations become part of your procurement process, or you will keep buying crap.
  • Try to be creative and find workarounds, don’t just bury your head in the sand. There’s always *something* you can do to improve things.
  • Even if something is terrible, that doesn’t stop you improving the processes around it.

I guess you should focus on the values, rather than trying to exactly match some prescriptive ideal.

Good luck!

Cheers

Tim…

PS. I’m pretty sure my boss is reading this laughing, as I’m following none of this advice myself, but instead stomping round the place like a thirteen year old having a strop because, “Everything is crap!” ๐Ÿ™‚

ORDS, SQLcl, SQL Developer 19.2 (Vagrant and Docker Builds)

The folks at Oracle dropped some new presents for us today, including version 19.2 of the following.

I’ve updated my Vagrant builds and ORDS Docker builds with the new versions and everything seems to be working fine so far.

Tomorrow I’ll probably try out some of our development ORDS containers with these releases and see how they work out. They are similar to this build, so I’m sure they will be fine…

Cheers

Tim…

Update: I rolled ORDS 19.2 out to all our Dev/Test environments this morning. We run them all on Docker, so it was really quick and easy. ๐Ÿ™‚

Driving vs. Being Driven : The reason you fail to get good at anything!

It doesn’t matter how many times I’ve gone somewhere. I only know the route when I’ve driven there myself. Everything makes sense when you see someone else do it. You don’t realise how distracted you are, and how much you’ve missed until you have to do it for yourself.

When we have consultants on site to help us with something new, I assume I’m going to drive and they are going to give directions. I make notes as necessary, but the main thing is *I’ve done it*, not them. If I’m told I have to “observe and make notes”, I say I’m not willing to support it, as experience tells me there will be important stuff that gets missed as the consultant rushes through it. Once again, it’s the difference between driving and being driven.

I’ve written a lot about Learning New Things, and I think it always starts with learning to learn for yourself. If you are always relying on other people to lead the way, they are driving and you are being driven. They are getting better and you are just drifting.

I suppose the obvious retort to this is,

โ€œOnly a fool learns from his own mistakes. The wise man learns from the mistakes of others.โ€

Otto von Bismark

There is some truth in that, but the import thing in the second sentence is the wise person *learns* from the mistakes of others. There is still something active going on here. You are learning, not just being passive and waiting to be told what to do.

Standing on the shoulders of giants requires you to climb up on to the shoulders in the first place!

Cheers

Tim…

MobaXterm & KeePass Updates

Yesterday I noticed some updates to a couple of tools I use all the time.

MobaXterm 12.0

Followers of the blog know I’ve had a long term love affair with MobaXterm. If you are using Windows and connect to servers using SSH, this is the best tool I’ve come across.

Downloads and Changelog are in the usual places.

KeePass 2.42.1

I use KeePass as my password manager.

Downloads and Changelog are in the usual places.

You can read about how I use KeePass and KeePassXC on my Windows, Mac and Android devices here.

Cheers

Tim…

Docker : New Builds Using Oracle Linux 8 (oraclelinux:8-slim)

Yesterday I noticed the oraclelinux section on Docker Hub included “oraclelinux:8-slim”, so when I got home a did a quick run through some builds using it.

  • ol8_ords : This build is based on “oraclelinux:8-slim” and includes OpenJDK 12, Tomcat 9, ORDS 19, SQLcl 19 and the APEX 19 images.
  • ol8_19 : This build is based on “oraclelinux:8-slim” and includes the 19c database and APEX 19.
  • ol8_183 : This build is based on “oraclelinux:8-slim” and includes the 18c database and APEX 19.

There are also some new compose files, so I could test database and ORDS containers working together.

Everything worked fine, but here come the inevitable warnings and comments.

  • The Oracle database is not certified on Oracle Linux 8 yet, so the database builds are just for playing around, not a recommendation.
  • The database preinstall packages don’t exist yet, so I installed the main required packages with DNF, but I didn’t do some of the additional manual setup I would normally do, so it’s not a perfect example of an installation. I assume the preinstall packages will eventually be released, and I will substitute them in.
  • The ORDS build is not subject to the same certification restrictions as the database, so as far as I know, I could consider using this, although the build I use for work differs a little to this and is still using Oracle JDK 8 and Tomcat 8.5.

If you are interested in playing around with Docker, you can find my articles on it here, and my public builds here.

Cheers

Tim…

Video : Vagrant : A Beginner’s Guide

Today’s video is an introduction to Vagrant, which I use to build test systems with VirtualBox.

This video is based on the following article.

The star or today’s video is Christian Antognini, who is being drowned out by the noise of a plane. ๐Ÿ™‚

Cheers

Tim…

PS. Sorry if you kept getting part way through, only to have the video be removed. I kept spotting mistakes, rendering artefacts and strange things YouTube was doing to the uploaded video.

An Eye for Efficiency : Why you are crap at your job!

One of my colleagues says that I think everyone is crap at their job, and to be honest that’s probably true. Most people are terrible, but they have so little self awareness they actually think they are good. The few people I think are good are those that have some self awareness and have an eye for efficiency. This isn’t just about technology, you can exercise these muscles in everyday life. I mentioned one example of how people approach parking barriers here, but here are some other things I’ve witnessed/experienced.

Shop Checkout

There’s a small, but busy, shop I go to several times a week. The process all the checkout staff go through is like this.

  • Scan all items, leaving them balanced on the checkout in a rather messy fashion.
  • Ask if you want a bag.
  • Pack all those scanned items into the bag.
  • Ask how you want to pay. If you pay by card, which most people do, they type in a code, wait a couple of seconds, then you touch your card to pay.

This drives me insane for a couple of reasons.

  • If they asked about the bag at the start, they could scan straight into the bag. This would save a significant amount of time in itself.
  • They could ask, “Are you going to pay by card?”, whilst they are scanning, and type in the code immediately once the last item is scanned.

Both items would shave quite a number of seconds off the transaction time. For each basket it might be just 30 seconds or so, but when there is a queue of people, which is very often, it makes a big difference. I stand there going crazy wanting to say something, but realising they will think I’m being a dick…

I worked in shops as a kid. I know how you should handle a checkout. In my day we didn’t have the scanners, so you would memorise the prices of popular items so you could get them through the checkout quicker than having to read the price tag then type it.

It amazes me the people on the checkout can’t see this and fix it themselves. It saddens me that their boss hasn’t taken the time to observe them and see this issue, then correct it. I guess they think they just need more staff. ๐Ÿ™

Production Line

I’ve done a couple of production line jobs in summer holidays during University. In one job I worked packing garlic bread for 3 months. There were several stations in the line, and not surprisingly the line manager tried to move people between the stations to keep the flow of product consistent between all stations. I worked on the last station, which involved putting the packaged garlic bread into a cardboard sleeve. It was murder on your hands. Although the line manager would add and remove people from our station, they never dealt with the final link in the chain, which was the real problem. Once we filled up a crate, someone had to walk it over to the other side of the room and bring back a new empty crate. That was one person missing from the station a lot of the time. I moved the crates next to our station and it was like I had done some witchcraft. It seemed like an obvious waste of time to me, so I dealt with it. I’m sure as soon as I left the crates were moved back to their original location, because that’s where they were meant to go…

In both these cases, and in the case of the parking barrier, all I’ve done is observe what is happening and think how it could be done better. I don’t think this needs a brain the size of a planet. It’s more about having the desire to see things running smoothly. Unfortunately, most people don’t seem to give a crap about that, which is why most people are crap at their jobs…

Now I could link this back to some discussion on automation, or the principle of flow in devops, but you should already be able to make that connection for yourself, and if you can’t, I don’t think me telling you is going to make a difference…

Cheers

Tim…