Category: Technology

A History of Tech Sprawl

Here’s a little story about how things are all the same but different…

Software Sprawl

Let’s cast our minds back to the bad old days, where x86 machines were so underpowered, the thought of using them for a server was almost laughable. In those days the only option for something serious was to use UNIX on kit from one of the “Big Iron” vendors.

The problem was they were very expensive, so we ended up having loads of software installations on a single box. In some cases we would have many versions of Oracle installed on a single machine, running databases of many different versions. In some cases those same machines also ran middle tier software too.

It may have been a small number of servers, but it was a software sprawl. To try and add some isolation to the sprawl we may have resorted to things like LPARs or Zones, but we often didn’t.

Physical Server Sprawl

Fast forward a few years and x86 kit became a viable alternative to big iron. In some cases a much better alternative. We replaced our big iron with many smaller machines. This gave us better isolation between services and cleaned up our software sprawl, but now we had a physical server sprawl, made up of loads of underutilized servers. We desperately needed some way to consolidate services to get better utilization of our kit, but keep the isolation we craved.

Virtual Machine (VM) Sprawl

Along comes virtualization to save the day. Clusters of x86 kit running loads of VMs, where each VM served a specific purpose. This gave us the isolation we desired, but allowed us to consolidate and reduce the number of idle servers. Unfortunately the number of VMs grew rapidly. People would fire up a new VM for some piddling little task, and forget that operating system and software licenses were a thing, and each virtualized OS came with an overhead.

Before we knew it we had invented VM sprawl. If ten VMs are good, twenty must be better, and put all of them on a physical host with 2 CPUs and one hard disk, because that’s going to work just fine! 🙁

Container Sprawl

Eventually we noticed the overhead of VMs was too great, so we switched to containers, which had a much lower overhead. That worked fine, working almost like lightweight virtualization, but it wasn’t special enough, so we had to make sure each container did as little as possible. That way we would need 50 of them working together to push out a little “Hello World” app. Managing all those containers was hard work, so we had to introduce new tools to cope with deploying, scaling and managing containerised applications. These tools came with there own overhead of extra containers and complexity.

We patted ourselves on the back, but without knowing it we had invented container sprawl, which was far more complicated than anything we had seen before.

Cloud Sprawl

Managing VM and container sprawl ourselves became too much of a pain. Added to that the limits of our physical kit were a problem. We couldn’t always fire up what we needed, when we needed it. In came the cloud to rescue us!

All of a sudden we had limitless resources at our fingertips, and management tools to allow us to quickly fire up new environments for developers to work in. Unfortunately, it was a bit too easy to fire up new things, and the myriad of environments built for every developer ended up costing a lot of money to run. We had invented cloud sprawl. We had to create some form of governance to control the cloud sprawl, so it didn’t bankrupt our companies…

What next?

I’m not sure what the next step is, but I’m pretty sure it will result in a new form of sprawl… 🙂

Cheers

Tim…

PS. I know there was a world before UNIX.

PPS. This is just a fun little rant. Don’t take things too seriously!

AI Search and the future of content creation

The recent announcements of GTP-4o by OpenAI and the AI updates from the Google IO keynote made me want to revisit the topic of how AI search will affect the future content creation. I’ve already touched on this here, but I think it’s worth revisiting the impact of AI search.

The view from the top

I’ve seen a few places talking about the Gartner post predicting a 25% reduction in search engine volume by 2026. This specifically relates to chatbots and virtual agents, but I think this figure could be higher if we separate AI search from traditional search.

Google have been experimenting with Gemini and search results for some time, hoping to offer a better search experience. According to the keynote, that service will become generally available soon. ChatGPT can already be considered a replacement for traditional search. Instead of doing a search and getting links, you just get an answer, which is after all what you are looking for.

Here lies the problem. If AI search presents answers directly, rather than referring you to the source websites, that represents a drop in traffic on the source websites. If there is indeed a 25% drop in traditional search by 2026, that will result in a drop of 25% in revenue for many online content creators.

Why is this a problem?

Professionally produced content will definitely be affected by a 25% reduction in traffic. Those content creators rely on traffic to their sites for their ad revenue. Without this, they can’t pay their workers. I don’t think many companies or people would be happy about a 25% cut from in their earnings.

The money from online advertisements has already fallen drastically over the last few years. Speaking from personal experience, for the same volume of traffic I’ve already seen ad revenue drop to about a quarter of what is was a few years back. Assuming that is true for professional content creators who rely on this income, they have already been hit hard, and now are likely to get hit again.

Even for those that don’t make money from publishing content, having a drop of 25% in their readership can be a demotivating factor.

So what?

So some people lose money. So what?

Well, AI typically relies on original source material to provide the information in the first place. If content creators give up, where is that new source content coming from? An endless recycling of AI generated content? That seems like a race to the bottom to me.

I spend a lot of time on YouTube and in recent months I’ve noticed the rise of AI generated content. I click on a video that looks interesting, only to find what sounds like an AI generated script being read by a very generic voice. Lots of words that sound related to the topic, but ultimately nothing of substance, leaving you with the feeling that you just wasted your time. I could easily see this happening to online publishing in general. The signal to noise ratio is likely to get really bad.

And another thing

I’ve focussed mostly on text publishing, as I’m mostly about articles and blog posts. Clearly there are other areas that are going to be massively affected by this.

  • Images : Unless you’ve been living under a rock you will already know about the complaints by people claiming AI image generation has stolen their material or art style. For companies that sell images online, AI image generation means game over for their business.
  • B-roll : When you watch videos on YouTube, you will notice many channels making use of b-roll footage. High quality clips inserted into their video to give it a more professional feel. Companies make money selling b-roll clips. That business will pretty much end overnight once the latest video generation is widely available. Why buy b-roll footage, when you can generate it for free?

Conclusion

Initially I see this as a win for the consumer, as we will be able to get access to information, images and video clips much more easily than we can currently. My concern is the initial progress may be followed by a gradual decline in quality to the point where everything becomes soulless dirge.

Cheers

Tim…

Oracle VirtualBox 7.0.18, Vagrant 2.4.1 and Packer 1.10.3

Oracle VirtualBox 7.0.18

VirtualBox 7.0.18 has been released.

The downloads and changelog are in the usual places.

I’ve installed it on my Windows 10 and 11 machines. Both gave me issues, which I put down to the new version of VirtualBox, but on further investigation it was actually because of my new Vagrant box builds.

If I used the new version of VirtualBox with the old version of my Vagrant boxes the builds worked fine. If I tried to use one of the newly built Vagrant box versions it failed with this error.

The SSH connection was unexpectedly closed by the remote end. This
usually indicates that SSH within the guest machine was unable to
properly start up. Please boot the VM in GUI mode to check whether
it is booting properly.

I tried using “config.ssh.keep_alive = true” in the Vagrantfile, but that didn’t help. I’m continuing to investigate the issue, but it seems like VirtualBox 7.0.18 is working fine. It’s something with my box builds which is the problem.

Vagrant 2.4.1

Releases of VirtualBox prompt me to check for new versions of Vagrant. The current version is Vagrant 2.4.1. All my test systems are built with Vagrant, so I installed it as well.

If you are new to Vagrant and want to learn, you might find this useful.

Once you understand that, I found the best way of learning more was to look at builds done by other people. You can see all my Vagrant builds here.

I’ll be doing some updates to my Oracle builds over the coming days, so this will get a lot of testing.

Packer 1.10.3

I use Packer to rebuild my Vagrant boxes (Oracle Linux 7, 8 and 9) so they have the latest guest additions. For the reasons mentioned above I’ve not released the new version of the Vagrant boxes yet. You can see the old ones here.

If you are interested in creating your own Packer builds, you might take inspiration from mine, available here.

Once I get to the bottom of the SSH issues on the new builds I’ll update the boxes.

How did it all go?

Most things worked fine. As mentioned there is an issue with my Vagrant box builds, but I’m assuming that is something on my side of things. 🙂

What about the VirtualBox GUI?

Just a quick warning. I do everything using Vagrant, so I rarely look at the VirtualBox GUI. Remember, when I say everything worked fine, I mean for what I use.

Cheers

Tim…

What is the end goal for automation and AI?

I’ve been a big proponent of automation over the years. I recognise that automation will cause the loss of some jobs, but I’ve always assumed people would move on to other things. A consistent theme of mine was, why waste humans on mundane work when they could do something of more value?

I recently read The Naked Sun by Isaac Asimov and now I feel rather uneasy about automation and AI…

I don’t want to give too much of the story away, but we are told fairly early that the planet Solaria only has 20,000 humans, and millions of robots. The robots do everything as far as the economy is concerned, and the humans are essentially idle custodians. Rather than having millions of people doing a variety of occupations, there are only the elite sitting at the top of the pile…

As I was reading this I was thinking about the economic elite on our planet and I started to feel some concern. My dream of a post-scarcity society is one where we are all equal, but I can’t imagine there are many people at the top that want to be equal. They would always want to be more than equal. Call me a pessimist, but when I think of Elon Musk calling himself the emperor of Mars, I wonder if a future post-scarcity society will be less like Star Trek, and more like Solaria…

Of course, the AI could just turn into Skynet and kill us all, but I have this feeling that greedy humans using AI and automation against the majority of the population poses the greater threat. Why have “them and us”, if there could just be “us”? 🙁

All of a sudden automation has lost some of its lustre…

Cheers

Tim…

AI Prompt Engineer (AI-fu). The new Google-fu?

The other day I came across the term AI Prompt Engineer. It was in the context of being the next big thing in the job market. I did a bit of Googling and sure enough there are courses about it, and a number of jobs being offered. These seem to break down into two main types of role.

  1. A technical AI development role, involving understanding of AI and coding against various AI backends via APIs.
  2. A person who types in stuff to get the right outcome from specific AI tools.

As mentioned, the first is a technical development role, but the second seems like AI-fu to me…

Google-fu

Here is the Wikipedia definition of Google-fu.

“Skill in using search engines (especially Google) to quickly find useful information on the Internet.”

https://en.wiktionary.org/wiki/Google-fu

After all these years I’m still surprised how weak people’s Google-fu is. Searching for information requires knowing what to include in your search criteria, as well as evaluating the results to make sure they are actually what you need.

If you don’t understand how to filter the results using targeted search terms, you may get incorrect or misleading results. If you don’t understand the subject matter, you have no way of validating correctness of the responses.

AI-fu

I’ve made up the term AI-fu. I can’t see any references to it on the internet. The closest I can find is ChatGPT-fu, which I also made up here. 🙂 I said Google-fu is about providing the appropriate inputs, and validating the outputs. I would suggest that AI-fu is very similar.

If you’ve played around with ChatGPT you will know there is a degree of “garbage in, garbage out” involved. You have to refine your inputs, either by altering the original question, or following a chain of thought (CoT) process.

How do you know you have a suitable result? Well, you have to understand the subject matter, and do some fact checking to make sure you are not generating garbage. If you don’t understand that a human hand typically has 5 digits, you can’t tell if that 7 fingered monstrosity you’ve just generated is wrong. 🙂

Thoughts

I suspect those people that have acquired good Google-fu, will also acquire good AI-fu. Those people who have proved to have consistently weak Google-fu, are unlikely to ever developer strong AI-fu.

Cheers

Tim…

PS. If you want a certificate in AI-fu, please send £500 to my PayPal… 😉

Automation : Increasing pressure on an existing constraint

Yesterday I tweeted that I was reminded of this post.

I was reminded of it because of something that is happening to me at work, so I thought I would talk about it here.

Production lines

If you’ve read anything about DevOps you will know it came from manufacturing. If you didn’t know that, check out The Goal, which was the basis for The Phoenix Project.

Manufacturing typically uses production lines made up of multiple stations, where each station performs a specific task, and the product moves forward through the stations until it is complete. If one station is slower than the others, it will become a blocker. Product will start to queue up behind it, and downstream stations will become starved. So production lines only work well if they are planned to enable a consistent flow.

What’s more, you can only sell the product when it is completed, so we could describe the product as having no value until it is finished and with the customer.

It’s not just about manufacturing

The processes born out of manufacturing also work really well for other industries. I would suggest most things can be described like a production line, and as soon as you do that, a similar approach can be adopted to identify and fix the constraints.

Tech is an obvious one, and variations on DevOps have grown in popularity because of that. My sister in law works in a medical practice, and we’ve discussed the processes used in the administration side of it. As a result of our discussions she’s started to use Kanban boards to visualise the flow of work.

Back to my problem

So with the concept of production lines and constraints in mind, we jump back to my problem…

We are in the process of replacing loads of Oracle Linux 7 servers with either Oracle Linux 8 or Oracle Linux 9, depending on vendor support. The first three links in that production line are as follows.

  • VMs are provisioned.
  • Operating system customisations are run.
  • Database or app server is installed and configured.

We are not perfect, but we’ve got pretty good at this part of the production line. When we are finished, the systems have to be tested, and go through various processes to get them live and used by the business. Those parts of the production line that follow us are slow due to a number of factors. So our improvements in the production line have just made things harder for those steps that follow us. A simplified view of the Kanban board looks something like this.

The obvious thing to do here is focus on the constraints and start working on downstream links in the chain, to improve the overall flow. That’s where we hit organisation and culture barriers, so we are pretty much stumped…

Thoughts

I’m pretty happy with what we’ve done over the last few years. We’ve definitely improved several aspects of our systems because of automation, but at the same time I can’t help thinking we’ve achieved nothing because ultimately the work is not getting completed as fast as it’s started.

I wrote here about reframing the goal, and I have to do that as copium. Unfortunately copium only goes so far… 🙂

Cheers

Tim…

Continuous Improvement : It’s easy to get lazy…

I like to think that continuous improvement is always part of my process, but occasionally things crop up and remind me that I am just as lazy and set in my ways as everyone else. Here’s a little story to illustrate that…

The Issue

The main thing I’m supposed to be focussed on at work is server rebuilds. We are moving a bunch of stuff from Oracle Linux 7 (OL7) to Oracle Linux 9 (OL9), or OL8 if the vendor doesn’t support OL9 yet.

Anyone who follows my Vagrant builds knows I’ve got loads of scripts that build things, but because of our company organisation structure, there is only so much I can do without pinging requests to other teams, so full automation on-prem is not currently possible for me at work.

As a result of this limitation, there is a bunch of things I can do in Vagrant that I can’t do at work. The fact that work is semi-automated for builds has indirectly made me lazy in respect to continuously improving my build processes. In my head I guess I’ve been saying, “What’s the point if I can’t finish it?”

Making Progress

Due to the large number of rebuilds we are working through now, I’ve taken a step back and started again. I created a new build repo at work, and populated it with all the relevant stuff from our existing build processes, and some bits from my Vagrant builds. It’s still not fully automated, due to the company organisation issues mentioned before, but every step in the process is up to date with the best we can do, and the process is documented in the README.txt of the repo.

Along the way I noticed a bunch of things that were pretty crap in my original approach. It worked, but it wasn’t something to be proud of. 🙂

So What?

This just goes to illustrate the point I made in the title of this post. It’s easy to get lazy and let things start to slip, even when you know better. This is especially true if you have a “completionist” mentality like me. The thought of being able to finish something is quite compelling to me. Unfortunately, when I know I will not be able to complete something, I find it really hard to make the effort, because it all feels kind-of pointless. 🙂

Reframe The Goal

I guess this is really about reframing the goal. Rather than thinking that completion is when I have a fully automated process, it should be that I’ve got everything up to date and as far down the automation path as I can get at the moment.

The funny thing is I wrote about this in a previous post, but it seems I’m not always capable of taking my own advice. 🙂

Cheers

Tim…

UNIX/Linux Time Command : Record elapsed time

In a recent post I mentioned using a scratchpad to record everything I do. As part of that process I try to make regular use of the UNIX/Linux “time” command to record elapsed times of long running commands.

It’s really simple to use. All you do is put “time” in front of the command and it will display how long it takes to complete the command. In this example I do a sleep for 10 seconds and use the time command to report the elapsed time. 🙂

$ time sleep 10

real    0m10.002s
user    0m0.001s
sys     0m0.001s
$

Clearly that’s a silly example, but it gives you an idea of how this works.

If you get into the habit of using this with all long running processes, you can get accurate timings for steps. That way, when someone asks you how long something takes you can give them a real answer, rather than making something up and hoping for the best.

Just remember, timings can vary if the load on the system varies between runs. Even so it’s always nicer to have some real data to inform your decisions going forward. Especially when planning for something that will cause disruption in production. 🙂

Cheers

Tim…



Using a scratchpad…

Followers of the blog know I’m a big advocate for writing things down. The main reason I do this is because I want a record of everything I do.

I rarely type a command directly into the command line. I nearly always type it in a scratchpad first. Currently I have 67,250 lines in my work scratchpad and 12,309 lines in my personal scratchpad.

When I say scratchpad, I just mean a text file, which I edit using a text editor. Nothing fancy.

Why do I do this?

Inspiration

Most of my articles and blog posts start life as notes in my personal scratchpad. At work some of my scratchpad notes become more formal documentation, like knowledge base notes and how-to files in Git etc.

I know if I don’t make the notes as I go along, I will forget what I did, and struggle to write the documentation later.

If something makes it as far as being written up, it gets removed from my scratchpads, so what’s in there at the moment are notes that have not made the cut, so to speak. 🙂

One of the reasons I’ve been able to produce content for so many years is there is a constant stream of stuff added to my scratchpads. Of course, some of it is junk, but some of it is not.

If you are struggling with documentation or inspiration, I think taking this approach will really help.

Reflection

One of the things that I find really useful about taking notes is it allows me to look back and reflect on what I did to complete something. For example I might search through my scratchpad to see what happened over the lifetime of a server. I can see all tickets that were raised and what firewall rules and configuration changes were required. When I get a similar request this allows me to estimate the amount of work that needs to be done, and I can see what teams will be involved in the process.

I could search though our ticketing system for much of this information, but I find it a lot easier to keep a record of my actions in a scratchpad, then drill into the tickets if I need more info, which I rarely do.

Rewrites

Much like my articles, if I read back through some notes and they aren’t 100% clear, I often rewrite them. Maybe adding some more text, or a clearer example. This process may result in something graduating into being a separate document, but sometimes it just stays in the scratchpad forever.

Give it a go

If you don’t already do this, give it a go and see how you feel about it. Especially you content creators.

Cheers

Tim…

URGENT : Why you should {almost} never put URGENT in your message

Just a little note about something that rubs me up the wrong way.

I quite often get messages with the word URGENT in the subject or text. I scan through the content, and if it doesn’t seem truly urgent to me, I put it at the bottom of my list of things to do. Why?

You are not the central character in my life

When someone is communicating with me, they are thinking it’s a 1-to-1 interaction. What they forget is that I am working on many different things. As a result, for me it is a 1-to-many relationship.

Just because something is urgent to you, it doesn’t mean it takes priority over the other work I am doing. You don’t know what I’m doing, so you can’t possibly know how your issue sits in my list of priorities. Assuming your needs are more important than the needs of others is really rude.

This is even more annoying when it comes from someone outside of work. If you are not paying me, you have no business sending me an “urgent” request.

Your bad planning is not my emergency

In many cases these “urgent” issues could have been solved well in advance. It’s bad planning that has caused this issue, so I don’t see why it should have a negative impact on my life.

Sometimes there are genuine reasons for something to be classed as an emergency, like P1 incidents, but that’s not what I’m talking about.

There are some people that bounce from one emergency to the next. It soon becomes obvious that these people are just really bad at planning, and as a result are constantly in the weeds, and asking you to help drag them out.

Personal heroics don’t help the company long term

Occasionally you have to dig deep to get through a real emergency, but for the constant stream of self-inflicted emergencies, the only solution is to let things fail so people can see the root cause.

Personal heroics may feel good to you in the short term, but in the long term it is bad for your company and for you. The company needs to know what is failing and do something about it. Relying on a small number of people to pull them out of the weeds is not a long term strategy. Sooner or later this will stop working because the “heroes” will get annoyed and leave, or quiet quit.

What does urgent even mean to you?

I once got a message late on a Friday about an “urgent” issue. I felt sorry for the person in question, so I cancelled my plans, worked on the issue and sent them back the solution. I then got a reply saying, “Great, I’ll have a look at it on Monday”. Needless to say I lost my shit. That clearly was not an urgent issue.

I’m not alone

Over the years I’ve had this conversation many times, and I know I’m not the only person that gets annoyed by messages marked as urgent. I also know I’m not the only person that puts them to the bottom on my to-do list if they are not truly urgent.

Conclusion

As you can see, unnecessarily marking things as urgent is a bad idea, and likely to result in a longer resolution time, so next time you consider adding that little word into your message, just don’t!

Cheers

Tim…

PS. Rant over…