Yesterday evening I went to the Docker Birmingham meetup, sponsored by Black Cat Technology Solutions.
This event was a single longer session by Matt Todd called “Make Data Science Great Again (Part 2)” I didn’t see part 1, but he gave a summary, so that wasn’t too much of a problem.
Matt started off by introducing the scientific method and discussed reducing variables when testing, so you know the impact of a change. The suggestion being that development and data science should be the same. What better way to reduce variables than to package up a data science lab to make sure everyone is working on the same thing, so there is no/less variability between researchers, and they can focus on their work, not piecing together the kit.
He then went on to discuss Cloud Native Application Bundles (CNAB), and how they can be used to more reliably package multi-container applications, guaranteeing dependencies to a level greater than that possible by using Docker Compose alone. As an example he demoed his Digital Scientists Lab, which is a CNAB bundle containing a bunch of common kit used by data scientists (Jupyter, Spark, Flink, Kafka, RabbitMQ, Nifi, Elastic Stack etc.), which he could in theory give to several people to run experiments. It’s just his playground, but it gives you an idea of what’s possible. Using Nifi to link stuff together looked cool!
I started to make a few mental connections when he discussed the relationship to experimental data, because I look after the infrastructure for a research publishing system, and being able to keep not only the published research, but also the experimental data and potentially a way to reproduce the research findings is pretty important. It keeps those academics honest, if you know what I mean. It’s funny how just a few links to something you know a little bit about, and other stuff starts falling into place.
It was all a bit mind-blowing, but in a good way. I’m still only scratching the surface of this stuff, but it’s really good to see what else is going on in this space. I’ve added a few more things to my list of things to play around with.
It’s a couple of months until the next event, but there’s a CNCF event next month, so watch out for that!
Thanks to the folks at Black Cat Technology Solutions for sponsoring and organising the event, and to the Matt Todd for doing a great session. See you soon!
Cheers
Tim…
