Research path forward for our infrastructure machine

We seem to be stable on Rocky 8.5 now, so closing this.

I previously said:

RHEL 8.5 would be expected end of year, around the same time that CentOS stops being a thing. So 8.4/2105 may be the last of CentOS as we know it.

From the recent CentOS Board of Directors meeting minutes:

https://blog.centos.org/2021/07/centos-board-of-directors-minutes-2021-07-14/

• There will be no automatic migration to CentOS 8 Stream

• They will ship a CentOS 8 release corresponding to RHEL 8.5

• "There will be no updates published beyond 31 December 2021, other than, if required, the CentOS Linux 8.5 release as noted above." so they will ship one more version

• They will ship the "zero-day updates" to 8.5

• Then, everything gets shut down and archived January 31, 2022

• unless there's a critical bug in which case that will happen sooner, rather than fix it

To me, that implies that updating to 8.5 is essentially pointless, as there will be no available fixes for it, ever. I think that means we should still regard it as important to move to whatever we're going to be relying on going forward before we get to the end of the year. I'd think that moving 8.4 -> Rocky 8.4 is a well enough trodden path that we'd have high confidence of it working, but if Red Hat start throwing 8.5 in the mix things might get sketchy again.

Rocky Linux 8.4 is now out:

• DVDs for Intel and ARM

• Docker Hub container images for Intel and ARM

• migrate2rocky tool for CentOS and various other things -> Rocky

• generic qcow cloud image

• Google Cloud Platform images

• EC2 AMIs supposedly coming

Announcement: https://forums.rockylinux.org/t/rocky-linux-8-4-available-now/3015

Downloads: https://rockylinux.org/download/

Migration tool: https://github.com/rocky-linux/rocky-tools/tree/main/migrate2rocky

Agreed, deploying our main Jenkins node via Docker is probably easy enough if we want to do it. Such a container would be able to do the things we do just now without issue as it's all poking AWS to fire up nodes.

Whether it works quite as well to fire up Docker containers to run Jenkins jobs is a different question. I don't know how they do that; in the GitLab system I use, the hard work of driving container creation is handed off to explicitly designated runner containers.

I definitely would never use Jenkins – running within Docker or otherwise – to orchestrate other general-purpose containers. It's not what it is for, and there are far better tools than Jenkins to do that. I personally use Docker Swarm for orchestration, with the CLI and Portainer as admin interfaces. The cool kids use Kubernetes but we should absolutely not do that.

My use of Docker-in-Docker is (deliberately) restricted to things like automated multi-architecture Docker image builds. It's not unreliable in my experience, just hard to get your brain around and hard to configure as a result. The whole transition to Buildx may or may not make things less infuriating, but that's not really relevant for what we do.

It might be interesting to explore how easily we could Dockerize ... Jenkins

Running Jenkins via Docker should be straightforward.

I think the main concern is whether we would ever use Jenkins running via Docker to spin up Docker containers, such as Ian's IdP image. Last time I read the docs, DIND (Docker-in-Docker) sometimes fails in weird ways. But maybe it's better now.

But we would probably never do that (spin up Docker containers) on Jenkins main itself.