If you have looked around this site, or know me and have heard me discuss it, I have an honest to goodness Red Hat Enterprise Linux subscription. I am not talking about the rebuilt version called Rocky Linux, which is the successor to CentOS after Red Hat changed that distro from being just a rebuilt/relabeled version of the corresponding RHEL version to a stream version sitting somewhere behind Fedora and before RHEL.
Originally, back in 2000, I paid actual money for this. I talked to a Red Hat sales person, who referred me to a reseller with a discount, and I paid $427 for it for a year-long subscription. Not something I really liked to do, but RHEL is one of the major leaders for Linux Server OS implementations, which represents ~63% of the total OS usage on the Internet, with RHEL taking 43% of that 63%, Debian based systems (Debian/Ubuntu) taking 38% of that 63%, of which Ubuntu alone is about 34% of the 63%. Then SUSE/SLES (SUSE Linux Enterprise Server) is 11.2% of that 63%, with CentOS/AlmaLinux and others accounting for the rest. And of those, RHEL and SLES required paid support subscriptions, which is part of the reason why I paid for the subscription... If I found a bug in CentOS, I could try to reproduce it in RHEL, and get support on questions I had, both for myself and for Monkey Knife Fight where I worked. I could have had work pay for it, but given what I was making at the time, I didn't worry about it. After all, when I joined May of 2019, I actually got my boss Peter and our President/co-Founder Nic to sign off on buying me a new high-end Dell laptop which would be mine to keep.
A year later, while I could still afford the $427 or whatever it was to be that year, I found out about Red Hat's Developer program, which would get me not only my RHEL 9 subscription license, but more for free. I would be stupid to not take advantage of it, so when I needed to renew, I talked to the Red Hat sales rep, and when I had problems signing up, they helped me get signed up. And here is just a sample of what it gets me:
- Red Hat Enterprise Linux (8, 9 and 10)
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Red Hat Enterprise Linux AI
- Red Hat AMQ Clients
- Red Hat Application Interconnect
- Cert Manager support for Red Hat OpenShift
- Red Hat Container Native Virtualization
- Red Hat CodeReady Linux Builder
- Red Hat Discovery
- Fast Datapath for RHEL
- Red Hat JBoss Fuse Service Works
- and so much more.
And quite a few of those, I am not entirely sure what the details are, like what AMQ Clients are, or Application Interconnect. But I really need to learn about them. Ansible, it is something I am familiar with, and have long used it. I really should upgrade the server I run Ansible on to Ansible Automation Platform, because it has even more features I need to learn. But anyways...
One of the problems with subscriptions is that they expire, be it software or a magazine. And sometime in the past week or two, I suspect mine did. And the "Dang it Red Hat!!!" comes from the fact that it expired without the usual email telling me that I needed to update my subscription. I normally don't go around installing software that often, just firing off a dnf update from time to time to pick up the latest versions, and last time I did that, there were no updates. But the past couple of days, I have been trying to install a dockerized tuleap instance, only to be told that there was an error connecting to the database server. Only, the container which is provided for the Community Edition has some unique features. Besides the usual exporting both the HTTP and HTTP ports, it exports an SSH port, and on top that, uses a weird ENTRYPOINT in the Dockerfile, which actually handles the fact that unlike most containerized software, they have in fact written a program to spin up not just the usual single process, but multiple processes, to allow for both a web server and the SSH server. And normally, when a container is fired off with a docker-compose.json file like the instructions stated, every container runs, as I stated, a single process, so defines multiple containers to run multiple services. This is one reason why, if you look at my Heimdallr or Job Application Tracker projects, there is a container running nginx and a separate container running either the Gunicorn WSGI server for a Python application, or a php-fpm server, if it is PHP. And so, every time I tried to get into the container doing a docker compose run to and giving /bin/sh as the command to run, it would start doing the same thing it did when I started the container normally... It got SQL errors trying to connect to my "mysql" server. I use the quotes, because when I first installed this server, way back in mid-2023 (prior to that, I may have had a RHEL server for this virtual host... or not), I did so doing the ISL (Initial System Load) of the virtual machine using Cobbler and Koan to drive the Kickstart install so that I don't have to do all the things like pick software, define the disk layout, etc. It just does it, using some config files and script I wrote to figure things out. And then, I used Ansible to do the customizations, such as installing additional software (some time ago, I shifted from defining multiple profiles for machines which defined things like what software to install, to using Ansible to do that, and just do an absolute minimal install instead). So, I could not remember if I installed MySQL, Community Edition (which comes from Oracle) or MariaDB, which was created as a MySQL fork by Michael Widenius when Sun Microsystems acquired the company which he founded, where at the time he was Chief Technical Officer, But when Sun acquired it, it turned out to be a mistake, as Sun was then acquired by Oracle, and relicensed the software to be proprietary, since it was a competitor of their own Oracle DB. And it actually matters, because the two projects have diverged. They both were the same for MySQL 5.1, but had poor performance. And when Widenius and a swath of the other MySQL developers left to start MariaDB on the day of the Oracle acquisition, the divergence started. Anyways, these days, MySQL has 8.4LTS and 9.7 LTS (LTS meaning Long Term Support), while MariaDB jumped clear to 10x, with 10.6LTS, 10.11LTS, 11.4LTS, 11.8LTS, 12.2.2-GA (General Availability) being the current supported versions. And it matters, because as I type this, I seem to remember MariaDB changing the way clients connect and authenticate.
But, earlier (now "last night", since it is 3am), between downloading the source and doing some reading, I finally figured out the trick, and was finally able to figure out that it was not some bug with tuleap or my misconfiguring things in the environment setup for the container (like the variable names for specifying the login credentials to the database). Rather an doing a docker compose run to run /bin/sh (I tried all the combinations which made sense, to get the same result), I needed to do a slightly different command, which was:
docker compose -f tuleap0.yml run --entrypoint /bin/bash tuleapand after finding out that not only is the container is based off Rocky 9, but after copying a particular script out, turning on debugging, and injecting that back into the container as a part of a script which also tested installing mysql and doing some connection tests and such, and then running the command which tuleap-cfg, which the default entry-point, itself runs (which is what I edited) I finally get the following error:
Tuleap only support MySQL 8.0.x and 8.4.x. Found: version MariaDB Server - 10.5.29-MariaDBNot sure why I did not see that in the logs when looking at the messages from the container, but there it was... why I very likely have been getting a non-helpful SQL error trying to connect. And this is where Red Hat comes into the picture...
You see, one of the things I started doing was to backup the databases in my MariaDB server, and using Ansible to uninstall MariaDB and to instead install MySQL. And rather than do this manually, forget what/how I did things, and have to re-figure things again, I did what any good developer who is interested in minimal effort and repeatability does for something involving multiple tasks... you write a script, or in this case, update an already existing Ansible playbook to do it. And this is where I start finding out more interesting things.
You see, a day or two ago, I started messing with GNU Emacs 29, which has improved handling for YAML files, just like that tuleap0.yml file you might have noticed above which I had been editing, and thus wanted the new version. And I had seen something which didn't register at the time. All my files in /etc/yum.repos.d were essentially empty. All they contained was this comment:
#
# Certificate-Based Repositories
# Managed by (rhsm) subscription-manager
#
# *** This file is auto-generated. Changes made here will be over-written. ***
# *** Use "subscription-manager repo-override --help" if you wish to make changes. ***
#
# If this file is empty and this system is subscribed consider
# a "yum repolist" to refresh available repos
#When I had looked at them to see why I was having issues with emacs-29, it did not hit me... I was too wrapped up in things between trying to figure out tuleap and writing YAML files for installing the latest version of emacs, where I got an unsatisfied package error, and just wrote it off as a combination of two things together, those being:
- Since RHEL based systems use
dnfand notyum, and it talks about the file being auto-generated, maybednfwas getting its configurations from, and maybe, just maybe, this was a new change which I had not noticed. - I hardly ever mess with the files in that directory. I just had had Ansible download and copy one for the new repo for
emacs-29, and did not even look at them, until the unsatisfied package requirement. I just removed the virtual host from the short list of machines with the new version, and went back totuleap.
But today, while getting ready to install MySQL instead of MariaDB, I got some errors which didn't make sense, tried uninstalling the old emacs version, which went fine, then got an error trying to reinstall the same packages I had just installed, and got a message saying the package for emacs itself was not found, and that it could not satisfy the requirement by emacs-yaml-mode. This was a major "WTF!!??" moment. I occasionally do a dnf list to show the latest packages, and there it was in my yum-packages.lst file:
emacs.x86_64 1:27.2-18.el9 @rhel-9-for-x86_64-appstream-rpms
emacs-common.x86_64 1:27.2-18.el9 @rhel-9-for-x86_64-appstream-rpms
emacs-filesystem.noarch 1:27.2-18.el9 @rhel-9-for-x86_64-appstream-rpms
emacs-yaml-mode.noarch 0.0.16-3.el9 @epel
emacs-auctex.noarch 12.3-2.el9 rhel-9-for-x86_64-appstream-rpms
emacs-lucid.x86_64 1:27.2-18.el9 rhel-9-for-x86_64-appstream-rpms
emacs-nox.x86_64 1:27.2-18.el9 rhel-9-for-x86_64-appstream-rpms
pinentry-emacs.x86_64 1.1.1-8.el9 epel
poke-emacs.noarch 2.3-1.el9 epel But when I ran the dnf install emacs emacs-yaml-mode by hand (which does not end up with a hard to read red text on my terminal background), I still got the message. So, I try yum repolist and get almost nothing. This rated a "WTAF!!!???". First, I try enabling the rhel-9-for-x86_64-appstream-rpms repo, get an error, search Google, get told to do some things with subscription-manager, those fail, and I log into Red Hat's Hybrid Cloud Console, look at the machine, try actually installing and running their insights-client, try more Google searches and subscription-manager stuff, double checking things like time synchronization, unregistering and reregistering the machine, and more subscription-manager stuff to list what repos I have, to get nothing. And then, I see on my Subscriptions Inventory page Active: 0, Expiring soon: 0, Expired: 2, Future dated: 0. Yep... my subscription had expired, and I had not gotten an email reminder to renew it. I checked... nothing in my email, even in the SPAM filter, nada... zilch... crickets. Thankfully, all I had to do is to login to yet another Red Hat site I almost never visit, accept the agreement, and seconds later, Active: 2, and I can re-install emacs. I may actually try emacs-29 again... It comes from a special repository called Fedora Copr, which is a build server where leading edge version of packages like emacs are kept. Maybe, rather than having to add another repository (rather than hundreds of software packages all being in one repo, it is broken down into things like "just Emacs 29", "just the latest version of Python 3.15" and so on, associated with the user who created the package. So for example, for emacs-29, the repository is copr:copr.fedorainfracloud.org:mlampe:emacs-29. But, emacs-27 (which is 5yrs and 9mos old as of this writing, compared to emacs-29's almost exactly 3 years. And I really should see if emacs-30, which is less than a year old for newest minor release, has hit Copr yet...).
But first thing tomorrow, I will finish up the install of MySQL, restore the data from my backup, and see what tuleap does this time around. Because, while I can open issues for things like Heimdallr, Job Application Tracker, or my work on pylint issues on GitHub on their respective source repositories, I want to also track things broken down a bit more granular than just "here is a bug". Which is where Agile and tuleap come into play. tuleap is an ALM (Application Lifecycle Management) package like GitLab, JIRA, Mylyn, tuleap and others, come into play. And when I think of something new which will need addressed, I can just add a task to the taskboard of tuleap, when I can then break down into tasks like opening the bug, coding the fix, etc., if not even finer than that, all Agile-like.
But first... sleep, perchance to dream... after I reboot that RHEL 9 VM, which I just told to update the installed software on, which included the latest kernel. And my old, personal WordPress site can stay down for the 12 hours or so... its not like anybody reads it, and this site is on my PostgreSQL server.
Oh... and another reason for playbooks... tomorrow in the background, I am going to install PHP 8.5 and the latest PostgreSQL version, all by simply changing a couple of lines of variables in Ansible configuration files. Yet another reason I use it... a few minutes of making those changes, maybe a couple of custom tasks which I might not have written yet, and it does all the work for me, while I work on other things, like tuleap, pylint (to wrap up tests, after having gotten a design decision from the lead maintainer) and then Heimdallr, where I am going to try out a HTMX starter kit for Laravel 13, and get the basic things like the About page, containers and such straightened out. But now, to save this and reboot, then bed.
Oh, and a second thing... one of those things I am putting here as a reminder... In digging through the tuleap source, I noticed that they don't support running with SELinux in enforcing mode, only permissive mode. That to me is a pet peeve which signals a lazy developer/development team, because it represents a critical security hole. It is not that hard to create the files for software. At worst, it might take a couple of hours. And then, I will submit the changes in a security bug, and let them add them to the software.