In my previous article, I talked about why I am interested in Tuleap. But in this one, I am going to talk about my impressions of the software, in trying to set it up and get it running. And while not quite an article warranting a Core Dump category tag, I did realize that I needed a new one, as I have done a number of similar posts. And that new category is for things where I look at things as a possible sign of poor designs, laziness (or a company skimping), or general bad practices. But, as I say in the category description, it may just be a case of not knowing better, or postponing certain work in favor of other work. But then, there are some things which should never be postponed. And as I said, here, I am just talking about some halfway deep first looks I have had to take to get to where I am right now, which still does not have Tuleap up and running, even in a docker container.
For a quick summary, here are my issues (in rough order of priority):
My first item is a biggie, and I would not have noticed it if I hadn't been digging through the source to figure out why the container would not properly start, but instead just sat there with the logs indicating an error connecting to the database server. And this one intersects with security, so it should be concerning, and in this case, ignorance is not an excuse, nor is postponing it to a later release. And this is that Tuleap requires switching from "enforcing" mode, where the application gets errors when it tries to do something it is not permitted to do, to "permissive" mode, where it is just logged. Outside of early development, which includes developing what is called the "Policy Module", you should never run a machine which has SELinux in permissive mode!!! Did I make that clear enough? These days, if you don't know SELinux, learn it, and address it from the start. This is as important, if not more important than sanitizing input you get from a user to avoid things like buffer overflows or SQL injection. Yes, I realize that Tuleap in most circumstances should be in an environment where it is isolated from the Internet, but then, what about things like Open Source projects, or situations where work is contracted out, but work is still tracked in part using Tuleap? Indeed, there is tuleap.net, which is owned by Enalean, the company behind Tuleap. And while today, 20-30% of the security breaches are by insiders (down from over 80% years ago), even systems well inside multiple layers of firewalls are not safe.
In digging through the code, trying to figure out where to debug why it was not starting, I found code which specifically checks to see if SELinux is in permissive mode, or if it is in enforcing mode, and if the latter, it errors out, saying that only the former is permitted. Why is this a big deal? It is, because SELinux is just like a second firewall, this one being between any given process and the rest of the system. Want to connect to a database, you need a permission for that. Want to read certain files, you need to have another permission for that, and if you want to write them, you need another, and if you want to access certain network ports, there are permissions for that too. But if you try to do so without permission, such as accessing critical system files, or files for a different service (such as say the DNS server), not only is it a no-can-do, but it actually gets logged to a file which has its own special access controls. It goes way beyond the original read/write/execute for the owner, members of the same group, and the rest of the world to the point where what SELinux calls a "file context" can be applied to a specific file, and even can be restricted down to specific syscall (the functions behind things like reading or writing files, creating child processes, etc.), so that a process and its children can only append to a log file, but not read it or overwrite it, much less execute it. This is possible because every file, directory, device and network port is tagged with a label called a context. And if in a really sensitive environment, there is even what is called Multi-Level Security (MLS). So if a black-hat (I won't glorify them with the name "hacker", which was appropriated from its original meaning, and is like a bank robber calling themselves a "cop" or "police officer") somehow compromises a program, such as a web server, they are confined in their little world, since even if they can somehow create a sub-process, that sub-process is bound by the same restrictions, because even to run programs in their own security domains, you have to have certain accesses granted.
Now some of you might be saying "What about if I run it in a container? Isn't that supposed to grant a level of security isolation?" The problem is, containers are not a strong security boundary. Interestingly enough, if you have a container running on your system, you can actually see the processes running on the container by running ps auxww, and grepping for them. They are isolated to a degree using things called "namespaces" and "control groups", but a determined black-hat can get around these. Think of it as being like relying on a locked storm door to keep burglars out of your home. For the opportunist, it works, but for the serious burglar, in the middle of the night on a weekend while you are out of town...it is worthless.
To give you an idea of how important I believe SELinux is, and why I always have it in enforcing mode, way back when, when I was working on a project at TeleTracking, we decided to use Zend Server and Zend Framework. And I installed it on a machine, only to find that SELinux was doing its job, and stopping that software from doing a whole list of things...including to a certain degree, even being installed. But, rather than just switching SELinux to permissive mode, I instead did the install over and over, addressing each and every error, and then ran a test site, addressing even more errors. You can read the details here, and another here, (with more to come on the SELinux category page) but much of it involves reading the log file /var/log/audit/audit.log and using tools like ausearch and sealert, combined with other tools like ls -Z, semanage and restorecon. And to begin with, you might temporarily switch SELinux from enforcing to permissive using the command setenforce 0, followed by getenforce to tell you what mode it is in, and later followed by setenforce 1 to switch it back to enforcing. And obviously, you don't do this on a production machine, or even on a permanent development machine, but preferably on a machine which you will be re-installing afterwards. This is one reason why I love virtual machines such as those running on KVM with libvirt. You can quickly take a snapshot of the system in a powered off state, and later, easily go back to that same exact state by rolling back to that snapshot.
The second on my list is the documentation for configuring the base system, especially at the initial startup. In reading the Tuleap Installation Guide, I have found some halfway significant holes. For example, what are all of the environment variables which are needed during the initial installation, and what configuration variables continue to reside on disk, and where, such as database credentials. Every web application must get the credentials for the database from somewhere. Traditionally, sites like WordPress, this was the file wp-config.php at the top of the directory tree for the site. For Drupal, this is by default in locations such as web/sites/default/settings.php, though many of us add a line to that file to make that file so that we can check it into source code control, and separate out the security sensitive information into a file such as web/sites/default/settings.local.php, which the former loads. That is a great way to be able to swap out database credentials and host names from a file which contains the other, non-sensitive information, and allow for no changes between a development or test environment or production. And for Python Django, Laravel and other frameworks, there are similar files I won't go into, but many of those rely on a .env file in the root directory of the tree for the site, sometimes directly, and sometimes by having another configuration file reading it. And then, it is common for dockerized applications to have this file supplied by the host, or in the case of Docker Swarm, Amazon and others to have them injected either as environment variables or from a secure secrets interface so that the file does not exist on the disk with cleartext credentials.
But Tuleap gives an example of it being in the docker-compose.yml file. And it spreads this information throughout the Installation Guide, and never quite makes it clear where the critical files to protect are, or what the actual settings which might be in there are, and what is needed after the install and what is not. And, to make matters worse, in initialization process, it writes many of these settings to files on the disk, but never seems to say where these files are so that you can protect them (much less change them in the future, such as say if a database credential was compromised and needs changed), and the path also seems to be hardcoded.
Now I will on first glance grant that writing the information to files is no different than supplying a .env file, but such a file really should not be writable by the application, and you definitely want to restrict access to this file, at least by making the parent directory owned by a separate user which is used to run the application, with a permission of 0400. But an even better way to do this is to carefully inject the settings into the environment, such as in the VirtualHost definition if it is run by an Apache web server, or by techniques to do that for containers. But, as I say... carefully... because containers can be inspected to review information, someone who has gained access can copy files out of containers, or do other things to compromise security. And security by obscurity is the same as no security at all. A black-hat can always to tricks like dump the strings from a compiled executable, or read the source if it is written using a language such as Python or PHP. And, if you have access and the program is not smart, it might even show up as a open file under lsof. But this is only part of the problem were databases are concerned.
I hit upon databases and configuration files. Now, for databases, well... database and database server themselves. First, they are really picky, far pickier than I have seen any software package be. Indeed, other packages/frameworks such as Laravel and Drupal not only support MySQL, but MariaDB and even more radically different PostgreSQL, MS SQL server and others, because they rely on PHP's "PHP Data Objects" to abstract the database. But not only is Tuleap demanding strictly MySQL, and not the MariaDB variant (which forked from MySQL back in the MySQL 5 days, but has diverged in a few areas), but they have very specifically locked things down to just two versions. And when you look at things, including one line in their database setup page, it is really only one version, that being MySQL 8.4, which was a LTS release made 2 years ago (and has 6 more years of support to go). Technically, the code still might let you get away with 8.0LTS, but that was released 8 years ago, and is no longer supported as of this April. I am actually surprised that they don't at least allow the 9.x releases, so that development, test and small "community" sites could at least track the latest releases, rather than to have to go through a huge development effort to make that change, but I have seen no signs of that going on. I am, however, going to keep an eye on things, to see if they don't pick up support for 9.7LTS, which was released over a month ago.
Then there are the options which Tuleap insists be removed from the server's sql_mode variable. They are:
- STRICT_TRANS_TABLES - meaning that Tuleap relies on older data -handling rules than is current today.
- NO_ZERO_IN_DATE - Meaning that Tuleap wants to be able to store invalid dates such as
2026-00-01or2026-05-00, contrary to the SQL standard, and which screws up date calculations. - NO_ZERO_DATE - Means that Tuleap wants to be able to store a date of
0000-00-00as dummy placeholders, which again screws up data integrity, screws up date calculations and is contrary to the SQL standard. - ERROR_FOR_DIVISION_BY_ZERO - It wants to be able to submit queries which result in a division by zero, which is undefined by the rules of maths, and would otherwise cause problems.
- NO_ENGINE_SUBSTITUTION - Which means that Tuleap relies on MySQL to use an old "relaxed" fallback behaviour, where it substitutes a default storage engine instead of reporting an error.
I have to say, it does not get better from there. First, they say "The database must be dedicated to Tuleap.". I take that to mean that there are no other databases on that database server. And this is a bit over the top. Many places throw heavy iron (well, silicon) at a machine dedicated to act as a database server, with different applications all having their databases on the same server, rather than having a bunch of small servers all over. And then, they demand very specific server settings, which make me wonder if the code is dated, and they are relying on the database to help them enforce data integrity, or are they just being picky. Yes, a number of these settings are a part of long-term trends in SQL, but again, it seems to indicate to me that this application is kinda like a kid which does not play well with others. And, they also talk about not having other applications running on the same server. Again, I know places where right after they throw heavy iron at a server for the database, they follow it up with a similar machine to run multiple applications as virtual hosts. Maybe they do it as containers with a main server process doing proxying, maybe they give the machine multiple host names and/or ports, or maybe use name based virtual machines. The thinking there is, when one application is under heavy load, other applications might be sitting idle.
Then there is the fact that Tuleap requires administrator access to the database, rather than just saying "Create a database and a user which can create the tables and run queries for us, and tell us what they are", like practically every other app out there. But wait, this seems to get even better!
In addition to telling Tuleap the credentials to an admin user are, you give it the name of a database, and the credentials for the user to normally use, as well as a password for the admin user. In my last test-install run, I found it not only used this access to create a user different from the one I told it to use. And it even appears (as I write this) to store this in clear text in the root home directory of the container, rather than in the other location. Talking about the environment variables for a docker container on this page, saying that the default for one is "disabled", then ignoring what was given and hitting another error because it is trying to setup SSL is in my book heinous at best. As for the database credentials it stores in /etc/tuleap/conf/database.inc, it seems that that one at least may be encrypted, even if they do provide a file named encryption_secret.key in the database, and not in a location where persistent data is documented to supposedly occur. If I confirm some of these facts, right along with opening a security bug for the SELinux issue, I will be writing several others.
I have not looked into this one in detail to see how it is implemented yet, but even when installed in a non-containerized environment, they seem to indicate that they will be running the SSH server, not the OS system processes. And this strikes me as an area of concern. First, if they have implemented their own, they are being downright stupid. That is like writing your own encryption algorithm and thinking it is going to be as strong as the standard ones out there (it won't be... often, it is far less secure). But, if they are controlling what normally comes with Linux instead of letting the OS handle it, why? And what happens if the application fails? Are you suddenly locked out of the system. Part of what had me digging for how to get into the container to debug things in this article was the tricks they pull to run not only what appears to be a home-rolled web server, but SSH server as well. And hand rolling your own servers is not a smart thing to do, because you often end up with security holes and performance problems. I will have to dig into this one. And, I have to wonder, why include an SSH server in a dockerized application in the first place. Most places use plenty of containers which don't provide such access, and rely on docker exec -it ... /bin/sh when they need to "log in" to a container... it is part of their security. But adding SSH, while nice on the face of things, actually increases the attack surface, and as documented, actually conflicts with SSH access to the host server, since it says to map port 22 of the container to port 22 of the host. Again, like in databases, companies don't drop a machine out there to run just one container, they tend to throw heavy silicon at the problem, maybe using some rack-sized file server from NetApp, VDURA (formerly Panasas), Dell EMC, HP and others, often using Fibre Channel, iSCSI or InfiniBand.
This one is going to take some more digging, but a real quick check (less than 30 seconds of looking at the unsorted output from rpm -q -a) showed me a number of unnecessary software packages installed on the container, such as rpm-build-libs and rpm-sign-libs (we shouldn't be producing software packages and signing them), capstone (a binary disassembly framework...WHY??), cracklib (a password cracking utility), libijs (for printing), ncurses (for terminal support, on a container which is supposed to host a web-based application), both Apache and nginx, X11 related packages and more. I will grant gpg and things related to email... maybe they want to send signed emails. But the rest? All it does is increase the attack surface, and if nothing else, causes the container image to bloat. Is it just that the Rocky 9 container they use for their base is that polluted, or did they install all that when ultimately producing their image? I kinda doubt the former, but will, for the moment, give them a tiny bit of a pass until I confirm one or the other.
That's it for now. I am going to fire off another round of trying to get things installed, and hopefully get past where it appears to be trying to setup SSL, in spite of it being implied that it was off by default... all while working on other things.