Centralized logging and observability is not a new concept. Decades ago, at CompuServe, we had over 1000 machines running BSD/OS, and the idea of visiting each one to check machine health was laughable at best. Indeed, even just pulling the logs back for analysis was just as bad. So on each machine, syslog had lines like the following in the syslog.conf file:
kern.crit;authpriv.* @mhaak
kern.crit;authpriv.* @mhaalThe actual lines were more than just that, but the gist of it was that we saw syslog records on those two machines (mhaak and mhaal) for every critical kernel error, as well as for other events, like someone using the `su` command, which we deemed essential to know about. Then, with a much more complex config file on those machines, we logged those events into separate files, to only be accessed by a very short list of individuals, or to be processed with special scripts we (mostly I) wrote.
Fast forward to today, where we have numerous applications out there for centralized processing of system logs. Some of these use processes to read and preprocess traditional log files on the various servers, some allow for receiving the messages used by the various revisions of the syslog protocol, and some allow for a combination of these. Splunk, Elastic, Grafana, Logstash, OpenObserve, the list goes on.
The reason I am writing about this is that I am currently in the process of implementing centralized logging and analysis for my home network. And while Splunk could no doubt do some of what I want in terms of analysis, my initial goals are a bit more specialized, and actually aiming for what could be considered to be Intrusion Detection and Prevention. But unlike a traditional system, I want a bit more intelligence to things, while running centrally. This is because while I could run something like Snort, I already have two very good sets of data. These are:
- The logs from my firewall, as it blocks/allows packets to enter into my network.
- The logs of the requests to my webserver.
For the first of these, I already have the data being digested into a SQL database, and can generate a very rough report which looks something like the following:
cnt | tm | source_ip
-------+------------------------+-------------------------------------
64903 | 2024-12-15 00:00:00-05 | 2001:470:e22e:a:eef4:bbff:fec2:fe78
52589 | 2024-12-16 00:00:00-05 | 2001:470:e22e:a:eef4:bbff:fec2:fe78
50170 | 2024-12-17 00:00:00-05 | 2001:470:e22e:a:eef4:bbff:fec2:fe78
30402 | 2024-12-14 00:00:00-05 | 2001:470:e22e:a:eef4:bbff:fec2:fe78
9843 | 2024-12-15 00:00:00-05 | 79.110.62.75
9756 | 2024-12-16 00:00:00-05 | 79.110.62.75
8783 | 2024-12-15 00:00:00-05 | 104.156.155.4
7999 | 2024-12-17 00:00:00-05 | 79.110.62.75
7974 | 2024-12-15 00:00:00-05 | 194.180.49.216
6952 | 2024-12-16 00:00:00-05 | 194.180.49.218
6864 | 2024-12-15 00:00:00-05 | 194.180.49.218
6838 | 2024-12-16 00:00:00-05 | 194.180.49.217
6714 | 2024-12-15 00:00:00-05 | 194.180.49.217
5588 | 2024-12-17 00:00:00-05 | 194.180.49.217
5492 | 2024-12-17 00:00:00-05 | 194.180.49.218
5448 | 2024-12-16 00:00:00-05 | 194.180.49.216
4442 | 2024-12-14 00:00:00-05 | 79.110.62.75
3604 | 2024-12-15 00:00:00-05 | 194.169.172.100
3337 | 2024-12-14 00:00:00-05 | 194.180.49.216
3111 | 2024-12-14 00:00:00-05 | 194.180.49.218
2908 | 2024-12-14 00:00:00-05 | 194.180.49.217
...But this is only a first pass report. First, from this, I don't know which of these lines might represent packets trying to connect from networks I am already blocking, vs. connection attempts to services I block vs. say trying to access a webpage such as this. Secondly, this is on a per-IP basis. To completely know of a potential threat, I need to group addresses together, such as all those which start with `194.180.49.`. In that case, I need to do an additional look at all attempts from the entire subnet. So there is quite a bit here, but it is in need of some in-depth processing, combined with data from sources such as the WhoIs database, because in the case of certain countries, I just want to go ahead and block. Then there is the fact that I want to escalate hosts toward being blocked by those which are attempting to connect to services I don't expose. And, to complicate matters, the amount of data is on the order of megabytes of dense data per day.
For the second, there are similar challenges, but in ways, the task is simpler. First, we know a bit more about what the connection is attempting. For example, for my sites, there should be almost POST requests to my server, and they should be very constrained on where they occur. Or, attempts to run PHP or get other files should be straight out, and increases the likelihood that the host should be blocked. The one main gotcha here is that when storing these log records to the database, care will need to be taken to make sure I am not opening myself up to other attack vectors such as SQL injection, since the URLs themselves represent data from the end user, be they white-hat or black-hat.
My intent is to develop both of these under the umbrella of Heimdallr, and to actually have it automatically add hosts/networks to the list of those being blocked. So, keep an eye out, as I am sure to be writing more about this topic in the coming days and weeks.