Icinga2 works reasonably well for us. It is easy to write new checks as small shell scripts (or any other binary that can print and set and exit status code).
To be fair USB sticks and SD cards seem to fail when you stare at them a bit too intensely. I think it has been at least a decade since I bought a USB stick for OS installations that lasted for more than three installs (each a few months apart at least since the need does not arise that often).
You people don’t seem to grasp that I am already not running any commands on the server as root that do not require root. This is all about administrative tasks.
The vast majority of commands when debugging actual issues on the system or performing administrative tasks do require root. Out of the others some give you incomplete results when called as a regular user and 90% of the rest shouldn’t be run on the server in the first place if you can avoid it but directly on your client computer (e.g. looking up documentation).
I was aware of the login UID for auditd logging as a difference but as you say, that is only really helpful if the logs are shipped somewhere else or tampering with them is otherwise prevented for admin users. It is not quite the same but the auth.log entries sshd produces on login also contain the key fingerprint used to login these days so on a more limited scale you can at least tell who logged in when from those (or whose key but that is no different than whose account for the sudo approach).
you should consider doing it right from the start.
Do you have any advice on how to use the sudo approach without having a huge slow down in every automated process that requires ssh user@host calls for manual password entry? I am aware of Ansible but I am honestly very sceptical of Python tools since they tend to break easily and often from my past experiences and I would like to avoid using additional ones for critical tasks. Plus Ansible in particular seemed to be very late with their Python 3 transition, as I recall I uninstalled it when it was one of the last tools left that did not work with Python 3.
I am asking why it is considered to be more secure for the use case where you aren’t limiting access to a few commands because it is access meant for all kinds of admin tasks, not just one specific one (as in access for the people who need to fix unexpected problems among other things).
FreeIPA and your password is the same on every machine: yours.
Any network based system like that sucks when you need to fix a system that has some severe issue (network, DNS, disk,…) which is exactly when root access is the most important.