As i see it, the term “firewall” was originally the neat name for an overall security concept for your systems privacy/integrity/security. Thus physical security is (or can be) as well part of a firewall concept as maybe training of users. The keys of your server rooms door could be part of that concept too.
In general you only “need” to secure something that actually is there, you won’t build a safe into the wall and hide it with an old painting without something to put in it or - could be part of the concept - an alarmsensor that triggers when that old painting is moved, thus creating sort of a honeypot.
if and what types of security you want is up to you (so don’t blame others if you made bad decisions).
but as a general rule out of practice i would say it is wise to always have two layers of defence. and always try to prepare for one “error” at a time and try to solve it quickly then.
example: if you want an rsync server on an internet facing machine to only be accessible for some subnets, i would suggest you add iptables rules as tight as possible and also configure the service to reject access from all other than the wanted addresses. also consider monitoring both, maybe using two different approaches: monitor the config to be as defined as well as setup an access-check from one of the unwanted, excluded addresses that fires an alarm when access becomes possible.
this would not only prevent those unwanted access from happening but also prevent accidental opening or breaking of config from happen unnoticed.
here the same, if you want monitoring is also up to you and your concept of security, as is with redundancy.
In general i would suggest to setup an ip filtering “firewall” if you have ip forwarding activated for some reason. a rather tight filtering would maybe only allow what you really need, while DROPping all other requests, but sometimes icmp comes in handy, so maybe you want ping or MTU discovery to actually work. always depends on what you have and how strong you want to protect it from what with what effort. a generic ip filter to only allow outgoing connections on a single workstation may be a good idea as second layer of “defence” in case your router has hidden vendor backdoors that either the vendor sold or someone else simply discovered. Disallowing all that might-be-usable-for-some-users-default-on-protocols like avahi & co in some distros would probably help a bit then.
so there is no generic fault-proof rule of thumb…
to number 5.: what sort of “not trusting” the software? might, has or “will” have: a. security flaws in code b. insecurity by design c. backdoors by gov, vendor or distributor d. spy functionality e. annoying ads as soon as it has internet connection f. all of the above (now guess the likely vendors for this one)
for c d and e one might also want to filter some outgoing connection…
one could also use an ip filtering firewall to keep logs small by disallowing those who obviously have intentions you dislike (fail2ban i.e.)
so maybe create a concept first and ask how to achieve the desired precautions then. or just start with your idea of the firewall and dig into some of the appearing rabbit holes afterwards ;-)