Once they have a shell they're free to do what they want. Local root exploits, etc are all fair game -- and there are plenty of 0-days out there. In fact, removing the compiler doesn't help; a lot of attackers and bots run a script to detect the system environment and auto-download what they need to move on. Remember: jails can still be broken out of, and there is some really neat work in the research space of attacking hypervisors...
There are numerous things you could remove from an OS install to make their lives a bit harder (dtrace, systemtap anyone? you can sniff passwords easily...), but once they have a shell and a directory with write access it's game over. You're better off leaving the tools there but having your auditing throw a serious fit when a compiler, etc is executed unexpectedly on a production box.
So yes, I think it's bunk to claim that removing a compiler will provide meaningful security benefit. In fact, I believe this has been suggested a few times to FreeBSD by users (don't install compilers, etc by default!) and was dismissed as ineffective for the same reasons.
Work harder to prevent the bad guys from getting that far. Keep your systems well patched. Have a thorough auditing and monitoring system in place. Use containerization and segregation everywhere possible. Limit scope of access. Some points you made apply here as well. Maybe once you've mastered all of those arts you can play with the obscurity angle.
edit: and L7 firewalls... I'm on the fence there. There has been some movement in that area by security researchers equating them to antivirus... they're only as good as their definitions, and a targeted attack will bypass it. They also seem to give a false sense of security and let people be lazy. Security in layers is important, though.
edit2: don't get me wrong, you should be hardening your production boxes (and dev... so the environments match...) but removing a compiler is not high on my list
My experience has been 80% + in Solaris and OVM for SPARC (also some AIX with LPARs) and I am not aware of any exploit that breaks the non-global zone security boundary to the control domains or other NGZs. I'm fairly sure I haven't seen any in FreeBSD either but I could be wrong here.
In Linux, I am aware of privilege escalation exploits involving symlinks and complied objects (out of the top of my head, might be old or inaccurate) but overall the thinking goes that having a compiler available opens up a few extra attack vectors while not having it installed doesn't break anything at all.
I published an entry level article on building a gold image with ubuntu where I try to map some best practices and my experiences here: https://blog.ukigumo.eu/anatomy-of-a-gold-image/ I never got around to publish the rest of the series but still plan to, so feedback is welcomed. Maybe I should point out that I'm not exactly a sysadmin though :-)
Regarding L7 firewalls, I have recommend that they are installed and _used_ in all environments I've worked in (banks, insurance companies, payment services providers, stock exchanges, etc) and I do agree with you that often the feeling is that if they are installed and set to defaults it's good enough which is worrying but that's not to say that they should be abandoned.
Security is all about risk mitigation and all we can do is not to make it too easy for our platforms to be broken into and data to be lost, and I think if you dismiss obscurity you might need to try out something like setting up a couple of honeypots with commonly used services in your network (one with default ports one without for instance) and keep some notes of your experience :-)
There are numerous things you could remove from an OS install to make their lives a bit harder (dtrace, systemtap anyone? you can sniff passwords easily...), but once they have a shell and a directory with write access it's game over. You're better off leaving the tools there but having your auditing throw a serious fit when a compiler, etc is executed unexpectedly on a production box.
So yes, I think it's bunk to claim that removing a compiler will provide meaningful security benefit. In fact, I believe this has been suggested a few times to FreeBSD by users (don't install compilers, etc by default!) and was dismissed as ineffective for the same reasons.
Work harder to prevent the bad guys from getting that far. Keep your systems well patched. Have a thorough auditing and monitoring system in place. Use containerization and segregation everywhere possible. Limit scope of access. Some points you made apply here as well. Maybe once you've mastered all of those arts you can play with the obscurity angle.
edit: and L7 firewalls... I'm on the fence there. There has been some movement in that area by security researchers equating them to antivirus... they're only as good as their definitions, and a targeted attack will bypass it. They also seem to give a false sense of security and let people be lazy. Security in layers is important, though.
edit2: don't get me wrong, you should be hardening your production boxes (and dev... so the environments match...) but removing a compiler is not high on my list