04-17-2019, 08:26 PM
Linux Server Hardening Using Idempotency with Ansible: Part 3
<div style="margin: 5px 5% 10px 5%;"><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/04/linux-server-hardening-using-idempotency-with-ansible-part-3.jpg" width="1280" height="720" title="" alt="" /></div><div><div><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/04/linux-server-hardening-using-idempotency-with-ansible-part-3.jpg" class="ff-og-image-inserted"></div>
<p>In the previous articles, we introduced idempotency as a way to approach your server’s security posture and looked at some specific Ansible examples, including the kernel, system accounts, and IPtables. In this final article of the series, we’ll look at a few more server-hardening examples and talk a little more about how the idempotency playbook might be used.</p>
<h4><strong>Time</strong></h4>
<p>Due to its reduced functionality, and therefore attack surface, the preference amongst a number of OSs has been to introduce “chronyd” over “ntpd”. If you’re new to “chrony” then fret not. It’s still using the NTP (Network Time Protocol) that we all know and love but in a more secure fashion.</p>
<p>The first thing I do with Ansible within the “chrony.conf” file is alter the “bind address” and if my memory serves there’s also a “command port” option. These config options allow Chrony to only listen on the localhost. In other words you are still syncing as usual with other upstream time servers (just as NTP does) but no remote servers can query your time services; only your local machine has access.</p>
<p>There’s more information on the “bindcmdaddress 127.0.0.1” and “cmdport 0” on this Chrony page (<a href="https://chrony.tuxfamily.org/faq.html">https://chrony.tuxfamily.org/faq.html</a>) under “2.5. How can I make chronyd more secure?” which you should read for clarity. This premise behind the comment on that page is a good idea: “you can disable the internet command sockets completely by adding cmdport 0 to the configuration file”.</p>
<p>Additionally I would also focus on securing the file permissions for Chrony and insist that the service starts as expected just like the syslog config above. Otherwise make sure that your time sources are sane, have a degree of redundancy with multiple sources set up and then copy the whole config file over using Ansible.</p>
<h4><strong>Logging</strong></h4>
<p>You can clearly affect the level of detail included in the logs from a number pieces of software on a server. Thinking back to what we’ve looked at in relation to syslog already you can also tweak that application’s config using Ansible to your needs and then use the example Ansible above in addition.</p>
<h4><strong>PAM</strong></h4>
<p>Apparently PAM (Pluggable Authentication Modules) has been a part of Linux since 1997. It is undeniably useful (a common use is that you can force SSH to use it for password logins, as per the SSH YAML file above). It is extensible, sophisticated and can perform useful functions such as preventing brute force attacks on password logins using a clever rate limiting system. The syntax varies a little between OSes but if you have the time then getting PAM working well (even if you’re only using SSH keys and not passwords for your logins) is a worthwhile effort. Attackers like their own users on a system with lots of usernames, something innocuous such as “webadmin” or similar might be easy to miss on a server, and PAM can help you out in this respect.</p>
<h4><strong>Auditd</strong></h4>
<p>We’ve looked at logging a little already but what about capturing every “system call” that a kernel makes. The Linux kernel is a super-busy component of any system and logging almost every single thing that a system does is an excellent way of providing post-event forensics. This article will hopefully shed some light on where to begin: <a href="http://www.admin-magazine.com/Archive/2018/43/Auditing-Docker-Containers-in-a-DevOps-Environment">http://www.admin-magazine.com/Archive/2018/43/Auditing-Docker-Containers-in-a-DevOps-Environment</a>. Note the comments in that article about performance, there’s little point in paying extra for compute and disk IO resource because you’ve misconfigured your logging so spend some time getting it correct would be my advice. </p>
<p>For concerns over disk space I will usually change a few lines in the file “/etc/audit/auditd.conf” in order to prevent there firstly being too many log files created and secondly logs that grow very large without being rotated. This is also on the proviso that logs are being ingested upstream via another mechanism too. Clearly the files permissions and the service starting are also the basics you need to cover here too. Generally file permissions for auditd are tight as it’s a “root” oriented service so there’s less changes needed here generally.</p>
<h4><strong>Filesystems</strong></h4>
<p>With a little reading you can discover which filesystems that are made available to your OS by default. You should disable these (at the “modprode.d” file level) with Ansible to prevent weird and wonderful things being attached unwittingly to your servers. You are reducing the attack surface with this approach. The Ansible might look something like this below for example.</p>
<pre>
name: Make sure filesystems which are not needed are forced as off lineinfile: dest="/etcmodprobe.d/harden.conf" line='install squashfs /bin/true' state=present</pre>
<h4><strong>SELinux</strong></h4>
<p>The old, but sometimes avoided due to complexity, security favourite, SELinux, should be set to “enforcing” mode. Or, at the every least, set to log sensibly using “permissive” mode. Permissive mode will at least fill your auditd logs up with any correct rule matches nicely. In terms of what Ansible looks like it’s simple and is along these lines:</p>
<pre>
name: Configure SElinux to be running in permissive mode replace: path=”/etc/selinux/config” regexp='SELINUX=disabled' replace='SELINUX=permissive'</pre>
<h4><strong>Packages</strong></h4>
<p>Needless to say the compliance hardening playbook is also a good place to upgrade all the packages (with some selective exclusions) on the system. Pay attention to the section relating to reboots and idempotency in a moment however. With other mechanisms in place you might not want to update packages here but instead as per the Automation Documents article mentioned in a moment.</p>
<h3><strong>Idempotency</strong></h3>
<p>Now we’ve run through some of the aspects you would want to look at when hardening on a server, let’s think a little more about how the playbook might be used.</p>
<p>When it comes to cloud platforms most of my professional work has been on AWS and therefore, more often than not, a fresh AMI is launched and then a playbook is run over the top of it. There’s a mountain of detail in one way of doing that in this article (<a href="http://www.admin-magazine.com/Archive/2018/45/AWS-Automation-Documents">http://www.admin-magazine.com/Archive/2018/45/AWS-Automation-Documents</a>) which you may be pleased to discover accommodates a mechanism to spawn a script or playbook. </p>
<p>It is important to note, when it comes to idempotency, that it may take a little more effort initially to get your head around the logic involved in being able to re-run Ansible repeatedly without disturbing the required status quo of your server estate.</p>
<p>One thing to be absolutely certain of however (barring rare edge cases) is that after you apply your hardening for the very first time, on a new AMI or server build, you will require a reboot. This is an important element due to a number of system facets not being altered correctly without a reboot. These include applying kernel changes so alterations become live, writing auditd rules as immutable config and also starting or stopping services to improve the security posture.</p>
<p>Note though that you’re probably not going to want to execute all plays in a playbook every twenty or thirty minutes, such as updating all packages and stopping and restarting key customer-facing services. As a result you should factor the logic into your Ansible so that some tasks only run once initially and then maybe write a “completed” placeholder file to the filesystem afterwards for referencing. There’s a million different ways of achieving a status checker.</p>
<p>The nice thing about Ansible is that the logic for rerunning playbooks is implicit and unlike shell scripts which for this type of task can be arduous to code the logic into. Sometimes, such as updating the GRUB bootloader for example, trying to guess the many permutations of a system change can be painful.</p>
<h3><strong>Bedtime Reading</strong></h3>
<p>I still think that you can’t beat trial and error when it comes to computing. Experience is valued for good reason. </p>
<p>Be warned that you’ll find contradictory advice sometimes from the vast array of online resources in this area. Advice differs probably because of the different use cases. The only way to harden the varying flavours of OS to my mind is via a bespoke approach. This is thanks to the environments that servers are used within and the requirements of the security framework or standard that an organisation needs to meet.</p>
<p>For OS hardening details you can check with resources such as the NSA (<a href="https://www.nsa.gov/">https://www.nsa.gov</a>), the Cloud Security Alliance (<a href="https://cloudsecurityalliance.org/working-groups/security-guidance/#_overview">https://cloudsecurityalliance.org/working-groups/security-guidance/#_overview</a>), proprietary training organisations such as GIAC (<a href="https://www.giac.org/">https://www.giac.org</a>) who offer resources (<a href="https://www.giac.org/paper/gcux/97/red-hat-linux-71-installation-hardening-checklist/102167">https://www.giac.org/paper/gcux/97/red-hat-linux-71-installation-hardening-checklist/102167</a>), the diverse CIS Benchmarks (<a href="https://www.cisecurity.org/">https://www.cisecurity.org</a>) for industry consensus-based benchmarking, the SANS Institute (<a href="https://uk.sans.org/score/checklists">https://uk.sans.org/score/checklists</a>), NIST’s Computer Security Research (<a href="https://csrc.nist.gov/">https://csrc.nist.gov</a>) and of course print media too.</p>
<h3><strong>Conclusion</strong></h3>
<p>Hopefully, you can see how powerful an idempotent server infrastructure is and are tempted to try it for yourself.</p>
<p>The ever-present threat of APT (Advanced Persistent Threat) attacks on infrastructure, where a successful attacker will sit silently monitoring events and then when it’s opportune infiltrate deeper into an estate, makes this type of configuration highly valuable.</p>
<p>The amount of detail that goes into the tests and configuration changes is key to the value that such an approach will bring to an organisation. Like the tests in a CI/CD pipeline they’re only as ever as good as their coverage.</p>
<p>Chris Binnie’s latest book, Linux Server Security: Hack and Defend, shows you how to make your servers invisible and perform a variety of attacks. You can find out more about DevSecOps, containers and Linux security on his website: <a href="https://www.devsecops.cc/">https://www.devsecops.cc</a></p>
</div>
<div style="margin: 5px 5% 10px 5%;"><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/04/linux-server-hardening-using-idempotency-with-ansible-part-3.jpg" width="1280" height="720" title="" alt="" /></div><div><div><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/04/linux-server-hardening-using-idempotency-with-ansible-part-3.jpg" class="ff-og-image-inserted"></div>
<p>In the previous articles, we introduced idempotency as a way to approach your server’s security posture and looked at some specific Ansible examples, including the kernel, system accounts, and IPtables. In this final article of the series, we’ll look at a few more server-hardening examples and talk a little more about how the idempotency playbook might be used.</p>
<h4><strong>Time</strong></h4>
<p>Due to its reduced functionality, and therefore attack surface, the preference amongst a number of OSs has been to introduce “chronyd” over “ntpd”. If you’re new to “chrony” then fret not. It’s still using the NTP (Network Time Protocol) that we all know and love but in a more secure fashion.</p>
<p>The first thing I do with Ansible within the “chrony.conf” file is alter the “bind address” and if my memory serves there’s also a “command port” option. These config options allow Chrony to only listen on the localhost. In other words you are still syncing as usual with other upstream time servers (just as NTP does) but no remote servers can query your time services; only your local machine has access.</p>
<p>There’s more information on the “bindcmdaddress 127.0.0.1” and “cmdport 0” on this Chrony page (<a href="https://chrony.tuxfamily.org/faq.html">https://chrony.tuxfamily.org/faq.html</a>) under “2.5. How can I make chronyd more secure?” which you should read for clarity. This premise behind the comment on that page is a good idea: “you can disable the internet command sockets completely by adding cmdport 0 to the configuration file”.</p>
<p>Additionally I would also focus on securing the file permissions for Chrony and insist that the service starts as expected just like the syslog config above. Otherwise make sure that your time sources are sane, have a degree of redundancy with multiple sources set up and then copy the whole config file over using Ansible.</p>
<h4><strong>Logging</strong></h4>
<p>You can clearly affect the level of detail included in the logs from a number pieces of software on a server. Thinking back to what we’ve looked at in relation to syslog already you can also tweak that application’s config using Ansible to your needs and then use the example Ansible above in addition.</p>
<h4><strong>PAM</strong></h4>
<p>Apparently PAM (Pluggable Authentication Modules) has been a part of Linux since 1997. It is undeniably useful (a common use is that you can force SSH to use it for password logins, as per the SSH YAML file above). It is extensible, sophisticated and can perform useful functions such as preventing brute force attacks on password logins using a clever rate limiting system. The syntax varies a little between OSes but if you have the time then getting PAM working well (even if you’re only using SSH keys and not passwords for your logins) is a worthwhile effort. Attackers like their own users on a system with lots of usernames, something innocuous such as “webadmin” or similar might be easy to miss on a server, and PAM can help you out in this respect.</p>
<h4><strong>Auditd</strong></h4>
<p>We’ve looked at logging a little already but what about capturing every “system call” that a kernel makes. The Linux kernel is a super-busy component of any system and logging almost every single thing that a system does is an excellent way of providing post-event forensics. This article will hopefully shed some light on where to begin: <a href="http://www.admin-magazine.com/Archive/2018/43/Auditing-Docker-Containers-in-a-DevOps-Environment">http://www.admin-magazine.com/Archive/2018/43/Auditing-Docker-Containers-in-a-DevOps-Environment</a>. Note the comments in that article about performance, there’s little point in paying extra for compute and disk IO resource because you’ve misconfigured your logging so spend some time getting it correct would be my advice. </p>
<p>For concerns over disk space I will usually change a few lines in the file “/etc/audit/auditd.conf” in order to prevent there firstly being too many log files created and secondly logs that grow very large without being rotated. This is also on the proviso that logs are being ingested upstream via another mechanism too. Clearly the files permissions and the service starting are also the basics you need to cover here too. Generally file permissions for auditd are tight as it’s a “root” oriented service so there’s less changes needed here generally.</p>
<h4><strong>Filesystems</strong></h4>
<p>With a little reading you can discover which filesystems that are made available to your OS by default. You should disable these (at the “modprode.d” file level) with Ansible to prevent weird and wonderful things being attached unwittingly to your servers. You are reducing the attack surface with this approach. The Ansible might look something like this below for example.</p>
<pre>
name: Make sure filesystems which are not needed are forced as off lineinfile: dest="/etcmodprobe.d/harden.conf" line='install squashfs /bin/true' state=present</pre>
<h4><strong>SELinux</strong></h4>
<p>The old, but sometimes avoided due to complexity, security favourite, SELinux, should be set to “enforcing” mode. Or, at the every least, set to log sensibly using “permissive” mode. Permissive mode will at least fill your auditd logs up with any correct rule matches nicely. In terms of what Ansible looks like it’s simple and is along these lines:</p>
<pre>
name: Configure SElinux to be running in permissive mode replace: path=”/etc/selinux/config” regexp='SELINUX=disabled' replace='SELINUX=permissive'</pre>
<h4><strong>Packages</strong></h4>
<p>Needless to say the compliance hardening playbook is also a good place to upgrade all the packages (with some selective exclusions) on the system. Pay attention to the section relating to reboots and idempotency in a moment however. With other mechanisms in place you might not want to update packages here but instead as per the Automation Documents article mentioned in a moment.</p>
<h3><strong>Idempotency</strong></h3>
<p>Now we’ve run through some of the aspects you would want to look at when hardening on a server, let’s think a little more about how the playbook might be used.</p>
<p>When it comes to cloud platforms most of my professional work has been on AWS and therefore, more often than not, a fresh AMI is launched and then a playbook is run over the top of it. There’s a mountain of detail in one way of doing that in this article (<a href="http://www.admin-magazine.com/Archive/2018/45/AWS-Automation-Documents">http://www.admin-magazine.com/Archive/2018/45/AWS-Automation-Documents</a>) which you may be pleased to discover accommodates a mechanism to spawn a script or playbook. </p>
<p>It is important to note, when it comes to idempotency, that it may take a little more effort initially to get your head around the logic involved in being able to re-run Ansible repeatedly without disturbing the required status quo of your server estate.</p>
<p>One thing to be absolutely certain of however (barring rare edge cases) is that after you apply your hardening for the very first time, on a new AMI or server build, you will require a reboot. This is an important element due to a number of system facets not being altered correctly without a reboot. These include applying kernel changes so alterations become live, writing auditd rules as immutable config and also starting or stopping services to improve the security posture.</p>
<p>Note though that you’re probably not going to want to execute all plays in a playbook every twenty or thirty minutes, such as updating all packages and stopping and restarting key customer-facing services. As a result you should factor the logic into your Ansible so that some tasks only run once initially and then maybe write a “completed” placeholder file to the filesystem afterwards for referencing. There’s a million different ways of achieving a status checker.</p>
<p>The nice thing about Ansible is that the logic for rerunning playbooks is implicit and unlike shell scripts which for this type of task can be arduous to code the logic into. Sometimes, such as updating the GRUB bootloader for example, trying to guess the many permutations of a system change can be painful.</p>
<h3><strong>Bedtime Reading</strong></h3>
<p>I still think that you can’t beat trial and error when it comes to computing. Experience is valued for good reason. </p>
<p>Be warned that you’ll find contradictory advice sometimes from the vast array of online resources in this area. Advice differs probably because of the different use cases. The only way to harden the varying flavours of OS to my mind is via a bespoke approach. This is thanks to the environments that servers are used within and the requirements of the security framework or standard that an organisation needs to meet.</p>
<p>For OS hardening details you can check with resources such as the NSA (<a href="https://www.nsa.gov/">https://www.nsa.gov</a>), the Cloud Security Alliance (<a href="https://cloudsecurityalliance.org/working-groups/security-guidance/#_overview">https://cloudsecurityalliance.org/working-groups/security-guidance/#_overview</a>), proprietary training organisations such as GIAC (<a href="https://www.giac.org/">https://www.giac.org</a>) who offer resources (<a href="https://www.giac.org/paper/gcux/97/red-hat-linux-71-installation-hardening-checklist/102167">https://www.giac.org/paper/gcux/97/red-hat-linux-71-installation-hardening-checklist/102167</a>), the diverse CIS Benchmarks (<a href="https://www.cisecurity.org/">https://www.cisecurity.org</a>) for industry consensus-based benchmarking, the SANS Institute (<a href="https://uk.sans.org/score/checklists">https://uk.sans.org/score/checklists</a>), NIST’s Computer Security Research (<a href="https://csrc.nist.gov/">https://csrc.nist.gov</a>) and of course print media too.</p>
<h3><strong>Conclusion</strong></h3>
<p>Hopefully, you can see how powerful an idempotent server infrastructure is and are tempted to try it for yourself.</p>
<p>The ever-present threat of APT (Advanced Persistent Threat) attacks on infrastructure, where a successful attacker will sit silently monitoring events and then when it’s opportune infiltrate deeper into an estate, makes this type of configuration highly valuable.</p>
<p>The amount of detail that goes into the tests and configuration changes is key to the value that such an approach will bring to an organisation. Like the tests in a CI/CD pipeline they’re only as ever as good as their coverage.</p>
<p>Chris Binnie’s latest book, Linux Server Security: Hack and Defend, shows you how to make your servers invisible and perform a variety of attacks. You can find out more about DevSecOps, containers and Linux security on his website: <a href="https://www.devsecops.cc/">https://www.devsecops.cc</a></p>
</div>