28.04.2020 13:24
timurxyz 0 793 Источник
The case: You fire up a professionally prepared Linux image at a cloud platform provider (Amazon, DO, Google, Azure, etc.) and it will run a kind of production level service moderately exposed to hacking attacks (non-targeted, non-advanced threats).

What would be the standard quick security related tuning to configure before you install the meat?


release: 2005, Ubuntu + CentOS (supposed to work with Amazon Linux, Fedora, Debian, RHEL as well)


image


Disclaimers:

  • Read the micro threat model given in "The case" paragraph above before you compare my advise with your requirements. Otherwise please help me improve this page with your comments!
  • It's not at all a guide for Docker containers or for a seriously hardened production server.
  • The author is not a certified unix admin or a devsecops guru.
  • It's a subjective advise!


Assumptions


  • You are already experienced in Linux administration.
  • As an admin you work on a trusted machine (desktop, laptop or tablet), properly hardened, continuously patched, with security not weared off by years of usage or being shared.
  • There is a handy facility to generate and store passwords available on the above admin client.
  • Your SSH keys or other credentials to admin the instance are handled securely on your admin machine. (An issue in itself how to satisfy this requirement. Like should your keys be enclosed in password or token protected key stores or drives?!)
  • Your access to the cloud platform provider is secure enough, and your user/account there is handled with security in mind.
  • You manage the cloud platform provider's web UI in a browser which you use solely for admin tasks.
  • To sum the above: The security context of your installation (instance cooking) work is on a higher level of assurance than the level you would expect from the instance you configure.
  • You use a Linux image provided by a party whose security competence is assumed (like the image baked by the cloud platform provider themselves or the Linux distro builder).
  • You login to the instance via a trusted and clean terminal app or within admin browser via the native web-CLI.

The best approach is to adhere to cyber hygiene practices from the zero moment and not rely on an idea that security can be hardened in a bonus step afterwards.


Our advice would be to use — a corporate managed — iPad as a trusted admin client at a remote location or a managed desktop at the enterprise internal network. In case you/the company are/is subject for targeted attacks then laptops are a weaker choice for paranoids. But that's an other story to which I will dedicate an article or a post later.

The steps


Have a matching firewall protection enabled


I mean which serves as the internet facing firewall behind which your instance is running. Amazon calls it the security group, in DO that's the firewalls feature of a project. This filters ports internet connections can hit on your instance. Plan which external firewall preset will match the open ports of the instance you are installing. SSH + the ports of services. The external firewall may allow more ports as the preset may serve several types of instances. The good old approach to minimize the open ports is still valid.


You may have different firewall presets ready for different stages of your installations. Like in the beginning port 22 is the one you start off, but when a non standard SSH port is configured you may switch to a preset with 22 closed forever and the corresponding internet noise will not hit your instance.


Choosing the initial connect/login method


  1. In case you have your working practice to connect to the cloud platform resources, connect as you deem it safe. Otherwise:
  2. If the cloud platform allows I would choose the option with connecting to ssh with a random root password generated by the platform (DO offers this). See the explanation of my pro-passwords (long random passwords) point of view in a below section ('Password authentication is wrong, key files rule! (Disagreed?)').

Don't let a fresh instance run as is in the wild


Be paranoid, don't let a fresh instance to run unpatched while exposed to internets. When fired up log in within minutes to start patching.


Login


… and


sudo -s

Unless otherwise indicated all below commands you issue as root.

Make sure the bare system is up-to-date


# Ubuntu
apt update
apt ugrade
# CentOS
dnf update
dnf upgrade
# + both optionally:
shutdown -r now

Restarting is technically not necessary, but it won't harm.


Basic steps


  • Enable the local firewall and allow some ports. (It's indeed useless to enable the firewall if you already enabled a frontfacing firewall on the provider's host, but still.)
  • Set the timezone.

Ubuntu:


ufw enable
ufw allow 22, <*your custom ssh, eg 52112*>
# see the SSHd setup below
ufw status
# make your time meaningful, change the location:
timedatectl set-timezone Europe/Berlin

CentOS:


# may firewalld not present:
dnf install firewalld
systemctl enable firewalld --now
# then
firewall-cmd --state
# should be '*running*'
firewall-cmd --list-all
# should be telling something like:
# *public (active)*
# *services: cockpit dhcpv6-client ssh*
firewall-cmd --list-all-zones | more
firewall-cmd --get-default-zone
# public
# add a custom ssh port, see the SSHd setup below
firewall-cmd --permanent --zone=public --add-port=<*your custom ssh, eg 52112*>/tcp
# success
firewall-cmd --complete-reload
firewall-cmd --list-all
# now should also contain your custom ssh port

# make your time meaningful, change the location:
timedatectl set-timezone Europe/Berlin

SELinux, AppArmor


Check the AppArmor or SELinux status:


# Ubuntu:
apparmor_status
# CentOS:
sestatus

It's not that you should bother with it, but it's still more secure to utilize an Linux distro preconfigured with LSM in enforcing mode active, so try to make it an aspect when you choose your provider and select from the factory maintained images there. However based on other's opinions and considering the "micro threat model" set out above I would not make it a prerequisite. In order for LSM to make real sense it should be tuned adequately to your services and the particular situation.


Comfort first


Make yourself comfortable and productive, like:


  • zsh, fish or a similar advanced shell will help you a lot (tuning it is not discussed here).
  • wget is used in this guide.
  • Don't hesitate to install the editor of choice as early as possible (I promote micro here, tho it's a bit more complicated to install).

apt/dnf install zsh wget
which zsh
echo $0
# Ubuntu
snap install micro --classic

Yes I start with suppressing a security alert. Let's be realistic. Let's face it all systems are non-secure, even bash had horrible security flaws, everything has, maybe except ssh written by paranoid and methodic freaks.))

Alternatively install micro from:


https://github.com/zyedidia/micro/releases/


  • See my suggestions in the below 'Then services, folders, extensions' section.

mkdir -p /tank/packagez
chown -R root:wheel /tank/packagez
mkdir /opt/bin

cd /tank/packagez
wget https://github.com/zyedidia/micro/releases/download/v2<...>/micro-2<...>-linux64.tar.gz
tar xf micro...
mv micro.../micro /opt/bin/

chown -R root:root /opt/bin
chmod -R 755 /opt/bin
export PATH="$PATH:/opt/bin"
micro /etc/profile.d/env.sh
# create or add
export PATH="$PATH:/opt/bin"
# * or to the PATH in /etc/environment if that one is in use by the os

Password quality


Mod the password quality settings:


# on Ubuntu you may need to install this first
apt install libpam-pwquality
# then
micro /etc/security/pwquality.conf
# enable:
minlen = 20
ocredit = -2

This will set the minimum password length to 20 and require 2 special characters in it. Why 20? Ok, let it be 25. See also my bellow comment regarding passwords vs key files. (I suggest using random passwords and SSH password authentication instead of key files, hence the length. See the explanation of my pov in the below 'Password authentication is wrong, key files rule! (Disagreed?)' section.


Your personal user


For sshing your persistent instance imo a good practice would look like follows:


ssh -p 52112 lola@acme.web

  • custom user, custom port

Create an admin user:


# Ubuntu
useradd --uid <*1111*> -N --home /home/.<*lola*> --shell /usr/bin/zsh -g admin -G users <*lola*>
# CentOS
useradd --uid <*1111*> -N --home /home/.<*lola*> --shell /bin/zsh -g wheel -G users <*lola*>

  • mind to rewrite 'lola' to your nick
  • UID does not matter actually
  • zsh may not be your choice
  • mind that home directory is hidden in the above example (yes I think obscurity adds to security a bit))

Create your password as a random token. Eg.:


< /dev/urandom tr -cd '[:alpha:][:digit:]_!$%' | head -c30 ; echo ""
# OR create it on the client device and copy or retype
passwd <*lola*>

Tweaking the choice of special characters try to use the ones which are available for manual entering on different devices.


SSHd mods


Consider this: a) SSHd config is already properly set, b) there are many settings in there which makes sense to harden. So you can choose from leaving it as is to diving into tuning it to death (given you read a lot about the meaning and the effects).


Consider the following quick mods:


  • custom port (52112 is an example)
  • disallowing root and any unexpected account to login

Note: A stupid mistake in sshd config may lock out your access forever. (Except that you may login via the providers console.) So:

Open two SSH connections to the remote instance, one with the original default user and one with your new custom user. SSHd will not lock out a live connection in most situations even if reloaded with a broken configuration (failed restart will not kick your live sessions in most cases).


Modify the below setting in the sshd config. The settings in < > are to be fixed by you. 


micro /etc/ssh/sshd_config
# mods:
Port <*52112*>
AllowUsers <*lola*>
DenyUsers root guest test admin toor *ec2-user bitnami <...default accounts>*
DenyGroups
AllowGroups

  • Mind to use the same extra port which you accepted with the local and external firewalls.

For better cipher, mac and key exchange settings add the following to the end of the config:


KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes256-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

You may also mod the below settings in the sshd_config (keep all the other original settings as they are!), walk thru it, edit (uncomment and edit) the mentioned below lines. (It's not the new content of the config, these are the lines in the original which I suggest to modify accordingly).


LoginGraceTime 45
PermitRootLogin no
MaxAuthTries 6
MaxSessions 3
ClientAliveInterval 60
ClientAliveCountMax 2
ChallengeResponseAuthentication no
GSSAPIAuthentication no
AllowAgentForwarding no
AllowTcpForwarding no
GatewayPorts no
X11Forwarding no
PermitUserEnvironment no

If there are duplicate settings, SSH will take the last one, so check the whole config file. (Technically if you add the above to the end of the config file that will override the previous settings.)


You may also disallow (with a leading hash) the SFTP subsystem unless you know scp is something you really need and can't solve it otherwise. (Consider that scp allows for dumping any amount of data from your system.)


#Subsystem sftp /usr/lib/openssh/sftp-server

Restart the service, check the status and review the effective settings:


systemctl restart sshd.service
systemctl status sshd.service
# check the listening port
sshd -T
# in case of errors:
# - Ubuntu:
tail -30 /var/log/syslog
# - CentOS:
tail -30 /var/log/messages
# - both:
journalctl -xe

Keeping the live SSH connections/terminals alive, open a new terminal and check if connecting with your custom admin user works:


ssh -p <52112> <lola>@<IP>

Restart your instance and ssh into it with your user.


Password authentication is wrong, key files rule! (Disagreed??)


You may wonder why don't I recommend to immediately forbid PasswordAuthentication and allow PubkeyAuthentication only?!


You may prove the opposite, but I see no practical cryptographic strength difference between a large random password and a key. Beyond a certain level of entropy we care to put into our password. A random password and a key are both just a bunch of random bytes.


From the practical point of view then we have a significant difference in handling the two kinds of secrets. A 20-30 char password you can even type in while looking at it stored in your password store well protected on an iOS device. With the keys there is always a drama of moving those around and protecting the key file. Unless you have an enterprise grade key store. So it's up to you. Obviously you can if you prefer so:


PasswordAuthentication no

May you want and have an MFA capability to use a token like Yubikey — that would be the winning solution to heighten the level of assurance.

Kill the root user


When you are done with the ssh configuration, and your custom user safely logs in to the instance, it's time to kill the root. Not that it's absolutely a good idea, but you never know how it was misconfigured. :) 'Killing' is done by assigning a random password.


Check interactive password holders (Ubuntu only):


passwd -Sa | grep P

Change the root password to a random one (Ubuntu, CentOS):


echo -n "root:$( < /dev/urandom tr -cd '[:lower:][:digit:]_!$%,.[=*=][=#=]' | head -c40 )" | chpasswd

Note that root can assign any password (despite the fine policy we created above), so be careful with issuing the command without tuning, or test what it does.


If there were other interactive users assign random passwords to them unless you understand the reason for their interactive presence at your instance.


Then services, folders, extensions


Now you are done with installing the base system — bada-bing-bada-boom.


Pproceed to installing your services. That may involve adding new available ports to the firewall and new users. But first make the standard folder structure.


/tank


My subjective practice is to create a /tank folder for storing custom stuff including the installation material:


mkdir -p /tank/packagez
chown -R root:admin /tank/packagez
# * under CentOS admin is wheel
cd /tank/packagez

So the 'packagez' (or anything like that, 'install', you name it) will be the location to download the installation sources which you can return to and check later what was the source of the running softwares. Like:


wget [https://github.com/caddyserver/caddy/releases/download/v2.0.0-rc.1/caddy_2.0.0-rc.1_Linux_x86_64.tar.gz](https://github.com/caddyserver/caddy/releases/download/v2.0.0-rc.1/caddy_2.0.0-rc.1_Linux_x86_64.tar.gz)

/opt


I would suggest to install your tools and service binaries to the /opt folder. This ensures that you can keep track of what additional software you deployed on the instance.


Using caddy as an example:


cd /tank/packagez
tar xf caddy -C /opt/caddy
chown -R root:root /opt/caddy
chmod -R 755 /opt/caddy
/opt/caddy/caddy —help

caddy is handy to fire an HTTPS service out of nothing:
caddy reverse-proxy --from acme.web --to localhost:9000

Configurations


As for the location of configuration stuff I would suggest to follow the standard guides. Like mostly that suggests /etc/:

mkdir /etc/caddy
touch /etc/caddy/Caddyfile
chown -R root:caddy /etc/caddy
mkdir /etc/ssl/caddy
chown -R root:caddy /etc/ssl/caddy
chmod 0770 /etc/ssl/caddy

Create a user for the service


For the service users a good idea would be to not assign a working shell to them, like:


groupadd --system caddy
useradd --system --gid caddy --create-home --home-dir /var/lib/caddy --shell /bin/false --comment "Caddy web server" caddy

  • Mind to assign a fake shell to it. Alternatively: --shell /usr/sbin/nologin

systemd


Mostly your services will require automatic start. For that you presumably use systemd. Like:


wget https://raw.githubusercontent.com/caddyserver/dist/master/init/caddy.service -P /etc/systemd/system
chmod 644 /etc/systemd/system/caddy.service
micro /etc/systemd/system/caddy.service
# General approach
User=caddy
Group=caddy
...
PrivateTmp=true
PrivateDevices=true
ProtectHome=true
ProtectSystem=full

# Then:
systemctl daemon-reload
systemctl status caddy
journalctl -u caddy -b -f
tail -40 /var/log/caddy.log
# when works
systemctl enable caddy

by: @timurxyz
org: secdev.eu, defdev.eu

References, sources