How to get the most out of your HostHatch VPS

Hi!

Recently I've migrated to HostHatch as my hosting provider, and while switching (even before, actually) I noticed that my target plan (NVME 16 GB) had only 75 GB of NVMe storage. This is why I also bought Storage VPS 1 TB on the side for $5 which has an HDD so it is not as expensive.

This blog post is meant to serve as a guide to how to get the most out of your HostHatch VPS by using NFSv4.2 (or whatever the latest is at the time you're reading this), Cachefilesd, zRAM, swap, and IPTables (as well as IPSet and Fail2ban), as well as trying to follow common security practices which should be "good enough" for any average person.

This guide, of course, may be applied to other hosting providers, but not everything might be applicable or as easy as described here on other hosting providers. The more changes you make on your end, the more changes you will need to make using this guide.

edit 2025-03-01: I have switched away from HostHatch due to DDoS attacks. HostHatch does not advertise DDoS protection, but they attempted to mitigate the attacks, however the attempts were not fruitful. I also tried to mitigate the attacks but ultimately gave up and switched to ETH-Services, which offers DDoS protection and has successfully handled two DDoS attacks so far! HostHatch was a great experience, with excellent support, performance, stability, and overall service, especially considering the price. However, if you expect to face significant DDoS attacks (such as mine), a hosting provider with DDoS protection might be a better choice. Overall, I would recommend HostHatch :)

# Disclaimer

While I strive to provide accurate and helpful information in this guide, please note that any actions you take based on the content provided are at your own risk. I am not liable for any damages, data loss, or other issues that may arise from following the instructions outlined in this post. Always ensure you have proper backups and consult with a professional if you're unsure about any steps.

This blog post is an independent review and guide on how to optimize your HostHatch VPS. It is not endorsed, sponsored, or affiliated with HostHatch or any other hosting providers mentioned in this blog post. The information provided is based on personal experience and research, and your experience may differ. The author disclaims all liability for any errors or omissions in this information and for its availability.

Happy optimizing!

# Knowledge

This guide assumes you have experience in system administration and understand what you are doing. Common issues while following this guide could be:

Please be careful, and make sure you understand what you are doing. Online resources can help you a lot, but please don't put your bets on AI and LLMs like ChatGPT right away. They tend to respond with error-prone commands and code, so you might not want to play with such fire while doing complex tasks like this where sensitive data might be involved. Don't force yourself into a situation where you have use that backup you (hopefully) made!

# Background

# Hardware

We have extremely limited resources on the storage VPS, so we will try to work around that.

# Operating Systems and Software Stack

This guide should work for pretty much all Linux-based operating systems. Most commonly it is Debian Linux, although nobody is stopping you from using another distribution, such as Alpine Linux, which may even decrease the resource usage.

Personally, I chose Debian Linux because it is very versatile and it has huge software repositories. It worked fine for me over and over again and I believe it to be a very reliable choice.

If you use anything other than Debian or Debian-based (such as Ubuntu) - adjust the procedures as needed based on your software stack.

# Changing the Root Password

HostHatch stores your root password on their end by default. Change this using the passwd command when you have the chance to:

1
passwd

The password should be secure and at least 128 characters long.

# Root Access on the Web VNC Console

If you get locked out of ssh you will need to use the HostHatch cloud web VNC console to access the VPS. For this, you will need to type that password, however this may be annoying. To automatically type this password, at least on Xorg, you can use the xdotool command as follows:

1
2
3
4
5
xdotool type -- 'your-password'
# Or
xdotool type -- "$(cat password.txt)"
# Or
xdotool type -- "$my_password"

On Wayland you could try wlrctl or wtype. Or, if you want a more generic solution that might not always work (at least it didn't work when a mutual tested it) try ydotool or dotool!

If you have the patience to, you could also type the 128+ character password as well, but is that really worth it?

# Reverse DNS

This is mainly a convenience feature, but you might want to change the rDNS of your HostHatch VPS(es). To change the rDNS of your VPS do the following steps:

  1. Log into HostHatch at https://cloud.hosthatch.com/.
  2. Go to your server's panel by clicking on its hostname.
  3. Go to the 'Network' tab.

Then:

# zRAM and Swap Space

Swap space is an extra bit of virtual RAM so to say on your computer which your computer can fall back onto if it runs out of RAM. zRAM is like swap, although, it is compressed and all in-memory.

zRAM might be useful for the processing VPS as it'll require CPU to compress and decompress the RAM, although, it will allow you to get better use out of RAM. While swap might be more useful on the storage VPS due to CPU and memory constraints.

Personally I have set up zRAM and normal swap (with a lower priority) on the processing VPS, and normal swap on the storage VPS.

# zRAM

Following the guide on zRAM on debian.org at https://wiki.debian.org/ZRam you can easily set up zRAM as follows:

1
2
3
apt install zram-tools
echo -e "ALGO=zstd\nPERCENT=60" | tee -a /etc/default/zramswap
systemctl restart zramswap

This will allow zRAM to compress up to 60% of your normal RAM using the ZSTD compression algorithm which provides fast (de)compression with great compression ratios (around 5:1, which means for every 5 units of data it can compress it down to 1 unit).

This is only useful if you have spare CPU to give as the process will be using your CPU more than just using normal swap or just uncompressed RAM.

To mount it on boot, add this to your /etc/fstab file:

1
/dev/zram0 none swap sw,pri=100 0 0

# Swap

There's two main ways of setting swap up on Linux:

I, personally, chose a swap file instead for both VPSes. This is how I set it up:

1
2
3
fallocate -l 4G /swapfile  # You can change the size at your accord
chmod 600 /swapfile
mkswap /swapfile

After doing this, I added this to my /etc/fstab on my server:

1
/swapfile none swap sw,pri=1 0 0

# Finishing

After setting swap up, you may want to reboot. Though in this case it's optional to reboot until the final reboot.

# Private Networking

If you were able to get both your storage VPS and processing VPS in the same location, do the following steps to enable and set private networking up. Do this for both of your VPSes:

  1. Log into HostHatch at https://cloud.hosthatch.com/.
  2. Go to your server's panel by clicking on its hostname.
  3. Go to the 'Network' tab.
  4. Press 'enable private networking'.
  5. Reboot the VPS.

After enabling private networking, reboot the VPSes.

After rebooting, log into your through ssh and follow the private networking guide by HostHatch:

  1. Log in as root (either by pre-sshing as root or using the su command)
  2. Identify the interface name and MAC address using the command ip -o link | grep 00:22 (the MAC address is the one that starts with 00:22:.., and interface will usually be enp2s0 or eth1)
  3. Identify the public IPv4 address of your VPS by running curl -4 ip.me. Remember the last number. (for example last number of 176.126.70.97 is 97)
  4. Run tee /etc/netplan/90-private.yaml and paste in or type out the following text: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
network:
  version: 2
  ethernets:
    [interface name]:
      addresses:
        - 192.168.10.[last number of the current server's public IP address]/24
      match:
        macaddress: 00:22:xx:xx:xx:xx
      dhcp4: no
      mtu: 9000

After you are done: press CTRL+D, and then reboot the VPS. (this is required for private networking to take change if running /usr/sbin/netplan apply won't work)

Note that mtu: 9000 is optional. If it causes issues, do proceed to remove it. Although, since HostHatch claims to support jumbo frames in their docs, you should try to enable it, and by enabling it get a ~33% boost in throughput.

Now you have private networking set up between the VPSes.

# No Private Networking?

No worries! Outside traffic will be blocked using IPTables, although, all the bandwidth will be taken into account while using NFS and the performance might be noticeably worse, especially if the locations are far apart.

If you decide against private networking: Just use the public IP addresses (the ones you see in your HostHatch UI) rather than private ones after setting up private networking (192.168.10.*).

# Firewall with IPTables (Storage Server)

After setting private networking up, you will most likely want to isolate the storage VPS from the rest of the internet to avoid leakage of data. This can be done easily using Iptables and iptables-persistent. This will only cover IPv4 rules, but this can be easily translated into ip6tables as well. I would recommend not using IPv6 on the storage VPS as it is pretty useless in the case of a storage server, and it'll only be more work to manage everything: keep it simple.

Firstly, install the required dependencies:

1
apt install iptables iptables-persistent

Then create a script called iptables.sh as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
#!/bin/sh

# Add /usr/sbin to PATH
export PATH="$PATH:/usr/sbin"

# Flush and discard all iptables policies
iptables -F
iptables -X

# Set default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Accept loopback traffic
iptables -A INPUT -i lo -j ACCEPT

# Accept established and related connections
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# Accept SSH connections on port 22
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Accept TCP connections on NFS ports on server IPs
iptables -A INPUT -s 192.168.10.[last number of the storage server's public IP address] -p tcp --dport 2049 -j ACCEPT
iptables -A INPUT -s 192.168.10.[last number of the processing server's public IP address] -p tcp --dport 2049 -j ACCEPT

# Rate limiting for new SSH connections
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --set

# Drop SSH connections if more than 5 attempts occur within 60 seconds
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 5 -j DROP

# Drop invalid packets
iptables -A INPUT -m state --state INVALID -j DROP

# Accept loopback traffic for outgoing connections
iptables -A OUTPUT -o lo -j ACCEPT

# Save iptables rules
iptables-save >/etc/iptables/rules.v4

You may also want to add the following rules as well to block IPv6 traffic:

1
2
3
4
5
6
7
# Block IPv6
ip6tables -F
ip6tables -X
ip6tables -P INPUT DROP
ip6tables -P OUTPUT DROP
ip6tables -P FORWARD DROP
ip6tables-save >/etc/iptables/rules.v6

After creating this script, go into your HostHatch console and do this:

  1. Click on your server's hostname.
  2. Go into the 'Console' tab.
  3. Log in as root.
  4. Run the script.
  5. Enable the netfilter-persistent service: systemctl enable netfilter-persistent

You should do it this way because you may experience connection issues while applying these IPTables rules.

This script will protect your VPS from brute-force attacks on the SSH port and it'll cut off the VPS from the rest of the internet for the most part.

# Sysctl for Disabiling IPv6

If you want to truly disable IPv6, you will need to edit /etc/sysctl.conf and add this to it:

1
2
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1

After which, run this as root to apply the settings:

1
sysctl -p

Now absolutely no IPv6 traffic will be available in the storage VPS.

# Firewall with IPTables (Processing Server)

If you want IPTables rules for your processing VPS, especially if you also allow IPv6, you are free to use my fw.sh script located at https://git.ari.lt/ari.lt/fw.sh:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
#!/bin/sh

set -eu

main() {
    for ip in iptables ip6tables; do
        echo '----------------------------------------------------------------'

        echo "[$ip] Setting up iptables rules..."

        echo "[$ip] Flushing all rules..."
        "$ip" -F
        "$ip" -X

        echo "[$ip] Allowing established connections..."
        "$ip" -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

        echo "[$ip] Allowing loopback interface..."
        "$ip" -A INPUT -i lo -j ACCEPT
        "$ip" -A OUTPUT -o lo -j ACCEPT

        echo "[$ip] Allowing SSH, HTTP, HTTPS, Email federation, Matrix federation, and XMPP federation on tcp..."
        "$ip" -A INPUT -p tcp --dport 22 -j ACCEPT                                                 # SSH
        "$ip" -A INPUT -p tcp --dport 80 -j ACCEPT                                                 # HTTP
        "$ip" -A INPUT -p tcp --dport 443 -j ACCEPT                                                # HTTPS
        "$ip" -A INPUT -p tcp -m multiport --dports 25,465,587,143,993,110,995,2525,4190 -j ACCEPT # Email federation
        "$ip" -A INPUT -p tcp --dport 8448 -j ACCEPT                                               # Matrix federation
        "$ip" -A INPUT -p tcp -m multiport --dports 5222,5269,5223,5270,5281 -j ACCEPT             # XMPP federation (without 5280 which is HTTP (not HTTPS))

        echo "[$ip] Rate limiting SSH traffic on tcp..."
        "$ip" -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --set
        "$ip" -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 5 -j DROP

        echo "[$ip] Dropping invalid packets on tcp..."
        "$ip" -A INPUT -p tcp -m state --state INVALID -j DROP

        echo "[$ip] Restricting the Git user..."
        "$ip" -A OUTPUT -p all -m owner --uid-owner git -j DROP
        "$ip" -A OUTPUT -p all -m owner --gid-owner git -j DROP

        echo "[$ip] Dropping other traffic..."
        "$ip" -P INPUT DROP
        "$ip" -P FORWARD DROP

        echo "[$ip] Rules:"
        "$ip" -vL

        echo '----------------------------------------------------------------'
    done

    echo '[ICMP] Allowing ICMP...'
    iptables -A INPUT -p icmp -j ACCEPT
    ipiptables -A OUTPUT -p icmp -j ACCEPT
    ip6tables -A INPUT -p icmpv6 -j ACCEPT
    ip6tables -A OUTPUT -p icmpv6 -j ACCEPT

    echo '----------------------------------------------------------------'

    echo '[iptables-save] Saving rules...'
    iptables-save | tee /etc/iptables/rules.v4

    echo '----------------------------------------------------------------'

    echo '[ip6tables-save] Saving rules...'
    ip6tables-save | tee /etc/iptables/rules.v6

    echo 'Meoww :3 done'
}

main "$@"

Make sure no iptables or ip6tables rules are set on the server already so they don't get flushed and you experience networking problems.

# IPSet

For blocking IPs, such as very spammy ones, you may want to use the ipset utility which is used for managing IPSets. To set it up you will have to do the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
apt install ipset ipset-persistent

# IPv4
ipset create blacklist hash:ip
ipset add blacklist <ipv4>
...
iptables -I INPUT -m set --match-set blacklist src -j DROP

# IPv6
ipset create blacklist6 hash:net hashsize 4096 family inet6
ipset add blacklist6 <ipv6>
...
ip6tables -I INPUT -m set --match-set blacklist6 src -j DROP

# Save IPSets
ipset save >/etc/iptables/ipsets
systemctl enable netfilter-persistent

At the end, don't forget to either save your IPTables and IP6Tables rules or add the rules to rules.v* as follows:

For v4:

1
2
3
4
5
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m set --match-set blacklist src -j DROP

For v6:

1
2
3
4
5
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m set --match-set blacklist6 src -j DROP

Ignore the first 4 lines, what I am trying to show is that it must be before all other rules to effectively drop all traffic from blocked IPs.

Now, you can proceed to monitor abusive IPs, for instance like using fail2ban or monitoring various things like /var/log/btmp, for example, to see the IPs that tried to brute force your SSH, you can try to run the following command:

1
lastb -a | awk '{print $10}' | grep -v ^192 | sed '/^$/d' | sort | uniq -c | sort -nr | head -n 32

This will print the top 32 IPs which have tried to brute force SSH to try to get in. I, personally, blocked the most abusive ones (with the most brute force attempts) after collecting data over 3 or so months.

You may also try to integrate things like IPAbuseDB or something, which is another can of worms I probably won't get into for now. You can read an article like https://www.abuseipdb.com/fail2ban.html to integrate it yourself based off the official documentation :)

note: hash:ip is for individual IP addresses, hash:net is for networks (ranges). I've noticed that hash:net behaves weird and doesn't always work, so be careful. To block IP ranges manually you can always do iptables -I INPUT -s x.y.0.0/16 -j DROP or whatever you want to drop. Then, you can iptables -L INPUT --line-numbers and iptables -D INPUT <n> to undo this.

# Fail2ban

To install fail2ban you can just do the following steps:

1
2
apt install fail2ban
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Then do the following:

  1. Open jail.local in your favourite $EDITOR and find [DEFAULT]. There do the following:
    • Set allowipv6 = auto
    • Uncomment ignoreip = 127.0.0.1/8 ::1 and possibly also add your home IP address too.
    • Find bantime.rndtime and set it to 300 or some other value (bantime.rndtime = 300)
    • Change bantime value to 32m.
    • Change findtime value to 16m.
  2. Find [sshd] and change the default section to this: 
1
2
3
4
5
[sshd]
enabled = true
backend = systemd
port = 22
ignoreip = 127.0.0.0/8
  1. Find [nginx-limit-req] and set enabled = true.
    • You may also want to create a custom nginx-429 rule to limit it not only on the proxy, but also the application layer: 
1
2
3
[Definition]
failregex = ^<HOST>.*" 429
ignoreregex =

To enable it you would do:

1
2
3
4
5
6
7
8
[nginx-429]
enabled = true
filter = nginx-429
port = http,https
logpath = /var/log/nginx/access.log
maxretry = 32
findtime = 600
bantime = 1200
  1. Find [nginx-bad-request] and set enabled = true.
  2. Find [php-url-fopen] and set enabled = true.

That's it! You have protected yourself with an automatic network firewall. There's other filters you can enable which you can write yourself or see already pre-written ones in /etc/fail2ban/filter.d/ :)

# NFS (Storage Server)

In this section, we will set up nfs-kernel-server on the storage server.

Firstly do the prerequisite steps:

  1. Make sure you are logged in as root.
  2. Install the required dependencies: apt install nfs-kernel-server nfs-common
  3. Create the shared exports directory, I personally chose /share/nfs: mkdir -p /share/nfs
  4. Set up the correct ownership for the directories: chown nobody:nogroup -R /share
  5. Set up the correct permissions for the directories: chmod 755 -R /share
  6. Enable the NFS service: systemctl enable nfs-kernel-server

Now, simply export the NFS share by editing /etc/exports:

1
/share/nfs 192.168.10.[last number of processing server's public IP](rw,sync,no_subtree_check,async)

If you are going to be using this share for database storage, make sure to remove the async flag as that may lead to data loss and/or corruption. I do that with PostgreSQL:

1
/share/<postgres path> 192.168.10.[last number of processing server's public IP](rw,sync,no_subtree_check)

Next, simply export the filesystems:

1
exportfs -a

And start the NFS service:

1
systemctl start nfs-kernel-server

Now, for the next steps, verify the available NFS versions:

1
2
$ cat /proc/fs/nfsd/versions
+3 +4 +4.1 +4.2

Remember the biggest number that has a + in front of it.

You have successfully set NFS up on the storage server! The NFS server will only be accessible by purely the processing server and noone else.

# NFS (Processing Server)

Now, we are going to set up NFS and Cachefilesd on the processing VPS.

Firstly do the prerequisite steps:

  1. Open /etc/fstab.
  2. Edit your / mount to have the following mount options: rw,discard,errors=remount-ro,x-systemd.growfs,user_xattr,acl.
  3. Reboot the VPS.
  4. Make sure you are logged in as root.
  5. Install the required dependencies: apt install nfs-common
  6. Make the NFS mountpoint: mkdir -p /mnt/nfs
  7. Set up correct ownership: chown nobody:nogroup /mnt/nfs
  8. Set up the correct permissions: chmod 755 /mnt/nfs

Now open up your /etc/fstab and add this:

1
192.168.10.[last number of the storage server's public IP]:/share/nfs /mnt/nfs nfs4 defaults,fsc,noatime,nodiratime,_netdev,x-systemd.automount,x-systemd.requires=network-online.target,timeo=600,rsize=65536,wsize=65536,hard,intr,nfsvers=[latest version of NFS available, such as 4.2],namlen=255,proto=tcp,retrans=2,sec=sys,clientaddr=192.168.10.[last number of the processing server's public IP],local_lock=none,addr=192.168.10.[last number of the storage server's public IP] 0 0

For database storage, you may want to modify these options to:

1
192.168.10.[same]:/share/[database path] /var/lib/[database path] nfs4 defaults,fsc,noatime,nodiratime,_netdev,x-systemd.automount,x-systemd.requires=network-online.target,timeo=600,rsize=65536,wsize=65536,hard,intr,nfsvers=[same],namlen=255,proto=tcp,retrans=2,sec=sys,clientaddr=192.168.10.[same],local_lock=none,addr=192.168.10.[same] 0 0

Don't yet do anything. First, we will set Cachefilesd up (fsc mount option). This will give us better performance by being able to utilize the mass storage of the HDD server and the performance of the NVMe server:

  1. Install Cachefilesd: apt install cachefilesd.
  2. Edit /etc/cachefilesd.conf if needed. (or just use default configuration - it is okay)
  3. Edit /etc/default/cachefilesd and change the RUN=no to RUN=yes.
  4. Start and enable the cachefilesd service: systemctl enable --now cachefilesd.
  5. Check the status, and debug if needed: systemctl status cachefilesd.
  6. Done. You should now reboot the VPS.

NFS is now successfully set up with caching. You can use the mountpoint as any mounted filesystem.

# SSHD (SSH daemon) Configuration

On the processing VPS you may want to use the following configuration only after adding an unprivileged user, adding your public ssh key in ~/.ssh/authorized_keys, and testing it for best security and access management:

First run rm /etc/ssh/ssh_host_* && dpkg-reconfigure openssh-server and then edit /etc/ssh/sshd_config:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
...
Port 22
AddressFamily any
...
SyslogFacility AUTH
LogLevel INFO
...
PermitRootLogin no
...
MaxAuthTries 3
...
PubkeyAuthentication yes
...
AuthorizedKeysFile .ssh/authorized_keys
...
IgnoreRhosts yes
...
PasswordAuthentication no
PermitEmptyPasswords no
...
KbdInteractiveAuthentication no
...
UsePAM yes
..
AllowAgentForwarding no
AllowTcpForwarding no
...
X11Forwarding no
...
PrintMotd no
...
TCPKeepAlive no
...
UseDNS no
...
Banner none
...
AcceptEnv none
...
Subsystem sftp /usr/lib/openssh/sftp-server
...
ChallengeResponseAuthentication no

KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

AuthenticationMethods publickey

HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key

AllowUsers <unprivileged users allowed to SSH into the server>

If you also run a git server you may want to restrict it even more:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
Match User git
    X11Forwarding no
    AllowTcpForwarding no
    AllowAgentForwarding no
    PermitTTY no
    AuthorizedKeysFile /home/git/.ssh/authorized_keys
    PermitTunnel no
    ClientAliveInterval 300
    ClientAliveCountMax 0
    Banner none
    PasswordAuthentication no
    ChallengeResponseAuthentication no
    KbdInteractiveAuthentication no
    PermitOpen none
    PermitListen none

When it comes to client configuration, you may just take one from Mozilla SSH standards pretty much:

1
2
3
4
5
6
ServerAliveInterval 60
HashKnownHosts yes
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256
KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

On the storage VPS you may want to have a singular unprivileged user and only allow traffic from IPv4 (AddressFamily inet). You may also want to specify a Banner /etc/issue instead of none to show a legal disclaimer by overwriting the issue and motd files in etc. Feel free to take this one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
********************************************************************************
*                       WARNING: AUTHORIZED ACCESS ONLY                        *
********************************************************************************
*                                                                              *
* You are accessing a private computer system owned by .......... and operated *
* under the domain ....... This system, including all related equipment,       *
* networks, and network devices (specifically including Internet access), is   *
* provided only for authorized use. This system may be monitored for all       *
* lawful purposes, including to ensure that its use is authorized, for         *
* management of the system, to facilitate protection against unauthorized      *
* access, and to verify security procedures, survivability, and operational    *
* security. Monitoring includes active attacks by authorized entities to test  *
* or verify the security of this system. During monitoring, information may be *
* examined, recorded, copied, and used for authorized purposes. Use of this    *
* system constitutes consent to monitoring for these purposes.                 *
*                                                                              *
* Unauthorized or improper use of this system may result in civil and criminal *
* penalties and administrative or disciplinary action, as appropriate. By      *
* continuing to use this system you indicate your awareness of and consent to  *
* these terms and conditions of use. LOG OFF IMMEDIATELY if you do not agree   *
* to the conditions stated in this warning.                                    *
*                                                                              *
********************************************************************************

            System owned by Jane Dane <jane@example.com> - example.com

# Regenerating Host SSH Keys

To regenerate the host SSH keys in OpenSSH on a Debian system run the following commands:

1
2
3
4
5
rm /etc/ssh/ssh_host_*
# Might need to also `export PATH="$PATH:/sbin:/usr/sbin"`
dpkg-reconfigure openssh-server  # or `ssh-keygen -A`
systemctl restart sshd
systemctl status sshd

This will ensure a fresh set of keys has been populated on your VPS so the keys are surely new and uncompromised in any way whatsoever. This step is very important when using VPS providers to ensure that nobody else but you has the private key just in case the base VMs are reused.

On the client side (your machine), you may need to remove ~/.ssh/known_hosts* or run ssh-keygen -R <hostname_or_IP>.

It is also a good practice to rotate the keys every so often :) Test the connection after you have regenerated the keys without closing the already open connection.

# Using a Different Port

If you use a different port, you can specify it using the Port configuration option, however, don't forget to change IPTables beforehand. When you do, you may want to add this to your ~/.ssh/config:

1
2
3
Host "your.domain.goes_here"
   Hostname "your.domain.goes_here"
   Port <port>

So you could simply ssh your.domain.goes_here instead of having to supply the port using -p every time.

# DNS Servers

For best privacy, security, and generally reliable services - I recommend using Quad9 DNS. You may use these DNS servers by editing /etc/systemd/resolved.conf and setting the following value as such:

1
DNS=9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net

Then either reboot or run:

1
systemctl restart systemd-resolved

# Unattended Upgrades

You may want to set up unattended upgrades meaning your VPS will automatically download stable updates:

1
dpkg-reconfigure unattended-upgrades

# Security Repositories

At least on Debian Linux, you may want to enable security patch repositories to stay up to date with security patches in various software, such as OpenSSH. The security repository allows you to have best security on your server while still keeping up to date with the stability of your Linux distribution of choice.

On Debian, you can create a file such as /etc/apt/sources.list.d/security.list with the following content:

1
2
deb http://security.debian.org/debian-security bookworm-security main contrib non-free
deb-src http://security.debian.org/debian-security bookworm-security main contrib non-free

This applies to Debian Linux 12 "Bookworm". You may change the codename of the repository depending on your Debian version.

# Miscellaneous Maintenance

Maintenance is a difficult never-ending task. Always rely on only yourself to maintain your servers long-term, however, the following commands may be of help:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# To undo iptables just use -D instead of -A

# Block invalid packets
iptables -A INPUT -p tcp \! --syn -m state --state NEW -j DROP

# Block new packets that are not SYN
iptables -A INPUT -p tcp --dport 80 -m connlimit --connlimit-above 20 -j DROP
iptables -A INPUT -p tcp --dport 443 -m connlimit --connlimit-above 20 -j DROP

# Limit connections per source ip
iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 443 -i eth0 -m state --state NEW -m recent --set

# Limit new TCP connections per second per source IP
iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds 1 --hitcount 10 -j DROP
iptables -I INPUT -p tcp --dport 443 -i eth0 -m state --state NEW -m recent --update --seconds 1 --hitcount 10 -j DROP

# Use SYNPROXY for SYN flood protection
iptables -t raw -A PREROUTING -p tcp -m tcp --syn -j CT --notrack
iptables -A INPUT -p tcp -m tcp --syn -j SYNPROXY --sack-perm --timestamp --wscale 7 --mss 1460
iptables -A INPUT -m state --state INVALID -j DROP

# Logging
# iptables -A INPUT -j LOG --log-prefix "IPTables-Dropped: "

# Drop packets from known malicious IPs
# iptables -A INPUT -s 123.123.123.123 -j DROP

# Limit to 400 mbit/s
tc qdisc del dev eth0 root  # Clear existing rules
# Apply new rule
tc qdisc add dev eth0 root handle 1: htb default 30
tc class add dev eth0 parent 1: classid 1:1 htb rate 400mbit ceil 400mbit
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip src 0.0.0.0/0 flowid 1:1

# To undo:
tc filter del dev eth0 protocol ip parent 1:0 prio 1 u32 match ip src 0.0.0.0/0 flowid 1:1
tc class del dev eth0 parent 1: classid 1:1
tc qdisc del dev eth0 root

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# cat /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    monthly
    rotate 0
    missingok
    notifempty
    compress
    delaycompress
    dateext
    dateformat -%Y-%m-%d
    postrotate
        /usr/bin/systemctl reload nginx > /dev/null 2>&1 || true
        /usr/bin/systemctl restart fail2ban > /dev/null 2>&1 || true
    endscript
}

Note that the fail2ban restart command should only be there if you use fail2ban - else remove it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
#!/bin/sh

#
# Automatically renices processes that are using 80% CPU
#

for pid in $(ps -eo pid,%cpu --sort=-%cpu | awk -v threshold=80 '$2 > threshold {print $1}'); do
    renice +10 -p "$pid"
    echo "[auto-renice.sh] Reniced process: $pid"
done

Save this, ensure its permissions are 755, and make cron run it every minute by running crontab -e and typing:

1
* * * * * /path/to/auto-renice.sh

Note that higher priority = less CPU time :)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
iptables -F
iptables -X

iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT

iptables -A INPUT -s xx.xx.xx.xx -j ACCEPT
# ... Repeat for all select IPs ...

1
2
3
4
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_rfc1337=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.default.rp_filter=1

And running sysctl -p :)

# Closing Note

That's about it. Good luck and have fun with your new infrastructure!

(btw that's basically the infrastructure ari.lt runs on at the moment, if I find any bottlenecks - I'll tackle them)

My storage server seems to be idling at about 100M of RAM and around 5% CPU on average, of course with spikes. That play room might seem crazy, but the spikes are even crazier - keep it light and simple on the storage server! It is literally responsible for your storage - be careful and make sure you understand what you are doing.

Cya next time!