how to get the most out of your HostHatch VPS

Hi!

Recently I've migrated to HostHatch as my hosting provider, and while switching (even before, actually) I noticed that my target plan (NVME 16 GB) had only 75 GB of NVMe storage. This is why I also bought Storage VPS 1 TB on the side for $5 which has an HDD so it is not as expensive.

This blog post is meant to serve as a guide to how to get the most out of your HostHatch VPS by using NFSv4.2 (or whatever the latest is at the time you're reading this), Cachefilesd, zRAM, swap, and IPTables.

This guide, of course, may be applied to other hosting providers, but not everything might be applicable or as easy as described here on other hosting providers. The more changes you make on your end, the more changes you will need to make using this guide.

# Disclaimer

While I strive to provide accurate and helpful information in this guide, please note that any actions you take based on the content provided are at your own risk. I am not liable for any damages, data loss, or other issues that may arise from following the instructions outlined in this post. Always ensure you have proper backups and consult with a professional if you're unsure about any steps. Happy optimizing!

# Knowledge

This guide assumes you have experience in system administration and understand what you are doing. Common issues while following this guide could be:

Please be careful, and make sure you understand what you are doing. Online resources can help you a lot, but please don't put your bets on AI and LLMs like ChatGPT right away. They tend to respond with error-prone commands and code, so you might not want to play with such fire while doing complex tasks like this where sensitive data might be involved. Don't force yourself into a situation where you have use that backup you (hopefully) made!

# Hardware

We have extremely limited resources on the storage VPS, so we will try to work around that.

# Operating systems & software stack

This guide should work for pretty much all Linux-based operating systems. Most commonly it is Debian Linux, although nobody is stopping you from using another distribution, such as Alpine Linux, which may even decrease the resource usage.

Personally, I chose Debian Linux because it is very versatile and it has huge software repositories. It worked fine for me over and over again and I believe it to be a very reliable choice.

If you use anything other than Debian or Debian-based (such as Ubuntu) - adjust the procedures as needed based on your software stack.

# Reverse DNS

This is mainly a convenience feature, but you might want to change the rDNS of your HostHatch VPS(es). To change the rDNS of your VPS do the following steps:

  1. Log into HostHatch at https://cloud.hosthatch.com/.
  2. Go to your server's panel by clicking on its hostname.
  3. Go to the 'Network' tab.

Then:

# zRAM and Swap space

Swap space is an extra bit of virtual RAM so to say on your computer which your computer can fall back onto if it runs out of RAM. zRAM is like swap, although, it is compressed and all in-memory.

zRAM might be useful for the processing VPS as it'll require CPU to compress and decompress the RAM, although, it will allow you to get better use out of RAM. While swap might be more useful on the storage VPS due to CPU and memory constraints.

Personally I have set up zRAM and normal swap (with a lower priority) on the processing VPS, and normal swap on the storage VPS.

# zRAM

Following the guide on zRAM on debian.org at https://wiki.debian.org/ZRam you can easily set up zRAM as follows:

apt install zram-tools
echo -e "ALGO=zstd\nPERCENT=60" | tee -a /etc/default/zramswap
systemctl restart zramswap

This will allow zRAM to compress up to 60% of your normal RAM using the ZSTD compression algorithm which provides fast (de)compression with great compression ratios (around 5:1, which means for every 5 units of data it can compress it down to 1 unit).

This is only useful if you have spare CPU to give as the process will be using your CPU more than just using normal swap or just uncompressed RAM.

To mount it on boot, add this to your /etc/fstab file:

/dev/zram0 none swap sw,pri=100 0 0

# Swap

There's two main ways of setting swap up on Linux:

I, personally, chose a swap file instead for both VPSes. This is how I set it up:

fallocate -l 4G /swapfile  # You can change the size at your accord
chmod 600 /swapfile
mkswap /swapfile

After doing this, I added this to my /etc/fstab on my server:

/swapfile none swap sw,pri=1 0 0

# Finishing

After setting swap up, you may want to reboot. Though in this case it's optional to reboot until the final reboot.

# Private networking

If you were able to get both your storage VPS and processing VPS in the same location, do the following steps to enable and set private networking up. Do this for both of your VPSes:

  1. Log into HostHatch at https://cloud.hosthatch.com/.
  2. Go to your server's panel by clicking on its hostname.
  3. Go to the 'Network' tab.
  4. Press 'enable private networking'.
  5. Reboot the VPS.

After enabling private networking, reboot the VPSes.

After rebooting, log into your through ssh and follow the private networking guide by HostHatch:

  1. Log in as root (either by pre-sshing as root or using the su command)
  2. Identify the interface name and MAC address using the command ip -o link | grep 00:22 (the MAC address is the one that starts with 00:22:.., and interface will usually be enp2s0 or eth1)
  3. Identify the public IPv4 address of your VPS by running curl -4 ip.me. Remember the last digit. (for example last digit of 153.92.126.215 is 215)
  4. Run tee /etc/netplan/90-private.yaml and paste in or type out the following text: 
network:
  version: 2
  ethernets:
    [interface name]:
      addresses:
        - 192.168.10.[last digit of the current server's public IP address]/24
      match:
        macaddress: 00:22:xx:xx:xx:xx
      dhcp4: no

After you are done: press CTRL+D, and then reboot the VPS. (this is required for private networking to take change if running /usr/sbin/netplan apply won't work)

Now you have private networking set up between the VPSes.

# No private networking?

No worries - outside traffic will be blocked using IPTables, although, all the bandwidth will be taken into account while using NFS and the performance might be noticeably worse, especially if the locations are far apart.

If you decide against private networking: Just use the public IP addresses (the ones you see in your HostHatch UI) rather than private ones after setting up private networking (192.168.10.*).

# IPTables rules

After setting private networking up, you will most likely want to isolate the storage VPS from the rest of the internet to avoid leakage of data. This can be done easily using Iptables and iptables-persistent. This will only cover IPv4 rules, but this can be easily translated into ip6tables as well. I would recommend not using IPv6 on the storage VPS as it is pretty useless in the case of a storage server, and it'll only be more work to manage everything: keep it simple.

Firstly, install the required dependencies:

apt install iptables iptables-persistent

Then create a script called iptables.sh as follows:

#!/bin/sh

# Add /usr/sbin to PATH
export PATH="$PATH:/usr/sbin"

# Flush and discard all iptables policies
iptables -F
iptables -X

# Set default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Accept loopback traffic
iptables -A INPUT -i lo -j ACCEPT

# Accept established and related connections
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# Accept SSH connections on port 22
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Accept TCP connections on NFS ports on server IPs
iptables -A INPUT -s 192.168.10.[last digit of the storage server's public IP address] -p tcp --dport 2049 -j ACCEPT
iptables -A INPUT -s 192.168.10.[last digit of the processing server's public IP address] -p tcp --dport 2049 -j ACCEPT

# Rate limiting for new SSH connections
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --set

# Drop SSH connections if more than 5 attempts occur within 60 seconds
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 5 -j DROP

# Drop invalid packets
iptables -A INPUT -m state --state INVALID -j DROP

# Accept loopback traffic for outgoing connections
iptables -A OUTPUT -o lo -j ACCEPT

# Save iptables rules
iptables-save >/etc/iptables/rules.v4

You may also want to add the following rules as well to block IPv6 traffic:

# Block IPv6
ip6tables -F
ip6tables -X
ip6tables -P INPUT DROP
ip6tables -P OUTPUT DROP
ip6tables -P FORWARD DROP
ip6tables-save >/etc/iptables/rules.v6

After creating this script, go into your HostHatch console and do this:

  1. Press on your server's hostname.
  2. Go into the 'Console' tab.
  3. Log in as root.
  4. Run the script.
  5. Enable the netfilter-persistent service: systemctl enable netfilter-persistent

You should do it this way because you may experience connection issues while applying these IPTables rules.

This script will protect your VPS from brute-force attacks on the SSH port and it'll cut off the VPS from the rest of the internet for the most part.

# Sysctl for IPv6

If you want to truly disable IPv6, you will need to edit /etc/sysctl.conf and add this to it:

net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1

After which, run this as root to apply the settings:

sysctl -p

Now absolutely no IPv6 traffic will be available in the storage VPS.

# Firewall with IPTables

If you want IPTables rules for your processing VPS, especially if you also allow IPv6, you are free to use my fw.sh script located at https://gist.github.com/TruncatedDinoSour/e0c0c5076448a09c68ff50b00fbfab2b:

#!/bin/sh

set -eu

main() {
    for ip in iptables ip6tables; do
        echo '----------------------------------------------------------------'

        echo "[$ip] Setting up iptables rules..."

        echo "[$ip] Flushing all rules..."
        "$ip" -F
        "$ip" -X

        echo "[$ip] Allowing established connections..."
        "$ip" -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

        echo "[$ip] Allowing loopback interface..."
        "$ip" -A INPUT -i lo -j ACCEPT
        "$ip" -A OUTPUT -o lo -j ACCEPT

        echo "[$ip] Allowing SSH, HTTP, HTTPS, Email federation, Matrix federation, and XMPP federation on tcp..."
        "$ip" -A INPUT -p tcp --dport 22 -j ACCEPT                                                 # SSH
        "$ip" -A INPUT -p tcp --dport 80 -j ACCEPT                                                 # HTTP
        "$ip" -A INPUT -p tcp --dport 443 -j ACCEPT                                                # HTTPS
        "$ip" -A INPUT -p tcp -m multiport --dports 25,465,587,143,993,110,995,2525,4190 -j ACCEPT # Email federation
        "$ip" -A INPUT -p tcp --dport 8448 -j ACCEPT                                               # Matrix federation
        "$ip" -A INPUT -p tcp -m multiport --dports 5222,5269,5223,5270,5281 -j ACCEPT             # XMPP federation (without 5280 which is HTTP (not HTTPS))

        echo "[$ip] Rate limiting SSH traffic on tcp..."
        "$ip" -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --set
        "$ip" -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 5 -j DROP

        echo "[$ip] Dropping invalid packets on tcp..."
        "$ip" -A INPUT -p tcp -m state --state INVALID -j DROP

        echo "[$ip] Dropping other traffic..."
        "$ip" -P INPUT DROP
        "$ip" -P FORWARD DROP

        echo "[$ip] Rules:"
        "$ip" -vL

        echo '----------------------------------------------------------------'
    done

    echo '[ICMP] Allowing limited ICMP traffic...'
    iptables -A INPUT -p icmp -m limit --limit 5/sec --limit-burst 10 -j ACCEPT
    iptables -A INPUT -p icmp -j DROP
    ip6tables -A INPUT -p icmpv6 -m limit --limit 5/sec --limit-burst 10 -j ACCEPT
    ip6tables -A INPUT -p icmpv6 -j DROP

    echo '----------------------------------------------------------------'

    echo '[iptables-save] Saving rules...'
    iptables-save | tee /etc/iptables/rules.v4

    echo '----------------------------------------------------------------'

    echo '[ip6tables-save] Saving rules...'
    ip6tables-save | tee /etc/iptables/rules.v6

    echo 'Meoww :3 done'
}

main "$@"

Make sure no iptables or ip6tables rules are set on the server already so they don't get flushed and you experience networking problems.

# NFS (storage server)

In this section, we will set up nfs-kernel-server on the storage server.

Firstly do the prerequisite steps:

  1. Make sure you are logged in as root.
  2. Install the required dependencies: apt install nfs-kernel-server nfs-common
  3. Create the shared exports directory, I personally chose /share/nfs: mkdir -p /share/nfs
  4. Set up the correct ownership for the directories: chown nobody:nogroup -R /share
  5. Set up the correct permissions for the directories: chmod 755 -R /share
  6. Enable the NFS service: systemctl enable nfs-kernel-server

Now, simply export the NFS share by editing /etc/exports:

/share/nfs 192.168.10.[last digit of processing server's public IP](rw,sync,no_subtree_check,async)

If you are going to be using this share for database storage, make sure to remove the async flag as that may lead to data loss and/or corruption. I do that with PostgreSQL:

/share/<postgres path> 192.168.10.[last digit of processing server's public IP](rw,sync,no_subtree_check)

Next, simply export the filesystems:

exportfs -a

And start the NFS service:

systemctl start nfs-kernel-server

Now, for the next steps, verify the available NFS versions:

$ cat /proc/fs/nfsd/versions
+3 +4 +4.1 +4.2

Remember the biggest number that has a + in front of it.

You have successfully set NFS up on the storage server! The NFS server will only be accessible by purely the processing server and noone else.

# NFS (processing server)

Now, we are going to set up NFS and Cachefilesd on the processing VPS.

Firstly do the prerequisite steps:

  1. Open /etc/fstab.
  2. Edit your / mount to have the following mount options: rw,discard,errors=remount-ro,x-systemd.growfs,user_xattr,acl.
  3. Reboot the VPS.
  4. Make sure you are logged in as root.
  5. Install the required dependencies: apt install nfs-common
  6. Make the NFS mountpoint: mkdir -p /mnt/nfs
  7. Set up correct ownership: chown nobody:nogroup /mnt/nfs
  8. Set up the correct permissions: chmod 755 /mnt/nfs

Now open up your /etc/fstab and add this:

192.168.10.[last digit of the storage server's public IP]:/share/nfs /mnt/nfs nfs4 defaults,fsc,noatime,nodiratime,_netdev,x-systemd.automount,x-systemd.requires=network-online.target,timeo=600,rsize=65536,wsize=65536,hard,intr,nfsvers=[latest version of NFS available, such as 4.2],namlen=255,proto=tcp,retrans=2,sec=sys,clientaddr=192.168.10.[last digit of the processing server's public IP],local_lock=none,addr=192.168.10.[last digit of the storage server's public IP] 0 0

For database storage, you may want to modify these options to:

192.168.10.[same]:/share/[database path] /var/lib/[database path] nfs4 defaults,fsc,noatime,nodiratime,_netdev,x-systemd.automount,x-systemd.requires=network-online.target,timeo=600,rsize=65536,wsize=65536,hard,intr,nfsvers=[same],namlen=255,proto=tcp,retrans=2,sec=sys,clientaddr=192.168.10.[same],local_lock=none,addr=192.168.10.[same] 0 0

Don't yet do anything. First, we will set Cachefilesd up (fsc mount option). This will give us better performance by being able to utilize the mass storage of the HDD server and the performance of the NVMe server:

  1. Install Cachefilesd: apt install cachefilesd.
  2. Edit /etc/cachefilesd.conf if needed. (or just use default configuration - it is okay)
  3. Edit /etc/default/cachefilesd and change the RUN=no to RUN=yes.
  4. Start and enable the cachefilesd service: systemctl enable --now cachefilesd.
  5. Check the status, and debug if needed: systemctl status cachefilesd.
  6. Done. You should now reboot the VPS.

NFS is not successfully set up with caching. You can use the mountpoint as any mounted filesystem.

# SSHD (SSH daemon) configuration

On the processing VPS you may want to use the following configuration only after adding an unprivileged user, adding your public ssh key in ~/.ssh/authorized_keys, and testing it for best security and access management:

First run rm /etc/ssh/ssh_host_* && dpkg-reconfigure openssh-server and then edit /etc/ssh/sshd_config:

...
Port 22
AddressFamily any
...
SyslogFacility AUTH
LogLevel INFO
...
PermitRootLogin no
...
MaxAuthTries 3
...
PubkeyAuthentication yes
...
AuthorizedKeysFile .ssh/authorized_keys
...
IgnoreRhosts yes
...
PasswordAuthentication no
PermitEmptyPasswords no
...
KbdInteractiveAuthentication no
...
UsePAM yes
..
AllowAgentForwarding no
AllowTcpForwarding no
...
X11Forwarding no
...
PrintMotd no
...
TCPKeepAlive no
...
UseDNS no
...
Banner none
...
AcceptEnv none
...
Subsystem sftp /usr/lib/openssh/sftp-server
...
ChallengeResponseAuthentication no

KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

AuthenticationMethods publickey

HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key

AllowUsers <unprivileged users allowed to SSH into the server>

If you also run a git server you may want to restrict it even more:

Match User git
    X11Forwarding no
    AllowTcpForwarding no
    AllowAgentForwarding no
    PermitTTY no
    AuthorizedKeysFile /home/git/.ssh/authorized_keys
    PermitTunnel no
    ClientAliveInterval 300
    ClientAliveCountMax 0

When it comes to client configuration, you may just take one from Mozilla SSH standards pretty much:

ServerAliveInterval 60
HashKnownHosts yes
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256
KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

On the storage VPS you may want to have a singular unprivileged user and only allow traffic from IPv4 (AddressFamily inet). You may also want to specify a Banner /etc/issue to show a legal disclaimer by overwriting the issue and motd files in etc. Feel free to take this one:

********************************************************************************
*                       WARNING: AUTHORIZED ACCESS ONLY                        *
********************************************************************************
*                                                                              *
* You are accessing a private computer system owned by .......... and operated *
* under the domain ....... This system, including all related equipment,       *
* networks, and network devices (specifically including Internet access), is   *
* provided only for authorized use. This system may be monitored for all       *
* lawful purposes, including to ensure that its use is authorized, for         *
* management of the system, to facilitate protection against unauthorized      *
* access, and to verify security procedures, survivability, and operational    *
* security. Monitoring includes active attacks by authorized entities to test  *
* or verify the security of this system. During monitoring, information may be *
* examined, recorded, copied, and used for authorized purposes. Use of this    *
* system constitutes consent to monitoring for these purposes.                 *
*                                                                              *
* Unauthorized or improper use of this system may result in civil and criminal *
* penalties and administrative or disciplinary action, as appropriate. By      *
* continuing to use this system you indicate your awareness of and consent to  *
* these terms and conditions of use. LOG OFF IMMEDIATELY if you do not agree   *
* to the conditions stated in this warning.                                    *
*                                                                              *
********************************************************************************

            System owned by Jane Dane <jane@example.com> - example.com

# DNS servers

For best privacy, security, and generally reliable services - I recommend using Quad9 DNS. You may use these DNS servers by editing /etc/systemd/resolved.conf and setting the following value as such:

DNS=9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net

Then either reboot or run:

systemctl restart systemd-resolved

# Unattended upgrades

You may want to set up unattended upgrades meaning your VPS will automatically download stable updates:

dpkg-reconfigure unattended-upgrades

# Closing note

That's about it. Good luck and have fun with your new infrastructure!

(btw that's basically the infrastructure ari.lt runs on at the moment, if I find any bottlenecks - I'll tackle them)

My storage server seems to be idling at about 100M of RAM and around 5% CPU on average, of course with spikes. That play room might seem crazy, but the spikes are even crazier - keep it light and simple on the storage server! It is literally responsible for your storage - be careful and make sure you understand what you are doing.

Cya next time!