how to set up a matrix server with dendrite, linux and nginx

hello

recently i've set up a matrix server and i went through a lot of pain, so i'm here to document issues i've faced and hopefully i can help more people set up their homeservers quicker and with less issues

also, before we start, i want to clarify that all commands that start with # must be ran as the root user ( for example through sudo or su ), and $ should be ran as normal user ( for example matrix or user or something ), unless stated otherwise

# setup

i wouldn't suggest going below the contabo VPS S SSD hardware-level because it may get slow and painful, especially when joining bigger rooms, i'd even suggest going with contabo VPS M SSD, which is why i'll upgrade soon

# delegation of the main domain

i assume you won't be running your website ( say like https://ari.lt/ ) on the same server you run your matrix server, in my case, i actually even couldn't because of how my website is hosted on netlify and ye, but regardless, i'd very much suggest running matrix ( dendrite )

i personally went for the .well-known delegation method, but you can go for anything you like as there's multiple methods

here's how my .well-known stuff looks :

.well-known/matrix/client :

1
2
3
4
5
{
    "m.homeserver": {
        "base_url": "https://matrix.ari.lt"
    }
}

.well-known/matrix/server :

1
2
3
{
    "m.server": "matrix.ari.lt:443"
}

as seen at https://ari.lt/git, they also don't have to be pretty-printed, i don't know why i made them pretty, but it's fine

few key notes :

this is the easy part

# golang

before anything, we will need to install golang, on debian you can do apt install go-golang, but that may install an old version of go, which isn't desirable, here's how i did it :

this gave me the latest go language compiler, which we will use to compile dendrite as it's written in go

# installing other dependencies

other dependencies are defined in https://matrix-org.github.io/dendrite/installation/planning#dependencies, but at the moment they're :

and for SSL stuff we will also add certbot to our dependencies so we could have a secure SSL connection

to install them, you can run the following :

1
# apt install postgresql postgresql-client nginx certbot python3-certbot-nginx

# preparing database

preparing the database is fairly easy as per the matrix dendrite database setup

firstly, you need to start and enable the postgresql service :

1
# systemctl enable --now postgresql

next, run the following to switch to postgresql control user :

1
2
# su postgres
$ cd

now you should be in the postgres user's home dir, now you will have to create the role for dendrite, set its password and create its database, but before, i have to warn you to create a password such as it shouldn't include non-url-safe characters, else it may be a pain to configure dendrite in the future, this is why i'd recommend you just generate the password using the following command :

1
$ head -n 16 /dev/urandom | base64 -w 0 | shuf | sed 's/[^A-Za-z0-9]//g' | head -c 123

and then using these commands to create the role, set its password and create the database :

1
2
$ createuser -P dendrite
$ createdb -O dendrite -E UTF-8 dendrite

now you're all set with the database

# compiling dendrite

firstly, let's set up the user we'll run dendrite on, it is a good practice to run applications such as this under lowest possible privileges so we don't run into nasty attacks in the future, here's how you do it :

run the following command

1
# useradd -m matrix

this will create a new user called matrix with its own /home/matrix/ directory, we will run dendrite under this user,, next -- set the password for the user :

1
# passwd matrix

make sure to use a secure password, may i recommend pwdtools ? though you can use anything

you may also want to run this to make the home directory of this user only readable by that user :

1
$ chmod 700 -R /home/matrix/

but it's optional

then, switch to the matrix user, you can use any user to do this :

1
$ su matrix

now as you're the matrix user, you should go into your ~ ( home ) directory :

1
$ cd

now as you're the matrix user in its home direcotory, download the latest release tarball ( .tar.gz ) off https://github.com/matrix-org/dendrite/releases/latest and extract it

at the moment for me it's https://github.com/matrix-org/dendrite/archive/refs/tags/v0.13.5.tar.gz so i'll assume the same, although for future readers -- please grab the latest version, here's an example of how to download it and extract it :

1
2
3
$ curl -fLO https://github.com/matrix-org/dendrite/archive/refs/tags/v0.13.5.tar.gz
$ tar xvf v0.13.5.tar.gz
$ cd dendrite-0.13.5/

now you should end up in the latest release of dendrite

according to https://matrix-org.github.io/dendrite/installation/manual/build you should now run the following :

1
$ go build -o bin/ ./cmd/...

keep in mind it's LITERALLY go build -o bin/ ./cmd/... and not for example go build -o bin/ ./cmd/*, it's a literal elipsis, run the command as-is with the 3 dots as go is weird and quirky like that i guess

this should build dendrite, keep in mind this will be network and resource heavy

now you can install it :

1
$ go install ./cmd/dendrite

and your dendrite installation should end up in ~/go/bin/dendrite :)

# signing keys

now, you will generate signing keys for your matrix encryption, which will be used in authentication of federation requests as explained here, the tutorial for setting up keys in dendrite

run the following command in the dendrite directory ( the one you ran commands in # compiling dendrite ) :

1
$ ./bin/generate-keys --private-key matrix_key.pem

never share this key with anyone

# configuring dendrite

this part is based off my own config of dendrite, which is set up by help of people, my own research and the setup docs, keep up-to-date with my configuration and the docs as this section may get outdated, although i'll do my best to keep this up to date as long as i run the ari-web matrix server

firstly copy the example config :

1
$ cp dendrite-sample.yaml dendrite.yaml

and now, open it in your favourite text editor, such as vim for example, maybe nano even :

1
$ vim dendrite.yaml

now, i will cover only some parts of the config which you may want to change, but also the default config includes a lot of comments, so you may want to look through all of it and see what you want or don't

global :

client_api :

sync_api :

user_api :

that's pretty much it with the configuration of dendrite, although i'd still suggest going through all config options at least once and thinking if you want them or not :)

# running dendrite

to run dendrite you can just run

1
$ ~/go/bin/dendrite -config ./dendrite.yaml

and if you want to run it in the background you just run this :

1
$ ~/go/bin/dendrite -config ./dendrite.yaml & disown

simple as that, now dendrite will be listening on port 8008

# dns records

the A record is required

1
2
matrix.yourdomain.tld 3600 IN A        <server ip>
matrix.yourdomain.tld 3600 IN CAA 0 issue <issuer>

you may remove CAA if you disable all the fancy SSL stuff in nginx ( which we're about to configure )

you can also optionally add AAAA for ipv6 support

for ari-web i've set it up like this :

1
2
matrix.ari.lt 3600 IN A            62.171.174.136
matrix.ari.lt 3600 IN CAA 0 issue letsencrypt.org

# configuring nginx

tldr # final nginx config

in our case we will use nginx as our reverse proxy, this i will base off my own nginx config for my vps

firstly, you need to make sure if either user www-data or http exists, you can do that by running

1
$ cat /etc/passwd

which will show you all users

if none do, run

1
# useradd www-data  # or http

after that, make sure that /etc/nginx/mime.types exists :

1
$ ls /etc/nginx/mime.types

if not, run the following command :

1
# curl https://raw.githubusercontent.com/nginx/nginx/master/conf/mime.types -fLo /etc/nginx/mime.types

now, open up /etc/nginx/nginx.conf in your favourite text editor and configure it :

1
# vim /etc/nginx/nginx.conf

we will start by setting up some base rules :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
user www-data;
worker_processes auto;
pid /run/nginx.pid;
worker_rlimit_nofile 8192;

events {
    use epoll;
    multi_accept on;
    worker_connections 4096;
}

make sure to replace www-data with http if you're using the http user instead, here's what this piece of config means :

next, we'll set up some basic config for our server :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
http {
    include mime.types;
    default_type application/octet-stream;

    access_log off;

    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 120;
    types_hash_max_size 2048;
    server_names_hash_bucket_size 256;

    sendfile on;
}

you can dig into this config deeper, but abstractly :

now, we can set up a basic server :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
http {
    ...

    server {
        listen 80;
        listen [::]:80;

        server_name matrix.yourdomain.tld;

        access_log off;

        return 301 https://$server_name$request_uri;
    }

    server {
        listen 443 ssl;
        listen [::]:443 ssl;

        listen 8448 ssl http2 default_server;
        listen [::]:8448 ssl http2 default_server;

        server_name matrix.yourdomain.tld;

        access_log off;
    }
}

here you can also now start nginx :

1
# systemctl enable --now nginx

this sets up the basic requirements for a server, make sure to replace yourdomain.tld / matrix.yourdomain.tld with whatever your preferred domain is, 443 is the https ( tls ) port and 8448 is the secure federation port,, this is where we have to take a step back, save our nginx config and move on for a little bit

# tls ( https )

to set up tls ( https ) on our server now, we will have to run this command :

1
certbot certonly --nginx

and follow the directions on the screen

now, as you have that set up, you can continue setting up your proxy, add SSL stuff :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
http {
    ...

    server {
        listen 443 ssl;
        listen [::]:443 ssl;

        listen 8448 ssl http2 default_server;
        listen [::]:8448 ssl http2 default_server;

        server_name matrix.yourdomain.tld;

        access_log off;

        ssl_certificate /etc/letsencrypt/live/matrix.yourdomain.tld/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/matrix.yourdomain.tld/privkey.pem;
        ssl_trusted_certificate /etc/letsencrypt/live/matrix.yourdomain.tld/fullchain.pem;

        ssl_stapling on;
        ssl_stapling_verify on;
        ssl_session_timeout 1d;
        ssl_session_cache shared:MozSSL:10m;
        ssl_session_tickets off;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256';
        ssl_prefer_server_ciphers on;
    }
}

no, we dont need to touch the :80 one, that will always stay the same as it'll just redirect to https

i won't explain these options, but basically only two lines of SSL things are required :

1
2
ssl_certificate /etc/letsencrypt/live/matrix.yourdomain.tld/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/matrix.yourdomain.tld/privkey.pem;

other stuff is sanity and security, i won't dig deep into it but basically this enables ssl stapling and also sets up some secure preferred ciphers so we know that secure encryption is happening at all times

if you didn't choose to use the CAA record you may use only those two lines instead of all those ssl_* configurations

and now, you are done setting ssl up on nginx

# after ssl ( https )

now, we add our locations :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
http {
    ...

    server {
        listen 443 ssl;
        listen [::]:443 ssl;

        listen 8448 ssl http2 default_server;
        listen [::]:8448 ssl http2 default_server;

        server_name matrix.yourdomain.tld;

        ...

        location = / {
            access_log off;
            return 301 https://$server_name/_matrix/static/;
        }

        location ~ ^(/_matrix|/_synapse/client) {
            access_log off;

            proxy_pass http://127.0.0.1:8008;

            proxy_http_version 1.1;

            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header X-Forwarded-Proto $scheme;

            client_max_body_size 512M;
            proxy_max_temp_file_size 0;
            proxy_buffering off;
        }
    }
}

location = / sets up the / location of your server, it'll always redirect to the dendrite static page so once you visit matrix.yourdomain.tld, it'll always redirect to the welcome page, so it doesn't look as boring, although you can just remove that, it's optional

now location ~ ^(/_matrix|/_synapse/client) is where the actual federation stuff happens, as always we turn off the access log and we pass our requests to our local dendrite instance running on port 8008 and then set up the following things :

# final nginx config

after all this work we end up with a config something like :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
user www-data;
worker_processes auto;
pid /run/nginx.pid;
worker_rlimit_nofile 8192;

events {
    use epoll;
    multi_accept on;
    worker_connections 4096;
}

http {
    include mime.types;
    default_type application/octet-stream;

    access_log off;

    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 120;
    types_hash_max_size 2048;
    server_names_hash_bucket_size 256;

    sendfile on;

    server {
        listen 80;
        listen [::]:80;

        server_name matrix.yourdomain.tld;

        access_log off;

        return 301 https://$server_name$request_uri;
    }

    server {
        listen 443 ssl;
        listen [::]:443 ssl;

        listen 8448 ssl http2 default_server;
        listen [::]:8448 ssl http2 default_server;

        server_name matrix.yourdomain.tld;

        access_log off;

        ssl_certificate /etc/letsencrypt/live/matrix.yourdomain.tld/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/matrix.yourdomain.tld/privkey.pem;
        ssl_trusted_certificate /etc/letsencrypt/live/matrix.yourdomain.tld/fullchain.pem;

        ssl_stapling on;
        ssl_stapling_verify on;
        ssl_session_timeout 1d;
        ssl_session_cache shared:MozSSL:10m;
        ssl_session_tickets off;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256';
        ssl_prefer_server_ciphers on;

        location = / {
            access_log off;
            return 301 https://$server_name/_matrix/static/;
        }

        location ~ ^(/_matrix|/_synapse/client) {
            access_log off;

            proxy_pass http://127.0.0.1:8008;

            proxy_http_version 1.1;

            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header X-Forwarded-Proto $scheme;

            client_max_body_size 512M;
            proxy_max_temp_file_size 0;
            proxy_buffering off;
        }
    }
}

you can now save the file and quit the editor, also, a tip : avoid gzip compression, it kills performance and cpu on your vps, it's painful, stick to vanilla

you can also add this to http block to enable HSTS preload with which you can apply to hstspreload.org :

1
2
3
4
5
6
7
http {
    ...

    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";

    ...
}

# finalizing

now, after all this your matrix server is almost done, all you have to do is :

log into the matrix user :

1
$ su matrix

change to its home :

1
$ cd

kill dendrite :

1
$ pkill -f dendrite

restart nginx

1
# systemctl restart nginx

restart dendrite :

1
$ ~/go/bin/dendrite -config ./dendrite.yaml & disown

# sanity check

now, as everything is up and running, make sure everything's okay by going to matrix.yourdomain.tld, it should show you the dendrite index page, if it doesn't, please verify everything and make sure everything's okay

if you cannot figure it out, you can come and ask in #root:ari.lt or #dendrite:matrix.org

# new user

now, as everything works, you can log in as the matrix user and create a new account for yourself by going into the directory which you built dendrite in ( the one with bin/ directory in it ) and run this :

1
$ ./bin/create-account --config dendrite.yaml -username some_username -admin

this will prompt you for a password, dendrite doesn't seem to like passwords over 72 characters, so make sure it fits

you can also create normal user accounts by doing :

1
$ ./bin/create-account --config dendrite.yaml -username some_username

aka removing the -admin argument

now, you can log in with your favourite matrix client such as for example schildi or element, have fun

# concluding

i hope i could help at least a little, it took me a while to figure out issues, solve problems, find answers and come up with my own solutions, ask people, debug, etc etc etc

a lot of trouble went into this and i hope this popped up in your search engine whenever you're looking to solve such issues as :

( just in case someone decides to look them up and can't find an answer )

1
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

( from curl )

1
2
# [WARNING] Syncloop failed: Client has not connection to the server
# [WARNING] Something went wrong: - Instance of 'SyncConnectionException'

( from fluffychat android )

1
2
Initial sync:
Downloading data...

( from element android and schildichat android )

1
2
Loading... Please wait.
Oops something went wrong...

( from fluffychat android )

1
INFO[2023-12-26T19:32:19.277303943Z] Starting queue due to pending events or forceWakeup

( from dendrite logs )

1
time="2023-12-26T19:16:03.938083486Z" level=info msg="Starting queue due to pending events or forceWakeup" func="github.com/matrix-org/dendrite/federationapi/queue.(*destinationQueue).wakeQueueIfEventsPending" file="/home/matrix/dendrite/federationapi/queue/destinationqueue.go:158"

( from dendrite logs )

1
2
3
2023-12-26T21:42:30*130GMT+00:00Z 171 E/ /Tag: ## Sync: sync service did fail true
java.net.ProtocolException: unexpected end of stream
    ...

( from fluffychat android data logs )

1
Testing matrix.ari.lt failed: mismatching server name, tested: matrix.ari.lt, got: ari.lt // Dendrite 0.13.5+9a5a567

( from @version:envs.net )

1
Homeserver URL does not appear to be a valid Matrix homeserver

( from element web )

1
{"errcode":"M_UNRECOGNIZED","error":"Unrecognized request"}

( from dendrite response )

1
INFO[2023-12-26T00:00:08.867552600Z] Invalid request signature error="Bad signature from \"4d2.org\" with ID \"ed25519:a_MgDi\"" req.id=... req.method=PUT req.path="/\_matrix/federation/v2/invite/!...:4d2.org/$..."

( from dendrite logs )

good luck !