Add installation using Docker Compose and many other improvements for implementation in Docker. #614

Keonik1 · 2025-08-09T13:29:04Z

Hi, when I decided to set up a chatmail server for personal use, I was frustrated to discover that there was no Docker installation, so I created a preliminary implementation.

Brief description of changes

Add installation via docker compose (MVP 1)
Add markdown tabs blocks
Fix Issue 604
Add --skip-dns-check argument to cmdeploy run command
Add --force argument to cmdeploy init command
Add startup for fcgiwrap.service
Add extended check when installing unbound.service
Add configuration parameters (is_development_instance, use_foreign_cert_manager, acme_email, change_kernel_settings, fs_inotify_max_user_instances_and_watchers)

Full description of changes

The main thing that was done was to add support for running in Docker, but in addition to that, the problems that made this method of running impossible were also solved.

I think the advantages of running in Docker are obvious:

faster deployment from scratch
greater repeatability of runs
easier migration
better isolation
more comfortable management

This is only the first implementation of installation using Docker, but it works and seems to be stable (at least I haven't found any anomalies), but in the future it will need to be brought to a more adequate state (with the abandonment of using systemd inside and the division of everything into several containers or 1 container, but without systemd management, but that's just a thought for the future).

The modifications were made with a focus on full compatibility with the current installation and are additional optional parameters.

In addition, some QoL features and instance documentation unification were added so that administrators would not have to create separate pages for different language groups of users.

Some errors that I encountered during deployment have also been fixed (and to be honest, I don't understand why some of them didn't occur for others -_-), namely:

Issue 604 with a non-working --ssh_host argument
Manual launch of the fcgiwrap.service service, which does not download or start when rolled out using the current method
Use of a third-party certificate manager
Error when installing unbound.service, when it is installed but does not show that it occupies port 53

All changes are described in detail in the changelog.md.

I hope my improvements meet your requirements and you accept the PR, because I really believe that this is a very important part of the project's development — installation using Docker is easier, which means that many more people will be able to install it and thus popularize it.

- Add markdown tabs blocks - Fix [Issue 604](chatmail#604) - Add `--skip-dns-check` argument to `cmdeploy run` command - Add `--force` argument to `cmdeploy init` command - Add startup for `fcgiwrap.service` - Add extended check when installing `unbound.service` - Add configuration parameters (`is_development_instance`, `use_foreign_cert_manager`, `acme_email`, `change_kernel_settings`, `fs_inotify_max_user_instances_and_watchers`)

hpk42 · 2025-08-11T09:43:47Z

great work and offer of contribution, @Keonik1 -- the silence since friday is more accidental but several people are excited about this PR and want to try it out and feedback. Not doing that myself right now but wanted to let you know :)

cliffmccarthy · 2025-08-11T15:15:42Z

I am very interested in deploying with containers as well. In particular, I would like to decouple from systemd and the host operating system, so that the software content of the components can be managed independently. Doing that will require an understanding of what all the components are, and toward that end, I created #615 to document what I know so far.

I like the re-use here of the pyinfra-based script to populate the container; I think sharing the process as much as possible between containers and host-based deploys is a good idea. I think with some revisions the process could use pyinfra's @local functionality, which would be beneficial with or without containers. Thanks for getting things started on the path towards container-based deploys!

missytake

Soo many changes :) thanks for all the effort and thoughts that went into this.

Sorry for taking some days to reply, but I wanted to look at it in full detail, and have some fix suggestions already. I got acmetool to work, for example.

The tests are mostly fine with this PR, but unfortunately many tests try to login via SSH - maybe we can try to run docker exec if a chatmail container is running on the same machine?
A github action which deploys the docker container automatically would also be nice :) We can use a hetzner VPS for that, like for the cmdeploy CI. It would be good to have CI notice when something fails with a new change, to avoid regressions.

In general, not bad how many variables and configuration options you added, e.g. that CHANGE_KERNEL_SETTINGS is set to False by default in setup_chatmail_docker. In most situations the defaults will probably never be overridden, but it's good that it's explicit what's the behavior.

The markdown tabs are also a nice touch, but we might need more discussion to merge them into main. The question is, which languages do you prefer, which do you offer? Only offering english is not much better, sure. It requires some thought, I think, or could go to an example page so admins can choose to have the tabs if they want to be multi-lingual. Right now, many admins just translate the www directory to their mother tongue.

All in all, thanks for all of this! I suggest to address the comments, and merge it to a docker branch for now, so we can make it more stable in subsequent PRs. But in any case there will be some interested testers already :)))

docs/DOCKER_INSTALLATION_EN.md

CHANGELOG.md

chatmaild/src/chatmaild/ini/chatmail.ini.f

cmdeploy/src/cmdeploy/__init__.py

www/src/page-layout.html

missytake · 2025-08-15T19:04:03Z

cmdeploy/src/cmdeploy/cmdeploy.py

    def get_sshexec():
-        print(f"[ssh] login to {args.config.mail_domain}")
-        return SSHExec(args.config.mail_domain, verbose=args.verbose)
+        host = args.ssh_host if hasattr(args, "ssh_host") and args.ssh_host else args.config.mail_domain
+        print(f"[ssh] login to {host}")
+        return SSHExec(host, verbose=args.verbose)


This is a good step towards getting cmdeploy dns to run as well... but maybe we can completely avoid logging in to localhost? That would be much simpler. Let's see, for now at least deploying works, after all :)

docker/docker-compose-default.yaml

docker/files/setup_chatmail_docker.sh

cmdeploy/src/cmdeploy/cmdeploy.py

Keonik1 · 2025-08-16T10:52:45Z

@missytake, Hello!
Thank you for your feedback. I quickly scanned through it and will try to answer all questions and fix everything that needs to be fixed this week. Unfortunately, I only have free time in the evenings, so I hope the fixes won't take too long.

I've also thought about it and will try to send fixes in the format of 1 fix = 1 commit to make it easier to check.

Keonik1 · 2025-08-17T09:19:33Z

@missytake

The markdown tabs are also a nice touch, but we might need more discussion to merge them into main. The question is, which languages do you prefer, which do you offer?

i have 2 ideas:

First, introduce the languages that we know and can translate well. It is even possible to take a transfer from known current instances.
Add a number of the most popular languages and translate them through a neural network or standard translator, and then wait for a native speaker to come and fix the errors :D

It's also probably worth adding the ability to determine which languages will be displayed in the installed instance, so that admins can simply enable or disable if necessary. I'll think about the implementation of this at the end, when I fix everything else.

Docker

chatmail#614 (comment)

- chatmail#614 (comment) - chatmail#614 (comment)

- chatmail#614 (comment)

chatmail#614 (comment)

…mpose-default chatmail#614 (comment)

chatmail#614 (comment)

- chatmail#614 (comment) - chatmail#614 (comment) - chatmail#614 (comment) - chatmail#614 (comment)

- chatmail#614 (comment)

docker/docker-compose-traefik.yaml

missytake · 2025-08-25T10:24:10Z

cmdeploy/src/cmdeploy/cmdeploy.py

+    if sshexec == "localhost":
+        cmd = f"{pyinf} @local {deploy_path} -y"


This is a good quick fix, maybe we can add something like this to the cmdeploy dns calls as well, so they don't try to request DNS records from the chatmail relay itself, but from the local (docker) host.

docs/DOCKER_INSTALLATION_EN.md

docker/docker-compose-traefik.yaml

docker/files/entrypoint.sh

missytake · 2025-08-25T11:47:53Z

We're getting there! With the last comments addressed, I suggest to merge it into main, but maybe don't mention it in README.md for now (or only as experimental option). But @hpk42 and @link2xt should also add their opinion on this.

To summarize what's needed in the long run:

A refactoring of the SSHExec logic, so it also supports running commands locally with os.subprocess or so, and with docker exec chatmail $command. @hpk42 did the last one and has some opinions about it I think ;)
- We need this to get cmdeploy dns to work, which is a critical part of the setup. Federation will not work if users can't generate an opendkim record. A quick fix like the one in cmdeploy run might be good enough for now, I think I'll work on it today or tomorrow :)
- Getting cmdeploy test to work would be necessary as well, so we can check in the CI whether a change breaks one of the setup options. This might be slightly more involved than getting cmdeploy dns to work.
We need to resolve the question what is the priority: code maintainability or container needs like startup speed and traefik support?
- We should decide whether the traefik setup is going to be part of the supported options, or a suggestion on how to add chatmail to an existing docker host. It adds maintenance load (e.g. checking the traefik setup in the CI).

- chatmail#614 (comment)

missytake

I think we can hardcode this variable, as it is only used in one place anyway :)

missytake · 2025-08-26T09:31:44Z

docker/example.env

+MAIL_DOMAIN="chat.example.com"
+ACME_EMAIL="[email protected]"
+
+CERTS_ROOT_DIR_HOST="./traefik/data/letsencrypt/certs"


This variable is a bit confusing if you use the non-traefik-setup... and without traefik, it is actually not used, right?

It can be used not only for installation with Traefik, but also for any third-party certificate manager. Traefik is simply more convenient in this regard and is a good example.

I think you can write a comment directly in the example.env file stating that if you are using the standard installation, you can leave this field untouched, as it will not have any effect.

Or we can remove it altogether if we want a more linear installation.

missytake · 2025-08-26T09:32:47Z

docker/docker-compose-traefik.yaml

+      ## system
+      - /sys/fs/cgroup:/sys/fs/cgroup:rw # required for systemd
+      - ./:/opt/chatmail
+      - ${CERTS_ROOT_DIR_HOST}:${CERTS_ROOT_DIR_CONTAINER}:ro


Suggested change

- ${CERTS_ROOT_DIR_HOST}:${CERTS_ROOT_DIR_CONTAINER}:ro

- ./traefik/data/letsencrypt/certs:${CERTS_ROOT_DIR_CONTAINER}:ro

Then we could also just hardcode the value here.

link2xt

Have not read to the end yet.

link2xt · 2025-08-26T19:12:54Z

chatmaild/src/chatmaild/config.py

+            params.get("change_kernel_settings", "true").lower() == "true"
+        )
+        self.fs_inotify_max_user_instances_and_watchers = int(
+            params["fs_inotify_max_user_instances_and_watchers"]


This looks like it will fail if the setting is not in the config file.

link2xt · 2025-08-26T19:49:07Z

cmdeploy/src/cmdeploy/cmdeploy.py

-        out.green(f"created config file for {mail_domain} in {args.inipath}")
+        if not args.recreate_ini:
+            out.green(f"[WARNING] Path exists, not modifying: {inipath}")
+            return 1 ### need research - can we set return code as zero?


I think we can because nobody uses it, but why? Running init when it does nothing is likely not what you want.

Well, this is probably an architectural issue. I think that calling init again shouldn't cause an error.

@hpk42 @missytake
What do you think about this?

I agree with @link2xt that if re-creating the chatmail.ini file is not wished for, an existing chatmail.ini is an error; your command will not have the effect you want it to have (creating a default configuration file for a given domain).

Scripts which want to re-create the ini like setup_chatmail_docker.sh can simply pass --force or || true here, depending on the desired behavior.

link2xt · 2025-08-26T20:20:01Z

cmdeploy/src/cmdeploy/cmdeploy.py

+        "--skip-dns-check",
+        dest="dns_check_disabled",
+        action="store_true",
+        help="disable checks nslookup for dns",


Suggested change

help="disable checks nslookup for dns",

help="disable nslookup checks for DNS",

link2xt · 2025-08-26T20:22:48Z

docker/files/entrypoint.sh

+if [ "${USE_FOREIGN_CERT_MANAGER,,}" == "true" ]; then
+    if [ ! -f "$PATH_TO_SSL/fullchain" ]; then
+        echo "Error: file '$PATH_TO_SSL/fullchain' does not exist. Exiting..." > /dev/stderr
+        sleep 2


What is the reason for sleep here? Is it a workaround for something?

@link2xt because it will restart ~5 containers per second, but we need 2-10 seconds to obtain certificates

link2xt · 2025-08-26T20:27:29Z

docker/files/entrypoint.sh

+
+unlink /etc/nginx/sites-enabled/default || true
+
+if [ "${USE_FOREIGN_CERT_MANAGER,,}" == "true" ]; then


Suggested change

if [ "${USE_FOREIGN_CERT_MANAGER,,}" == "true" ]; then

if [ "${USE_FOREIGN_CERT_MANAGER,,}" = true ]; then

= is standard, == is a bash extension for pattern matching that is not needed here.

link2xt · 2025-08-26T20:31:39Z

docker/files/entrypoint.sh

+    fi
+fi
+
+SETUP_CHATMAIL_SERVICE_PATH="${SETUP_CHATMAIL_SERVICE_PATH:-/lib/systemd/system/setup_chatmail.service}"


Suggested change

SETUP_CHATMAIL_SERVICE_PATH="${SETUP_CHATMAIL_SERVICE_PATH:-/lib/systemd/system/setup_chatmail.service}"

: "${SETUP_CHATMAIL_SERVICE_PATH:=/lib/systemd/system/setup_chatmail.service}"

Same result, just shorter.

True, but also harder to read for people without an extensive bash background^^ 🤷

link2xt · 2025-08-26T20:33:06Z

docker/files/entrypoint.sh

+SETUP_CHATMAIL_SERVICE_PATH="${SETUP_CHATMAIL_SERVICE_PATH:-/lib/systemd/system/setup_chatmail.service}"
+
+env_vars=$(printenv | cut -d= -f1 | xargs)
+sed -i "s|<envs_list>|$env_vars|g" $SETUP_CHATMAIL_SERVICE_PATH


Do we actually need to pass all the environment variables? This is not easy to understand without reading the setup_chatmail.service template. At least needs a comment to explain what happens here.

@link2xt
systemd starts services without passing environment variables to them. A running service can only use the environment variables specified in the some.service file.

missytake reviewed Aug 15, 2025

View reviewed changes

Keonik1 closed this Aug 17, 2025

Keonik1 force-pushed the main branch from 3826de8 to 3ce350d Compare August 17, 2025 09:31

Keonik1 and others added 2 commits August 17, 2025 12:34

Merge branch 'main' into docker

5aef295

Merge pull request #1 from Keonik1/docker

b81e471

Docker

Keonik1 reopened this Aug 17, 2025

Keonik1 added 15 commits August 17, 2025 13:01

Update Changelog

4a92e50

chatmail#614 (comment)

Fix description for is_development_instance option

6425a83

- chatmail#614 (comment) - chatmail#614 (comment)

revert page-layout logo link

1c4c7b9

- chatmail#614 (comment)

rename dockerfile

aea6366

chatmail#614 (comment)

add port 80 to docker-compose-default

b6dce61

chatmail#614 (comment)

add 465 port

a6e5b9e

chatmail#614 (comment)

add RECREATE_VENV var

a01eebe

chatmail#614 (comment)

Add traefik config files

d545fc8

chatmail#614 (comment)

change "restart nginx" to "reload nginx"

9037409

chatmail#614 (comment)

pass values to MAIL_DOMAIN and ACME_EMAIL from vars for docker-co…

dc6d8b4

…mpose-default chatmail#614 (comment)

Fix bug with attaching certs

4fc672c

fix for lint test

4c42d0f

fix docs - nginx "restart" to "reload"

87615b6

chatmail#614 (comment)

Delete ssh connection from docker installation

1b3f419

- chatmail#614 (comment) - chatmail#614 (comment) - chatmail#614 (comment) - chatmail#614 (comment)

Fix issue with acmetool

d5329fa

- chatmail#614 (comment)

missytake reviewed Aug 25, 2025

View reviewed changes

docker/docker-compose-traefik.yaml Outdated Show resolved Hide resolved

missytake reviewed Aug 25, 2025

View reviewed changes

docs/DOCKER_INSTALLATION_EN.md Outdated Show resolved Hide resolved

missytake reviewed Aug 25, 2025

View reviewed changes

docker/docker-compose-traefik.yaml Outdated Show resolved Hide resolved

missytake reviewed Aug 25, 2025

View reviewed changes

docker/files/entrypoint.sh Outdated Show resolved Hide resolved

Keonik1 added 5 commits August 25, 2025 22:03

delete default value for ACME_EMAIL

5dcb002

- chatmail#614 (comment)

delete sudo from traefik init container cmd

f027afd

- chatmail#614 (comment)

fix unlink if default nginx conf is not exist

e1ca74e

- chatmail#614 (comment)

try to fix tests

c372c55

- chatmail#614 (comment)

fix docs; revert tests

929383d

- chatmail#614 (comment)

missytake reviewed Aug 26, 2025

View reviewed changes

Keonik1 added 2 commits August 26, 2025 21:04

Fix colored output file; return original exit code

910eeea

add a workaround to pass the test_init_not_overwrite test

a1301fa

link2xt reviewed Aug 26, 2025

View reviewed changes

		if sshexec == "localhost":
		cmd = f"{pyinf} @local {deploy_path} -y"

	- ${CERTS_ROOT_DIR_HOST}:${CERTS_ROOT_DIR_CONTAINER}:ro
	- ./traefik/data/letsencrypt/certs:${CERTS_ROOT_DIR_CONTAINER}:ro

	help="disable checks nslookup for dns",
	help="disable nslookup checks for DNS",


		unlink /etc/nginx/sites-enabled/default \|\| true

		if [ "${USE_FOREIGN_CERT_MANAGER,,}" == "true" ]; then

	if [ "${USE_FOREIGN_CERT_MANAGER,,}" == "true" ]; then
	if [ "${USE_FOREIGN_CERT_MANAGER,,}" = true ]; then

	SETUP_CHATMAIL_SERVICE_PATH="${SETUP_CHATMAIL_SERVICE_PATH:-/lib/systemd/system/setup_chatmail.service}"
	: "${SETUP_CHATMAIL_SERVICE_PATH:=/lib/systemd/system/setup_chatmail.service}"

Uh oh!

Add installation using Docker Compose and many other improvements for implementation in Docker. #614

Are you sure you want to change the base?

Add installation using Docker Compose and many other improvements for implementation in Docker. #614

Uh oh!

Conversation

Keonik1 commented Aug 9, 2025

Brief description of changes

Full description of changes

Uh oh!

hpk42 commented Aug 11, 2025

Uh oh!

cliffmccarthy commented Aug 11, 2025

Uh oh!

missytake left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Keonik1 commented Aug 16, 2025

Uh oh!

Keonik1 commented Aug 17, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

missytake commented Aug 25, 2025

Uh oh!

missytake left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

link2xt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

missytake left a comment •

edited

Loading