Deploy a Self-Hosted SMTP Relay with Mailcow on Ubuntu VPS

This article provides a guide to deploy a self-hosted SMTP relay with Mailcow on Ubuntu VPS.

Setting up a self-hosted SMTP relay using Mailcow on an Ubuntu VPS allows you to manage email delivery securely and efficiently. This guide walks you through installing and configuring Mailcow for SMTP relaying.

What is an SMTP Relay?

An SMTP relay is a mail transfer process where an SMTP (Simple Mail Transfer Protocol) server forwards outgoing emails from one mail server to another until they reach the recipient’s mail server. This is commonly used to ensure reliable email delivery, especially for bulk or transactional emails.

How an SMTP Relay Works

1. An email is sent – A user or application sends an email through an SMTP server.
2. Mail transfer begins – The SMTP relay server processes the email and determines the recipient’s mail server.
3. The email is forwarded – The relay server forwards the email to the recipient’s mail server using the best available route.
4. Final delivery – The recipient’s mail server accepts the email, and it is stored in their inbox.

Why Use an SMTP Relay?

• Improves Email Deliverability – Reduces the chances of emails being flagged as spam.
• Handles High Volume Emails – Ideal for businesses sending bulk emails (e.g., newsletters, notifications).
• Authentication & Security – Prevents abuse and ensures only authorized users send emails.
• Prevents IP Blacklisting – Relays through a reputable SMTP service to maintain a good sender reputation.
• Outbound Email Control – Allows monitoring, logging, and filtering of outgoing emails.

Types of SMTP Relays

1. Internal SMTP Relay
   • Used within an organization to route internal emails before sending them externally.
   • Example: A company’s local mail server relays outgoing messages through a central relay server.
2. External SMTP Relay (Third-party SMTP Services)
   • Used to send emails via a third-party service (e.g., Mailgun, SendGrid, Amazon SES).
   • Helps businesses avoid setting up their own email infrastructure.
3. Self-hosted SMTP Relay
   • A self-managed email relay setup using Postfix, Exim, or Mailcow.
   • Provides more control but requires proper configuration to avoid spam issues.

What is Mailcow?

Mailcow is an open-source, self-hosted email server suite that provides a complete mail system with a modern web interface. It is based on Docker and includes several essential components such as Postfix, Dovecot, Rspamd, ClamAV, and SOGo to offer a secure and feature-rich mail server.

Key Features of Mailcow

1. Full Email Server Suite – Includes SMTP, IMAP, POP3, and webmail.
2. Modern Web Interface – Allows easy management of mailboxes, domains, and settings.
3. Containerized (Docker-based) – Each service runs in a separate Docker container for better modularity and security.
4. Advanced Spam Filtering – Uses Rspamd and ClamAV for filtering spam and malware.
5. DKIM, SPF, and DMARC Support – Improves email deliverability and prevents spoofing.
6. API Support – Automate email management using a REST API.
7. Multiple Domain Support – Host multiple domains under a single Mailcow instance.
8. Two-Factor Authentication (2FA) – Enhances login security.

Use Cases for Mailcow

• Running a self-hosted email service for personal or business use.
• Setting up an SMTP relay to send bulk or transactional emails.
• Replacing Google Workspace or Microsoft 365 with a self-managed alternative.
• Hosting multiple domains and email accounts on a single server.

Mailcow provides an all-in-one email hosting solution with privacy, control, and security while keeping things user-friendly. It’s a great choice for those who want to avoid third-party email providers and take control of their email infrastructure. 🚀

Prerequisites

Before proceeding, ensure you have:

• An Ubuntu 22.04 or 20.04 VPS with root access
• A fully qualified domain name (FQDN) (e.g., mail.example.com)
• Proper DNS records configured (A, MX, SPF, DKIM, and DMARC)
• A static IP address (recommended to avoid blacklists)
• Docker & Docker Compose installed (Guide: How to Install Docker Engine on Ubuntu VPS Server)

Deploy a Self-Hosted SMTP Relay with Mailcow on Ubuntu VPS

To deploy a self-hosted SMTP relay with Mailcow on Ubuntu VPS, follow the steps below:

1. Update System and Install Dependencies

   Update your system packages:

   sudo apt update && sudo apt upgrade -y

   Install required dependencies:

   sudo apt install git curl wget unzip -y

2. Install Docker and Docker Compose

   Ensure Docker is installed:

   sudo apt install docker.io -y sudo systemctl enable --now docker

   Install Docker Compose:

   sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose

   Verify installation:

   docker --version docker-compose --version

3. Clone and Set Up Mailcow

   Navigate to /opt and clone Mailcow:

   cd /opt sudo git clone https://github.com/mailcow/mailcow-dockerized.git cd mailcow-dockerized

   Copy the sample configuration file:

   cp mailcow.conf.example mailcow.conf

   Edit mailcow.conf to set your domain:

   nano mailcow.conf

   Modify:

   MAILCOW_HOSTNAME=mail.example.com

   Save and exit.

4. Start Mailcow

   Run the Mailcow setup:

   sudo ./generate_config.sh sudo docker-compose pull sudo docker-compose up -d

   This will download and launch Mailcow’s containers.

5. Configure Let’s Encrypt SSL and Automated Renewal

   1. Install Certbot

      sudo apt install certbot

   2. Stop Mailcow Services Temporarily

      cd /opt/mailcow-dockerized sudo docker-compose down

   3. Obtain SSL Certificates

      sudo certbot certonly --standalone -d mail.example.com

      This will generate SSL certificates located in /etc/letsencrypt/live/mail.example.com/.

   4. Configure Mailcow to Use Let’s Encrypt Certificates

      Edit the mailcow.conf file:

      nano mailcow.conf

      Modify the following lines:

      SSL_CERT_PATH=/etc/letsencrypt/live/mail.example.com/fullchain.pem SSL_KEY_PATH=/etc/letsencrypt/live/mail.example.com/privkey.pem

      Save and exit.

      Restart Mailcow Services

      sudo docker-compose up -d

   5. Set Up Automatic Renewal

      Create a renewal script:

      sudo nano /etc/cron.daily/mailcow-certbot-renew

      Add the following lines:

      #!/bin/bash certbot renew --quiet --post-hook "cd /opt/mailcow-dockerized && sudo docker-compose restart"

      Save and exit, then make it executable:

      sudo chmod +x /etc/cron.daily/mailcow-certbot-renew

6. Configure DNS Records

   Set up the following records for your domain:

   1. A Record (Points to VPS IP)

      mail.example.com -> VPS_IP

      See also: How to Point a Domain to Your VPS Server

   2. MX Record (Mail Exchange)

      example.com -> mail.example.com (Priority: 10)

   3. SPF Record (Sender Policy Framework)

      v=spf1 ip4:VPS_IP include:spf.example.com -all

   4. DKIM Record (DomainKeys Identified Mail)

      Generate DKIM keys via Mailcow UI and add the TXT record as instructed.

      DMARC Record (Domain-based Message Authentication)

      v=DMARC1; p=quarantine; rua=mailto:dmarc-reports@example.com;

7. Configure Mailcow for SMTP Relay

   1. Access Mailcow Admin Panel
      Open https://mail.example.com in your browser and log in.
   2. Create a Mailbox
      • Navigate to Configuration > Mailboxes
      • Click Add Mailbox and set up an email (e.g., relay@example.com)
   3. Enable SMTP Relay
      • Go to Configuration > Routing
      • Add an SMTP relay host (if relaying via another provider)
      • Set authentication credentials if required
   4. Adjust Rate Limits
      • Configure sending limits under Configuration > Settings

8. Test SMTP Relay

   Install swaks to test SMTP relay:

   sudo apt install swaks -y

   Send a test email:

   swaks --to recipient@example.com --from relay@example.com --server mail.example.com --auth LOGIN --auth-user relay@example.com --auth-password 'yourpassword'

   Check logs for issues:

   sudo docker-compose logs -f postfix

9. Secure and Maintain Mailcow

   1. Enable Firewall Rules

      sudo ufw allow 25/tcp sudo ufw allow 587/tcp sudo ufw allow 993/tcp sudo ufw enable

   2. Monitor Logs

      Check Mailcow logs regularly:

      sudo docker-compose logs -f postfix

   3. Set Up Automatic Updates

      Update Mailcow periodically:

      cd /opt/mailcow-dockerized sudo docker-compose pull sudo docker-compose up -d

Conclusion

By following this guide, you have successfully deployed a self-hosted SMTP relay using Mailcow on an Ubuntu VPS. This setup ensures secure and reliable email delivery for your applications and services.