Tips to secure mail server

Are you trying to secure your mail server ?

This guide will help you.

Email is inherently an insecure method of communication. All mail is sent through Simple Mail Transfer Protocol (SMTP), which does not use encryption or authentication
Email sent through SMTP can be accessed by outsiders due to the lack of security protocols.
It is quite common for people to exploit mail servers. Unfortunately, the same system that provides an efficient way to communicate with others can be exploited for malicious purposes if we don't configure it properly.
We know that there is no full-proof security; therefore optimal protection should substitute perfection.
Here at Ibmi Media, as part of our Server Management Services, we regularly help our Customers to perform Mail Server Queries.
In this context, we shall look into methods to secure mail server.

How to secure mail server ?

The security of our mailing infrastructure is closely tied to our sender reputation and is a building block for establishing long-lasting relationships with our customers.
A few after-effects of a hack or spam in our subscriber's inboxes, includes:
i. Creates a lot of complaints against our domains and IP addresses.
ii. A drop in subscriber engagement with our legitimate email.
iii. Both subscribers and Mailbox Providers (MBP’s) could block our mail.
iv. Malicious actors will likely send spam to random email addresses, which usually includes a high number of spam traps.
v. We are likely to be listed on publicly available blacklists.

Now let's look at some tips to apply in order to secure an email server.

1. Set maximum message size

There is a slight possibility, that the server might crash if it processes large mail messages, especially if we send them to multiple recipients at once.
To avoid this, we set an appropriate maximum message size for your server.

2. IP blacklists to block spammers

Another reliable way to stop spammers who only target us is the use of a local IP blacklist on the email server.

3. Set Reverse DNS to block bogus senders

Spamming always starts from a nonexistent email account. Hence, if we set RDNS for our server, we reduce it in major amounts.
Once Reverse DNS Lookup is active, our SMTP verifies that the sender’s IP address matches both the host and domain names that were submitted by the SMTP client in the EHLO/HELO command.

4. Encrypt POP3 and IMAP authentication for privacy concerns

POP3 and IMAP connections were not built with safety in mind. As a result, we may use them without strong authentication. This is a big weakness.
SSL/TLS is the best known and easiest way to implement strong authentication. When securing the mail server, encrypt POP3 and IMAP authentication and use SSL and TLS.

5. Activate SPF to prevent spoofed sources

Sender Policy Framework (SPF) allows domain owners to declare who is allowed to send email in their name. It is to prevent spoofed sender addresses.
When SPF is active, the sending server’s MX record validates before message transmission takes place. Evaluating the email source against the SPF policy of the sender can determine if the email is forged.

6. Set up SMTP authentication to control user access

To protect the server from unauthorized access, we can implement authentication and access control.
For example, SMTP authentication requires users who use our server to obtain permission to send mail by first supplying a username and password.
Our Support Experts highly recommend this when the mail server has a routed IP address.

7. Configure mail relay options carefully to avoid being an Open Relay

All mail servers have this option. With it, we can specify which domains or IP addresses our mail server will relay mail for.
In other words, this specifies for whom our SMTP protocol should forward mail.
However, misconfiguration of the same can harm us because spammers can use our mail server as a gateway to spam others, resulting in blacklisting our IP address.

8. Limit connections to protect our server against DoS attacks

More often we have a lot of connections to a server at a time. To set the limit for the connection to a server, we edit the configuration file. By setting this, we prevent our server from DoS attacks to a great extent.
However, to handle connection limits, check the parameters like the total number of connections, the total number of simultaneous connections, and the maximum connection rate.

9. Enable SURBL to verify message content

SURBL (Spam URI Real-time Block Lists) verify emails on the basis of invalid or malicious links within a message.
This filter helps to protect users from malware and phishing attacks. Not all mail servers will support SURBL. However, if our email server supports it, activate the same.

10. Have at least 2 MX records for failover

A failover configuration is very important for availability. Our Support Techs strongly recommend setting up at least 2 MX records for each domain.
The first one is the primary and the secondary is used if the primary goes down for any reason. This can be done on the DNS Zone level.

11. Implement DKIM (DomainKeys Identified Mail)

The DKIM (DomainKeys Identified Mail) is an email authentication protocol and a TXT type record.
This mechanism is based on encryption, a fingerprint hash, which validates the email so that the receiving mail server identifies the sender.

12. Implement DMARC

DMARC (Domain-based Message Authentication Reporting & Conformance) uses SPF and DKIM protocols to ensure even more security, providing reporting from receivers to senders.
This helps us monitor our domain and improve our mail server protection.

13. Use DNSBL to block malicious emails and domains

DNSBL (Domain Name System Blacklists) are spam blocking lists. It allows us to keep our server free of spam and threats.
The more connections with DNSBL, the better.

[Stuck with any email attack? We'd be happy to assist you. ]


This article covers some tips to secure a mail server.

Email on the internet is sent by the Simple Mail Transfer Protocol (SMTP). Where a mail flow between servers is not encrypted, it could be intercepted by an ISP or government agency and the contents can be read by passive monitoring.
Basically, When emails are sent between two parties, unless BOTH parties use encryption the message is open and can be read by anyone who intercepts it.
Any emails sent to and received from mailboxes that only send cleartext emails should be considered as security liabilities.

Tips on how to secure your mail server:
1. Encryption: When securing your mail server, make sure you are using secure connections. Encrypt POP3 and IMAP authentication and use SSL and TLS.

2. Mail relay configuration: Avoid being an open relay for spammers by specifying which domains/IP addresses your mail server will relay mail for.

3. Connections and default settings: To avoid DoS attacks, limit the number of connection and authentication errors that your systems will accept. Remove unneeded server functionality by disabling any unnecessary default settings. Have a dedicated mail server and move other services like FTP to other servers. Keep total, simultaneous, and maximum connections to your SMTP server limited.

4. Access Control: To protect your server from unauthorized access, implement authentication and access control. For example, SMTP authentication requires users to supply a username and password to be able to send mail from the server. Make sure access to your servers is on a need-to-have basis and is shared with as few people as possible.

5. Abuse prevention: Check DNS-based blacklists (DNSBLs) and reject email from any domains or IPs listed on them. Check Spam URI Real-time Blocklists (SURBL), and reject any messages containing invalid or malicious links.
Also, maintain a local blacklist and block any IP addresses that specifically target you. Employ outbound filtering and use CAPTCHA/reCAPTCHA with your web forms.