Enable Opportunistic TLS in IIS SMTP Service - Tutorial

IIS SMTP Service is a Windows built-in SMTP service. It is widely used to relay email on Windows Server. TLS is a cryptographic protocol designed to provide communications security over a computer network. Most SMTP servers support TLS encryption to create a secure connection for email transmission.

This tutorial gives you a quick overview of how to enable outbound TLS and opportunistic TLS in IIS SMTP Service.

Enable Outbound TLS

To enable Outbound TLS connection in IIS SMTP Server,

  • Open Internet Information Service (IIS) 6.0 Manager;
  • Select a SMTP Virtual Server -> Right Click -> Properties -> Delivery -> Outbound Security -> Check TLS encryption -> Click OK -> Click Apply.
  • After Outbound TLS is enabled, current SMTP server will connect the destination email server using TLS encryption.
Enable outbound TLS in IIS SMTP

However, if recipient’s server doesn’t support TLS encryption, the email will not be successfully delivered.

An ideal solution to above problem is Opportunistic TLS:

  • If remote server supports TLS encryption, IIS SMTP Server connects the remote server using TLS encryption;
  • If remote server doesn’t support TLS encryption, IIS SMTP Server connects the remote server using plain TCP connection (non-encrypted communication).

Opportunistic TLS

IIS SMTP Service doesn’t support opportunistic TLS natively, but we can use an IIS SMTP plugin to enable this function.

How it works:

  • Install TLS Router in IIS SMTP Service.
  • Create Plain TCP Virtual SMTP Server in IIS SMTP Service.

TLS Router checks if the remote server supports TLS before the current SMTP server sends the email to the remote server.

  • If the remote server supports TLS encryption, the current SMTP server to send the email to the remote server directly.
  • If remote server doesn’t support TLS encryption, it will forward the email to the Plain TCP Virtual Server, which in turn will send the email to the remote server using a plain TCP connection.
iis smtp opportunistic tls sink

IIS SMTP Service TLS Router Installation

You can download TLS Router for IIS SMTP Service Installer and install it on your server.

Double click installer file and the installation will be executed automatically. Installer requires IIS SMTP server to be installed. If no IIS SMTP Services detected in your operating system, Setup will be aborted.

After the installation is completed, click “IIS SMTP TLS Router Manager” from “Windows Start menu” -> “All Programs” -> “EA TLS Router for IIS SMTP” to begin the setup.

Important

After the installation is completed, I strongly suggest that you check “IIS Admin” and “Simple Mail Transfer Protocol (SMTP)” in Control Panel -> Administrative Tools -> Services, and check if those services are running, if those service are not running, please start it.

Create a Plain TCP Virtual SMTP Server

Before configurating TLS Router, we need to create a Plain TCP Virtual SMTP Server like this:

  • Open Internet Information Service (IIS) 6.0 Manager;
  • Select your current machine -> Right Click -> New -> SMTP Virtual Server;
  • Input Plain TCP Server or anything as "Name";
  • Select [All Unassigned] in "Select IP address";
  • Input C:\Plain TCP Server or any valid folder in "Select Home Directory";
  • Input PlainTcpServer.local in "Default Domain" and click "Finish".
Create Virtual SMTP server in IIS SMTP Service

After you created PlainTCP SMTP Server, you need to change TCP port to an unused port to prevent conflict with existed SMTP Server.

Right click "Plain TCP Server" in SMTP Server list -> Properties -> General -> Advanced -> Edit -> Change TCP port to 2525 or other port numbers. As this SMTP server does not accept email, you can use any port number except 25 and 587.

Change SMTP server port in IIS SMTP Service

After you changed the TCP port, select Plain TCP Server -> Right Click -> Start to start Plain TCP Server.

Set Fully-Qualified Domain Name (FQDN)

You need to set a valid FQDN for your current SMTP server. Please right click the virtual SMTP server -> Properties -> Delivery -> Advanced -> Fully-qualified domain name -> input a valid FQDN.

Set FQDN for Virtual SMTP server in IIS SMTP Service

It is recommended to set a A record and a SPF record for your FQDN in your DNS server. For example, you set smtp.emailarchitect.net as FQDN, you need to set a A record for smtp.emailarchitect.net in your DNS server, the A record should point to your server’s IP address.

SPF checker uses the SPF record of the sender domain by default. However if your server sends NDR with null sender address <>, SPF checker uses the SPF record of the FQDN instead of the sender domain, so it is recommended to add a SPF record for smtp.emailarchitect.net in your DNS server, which helps the email with null sender pass the SPF check.

If your server IP is 192.168.0.1, the A record and SPF record should be like this:

A record for smtp.emailarchitect.net
192.168.0.1

SPF record for smtp.emailarchitect.net
v=spf1 ip4:192.168.0.1 -all

Configurate TLS Router Rule

Open “IIS SMTP TLS Router Manager” from “Windows Start menu” -> “All Programs” -> “EA TLS Router for IIS SMTP

  • Click TLS to Plain Router -> New Rule;
  • Input rule description;
  • Select available IIS SMTP Servers from the list (Only Outbound TLS enabled servers can be selected).
  • Select Plain TCP Server from Destination Server and click "Save".
Create TLS Router rule

After the above rule is created, if the remote server doesn’t support TLS encryption, TLS Router will forward the email to the Plain TCP Server, and then the Plain TCP Server will send the email to the remote server using a non-encrypted connection.

Certificate Expired Domains

If recipient server SSL certificate has expired, although it supports TLS connection, however, local IIS SMTP service takes it as a temporal error and rejects to relay the message. IIS SMTP service keeps on re-sending the email until email expiration. You will see many "The remote SMTP service rejected the SSL handshake because the certificate has expired" errors in Event Viewer.

You can enable "Or if recipient server SSL certificate has expired" in TLS rule, then TLS router will add domain with certificate expired to a global cache, next time (ususally 15-30 minutes) when local IIS SMTP Service retries to relay the email, TLS router will redirect the email to Plain TCP Server.

Once the domain is added to expired list, subsequent email to the domain will be redirected to Plain TCP Server without delay. You can restart IIS Admin Service to clear the expired domain list in memory.

“The remote SMTP service does not support TLS” warning

If you see “The remote SMTP service does not support TLS” in Event Viewer, that means your sever requires a newer TLS protocol, but remote service only supports old SSL/TLS protocol.

  • You can enable "Or if recipient server SSL certificate has expired" in TLS rule to redirect it to Plain TCP Server.
  • Or you can enable old SSL/TLS protocol in your operating system by following in next section.

Enable TLS 1.0/1.1/1.2 Protocol

You can copy these lines in a file ending with *.reg and Right Click -> Merging to enable TLS 1.0/1.1/1.2.

The following setting will enable TLS 1.0/1.1/1.2 on client side. If remote SMTP server does not support latest TLS protocol, your server can use old SSL/TLS protocol to send email to remote SMTP service.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001

To activate these changes, you need to restart IIS Admin Service or reboot your server .

Bad TLS Domains

If remote recipient server supports TLS connection, but it didn’t deploy certificate correctly or cipher algorithm is weak, and you can find the email is failed to deliver to the recipient server using TLS server.

You can add the domain to "Or if recipient domain is in the following list", the email to the domain will be redirected to Plain TCP Server forcedly by TLS router.

TLS Enabled Domains

If remote recipient’s domain is well-known TLS enabled, you can add the domain to "But never redirect email if recipient domain is in the following list" to increase the performance.

Or if you have set a remote domain with smart host relay in current SMTP Virtual Server, you should add the remote domain to this list to prevent the email redirected to Plain TCP Server.

Inbound TLS enabled domains

TLS Router Plugin Troubleshooting

If TLS Router is not working properly, you can enable full debug log like this:

  • Go to TLS Router manager -> Journal -> change log level to Full Debug -> click Change Log Level.
  • Send a test email to a recipient which doesn’t support TLS encryption.
  • You can find log file in Journal shortly, check log content and see if the email was forwarded to the Plain TCP Server.
  • Change log level back to “Only Error” to reduce the log file size.

Installation Permission

An account with Administrative Privileges is required to run the installer for installing TLS Router on IIS SMTP service.

Verify IIS SMTP Sink Installation

If you installed TLS Router on IIS SMTP Service and created a rule, you can open DOS prompt, and change directory to EA TLS Router installation path\installer and input:

cscript smtpregex.vbs /enum

Press enter.

You will see EA IIS Smtp Tls Router Sink in the output.

iis smtp tls sink

Free Email Support

Not enough? Please contact our technical support team.

Support@EmailArchitect.NET

Remarks

We usually reply emails within 24hours. The reason for getting no response is likely that your SMTP server bounced our reply. In this case, please try to use another email address to contact us. Your Gmail, Hotmail or Office 365 email account is recommended.