Encrypt Email (S/MIME + RSA-OAEP) in Exchange Server and IIS SMTP Service


Email Encryption protects email content from exposure to inappropriate recipients. Encrypting email doesn’t require the sender certificate but the certificate and the public key of the recipient.

For example:

How S/MIME work

Therefore, you must receive a digital signed email from other people ( Most email clients such as outlook, outlook express will add the certificate to the Other People Storage automatically once an digital signed email is received) before you can return an encrypted email to this people.

To encrypt email with EA S/MIME in your IIS SMTP Service or Exchange Server, you need to export the recipient’s certificate first.


Encrypting Email (S/MIME) in Email Client

Most popular email clients support email encryption. To enable email encryption in your email client, you need the certificate of the email recipient. You can refer to the documentation of your email client for the settings.

This tutorial introduces how to use S/MIME Plugin to encrypt email in IIS SMTP Service or Exchange 2003 automatically without an email client. If you send emails from an email tool that doesn’t support email encryption, this plugin can help to encrypt email on server side.


How S/MIME Plugin Works?

In IIS SMTP Service or Exchange 2003, S/MIME Plugin works as a SMTP event sink; in Exchange 2007/2010/2013/2016, it works as a transport agent. It encrypts an email with a digital certificate based on pre-defined rules.

An encryption rule can be defined for a single recipient or multiple recipients.

How S/MIME plugin works with email encryption in IIS SMTP Service

Import, Export and Generate Test Certificate

To view/import/export certificate in Current User Store and Local Machine Store, you can open the Certificates MMC Snap-in like this:

Certificates MMC Snap-in

Now you can view all certificates in current user store and local machine store.

As S/MIME Plug can’t access certificates from the Current User Store, if your certificate is stored in the Current User Store, you must export it from the Current User Store.

Export Encryption Certificate from Certificate Store

You can export certificate from Current User Store like this:

Export certificate to .cer file

You can use the exported certificate file as Certificate Source in S/MIME Plugin.

Note

The certificate without a private key is used for email encryption only.

Pfx, Cert and Sst file

Import Certificate to Certificate Store

You can import a certificate file to Certificate Machine Store like this:

Import certificate to certificate machine store

After certificate file was imported to Certificate Machine Store, you can use Certificate Machine Store as Certificate Source instead of certificate file.

Generate Self-Signed Root Certificate

If the email digital certificate is not issued by third-party authorities, the email client will prompt an error of “Certificate is not issued by a trusted publisher”.

However, for internal enterprise usage or test purpose, you can create a self-signed root certificate and issue this digital certificate to internal users by New-SelfSignedCertificate PowerShell cmdlet.

Note

If you have an existing certificate from third-party authorities, you can skip this section.

Note

New-SelfSignedCertificate requires PowerShell 3.0 on Windows 10, Windows 8/8.1 and Windows Server 2012/2012 R2/2016.

You can create a root certificate like this:

New-SelfSignedCertificate -Type Custom -HashAlgorithm sha256 -KeyLength 2048 -KeyUsage CRLSign, CertSign, KeyAgreement, DataEncipherment, KeyEncipherment, DigitalSignature -KeyAlgorithm RSA -Subject "CN=Test Root V1" -CertStoreLocation "cert:\CurrentUser\My" -NotAfter (get-date).AddDays(7300)
			

Then the new root certificate can be found in Certificates MMC Snap-in, select Certificates - Current User -> Personal -> Certificates.

Test root certificate

Now you can issue the certificate to users from root certificate like this:

Generate Email Certificate Issued by Root Certificate

Here is an example to issue the certificate from the root certificate generated in the previous section.

Set-Location -path cert:\CurrentUser\My
$signer = Get-Childitem -DnsName "Test Root V1"
New-SelfSignedCertificate -HashAlgorithm sha256 -Type Custom -Subject "E=test@emailarchitect.net,CN=SMIME Tester 2048" -TextExtension @("2.5.29.37={text}1.3.6.1.5.5.7.3.4","2.5.29.17={text}email=test@emailarchitect.net&upn=test@emailarchitect.net") -KeyLength 2048 -KeyAlgorithm RSA -SmimeCapabilities -CertStoreLocation "Cert:\CurrentUser\My" -NotAfter (get-date).AddDays(365) -Signer $signer			

Then the new email certificate can be found in Certificates MMC Snap-in, select Certificates - Current User -> Personal -> Certificates.

Test root certificate

You can export it together with a private key as a pfx file and use it in Certificate Source. Alternatively, you can install a pfx file to Certificates (Local Computer) -> Personal -> Certificates and use Machine Store as Certificate Source.

Install Root Certificate to Client Machine

If you used the above certificate to sign emails, email client would give an error about Certificate is not issued by a trusted publisher. You can install the root certificate to client machines like this to avoid the error.

Now you can export the root certificate to a cert file and install it on Trusted Root Certification Authorities:

Export test root certificate

Install it to Local\Computer\Trusted Root Certification Authorities

Install it to Current User\Trusted Root Certification Authorities

After you installed the root certificate to Trusted Root Certification Authorities, user certificate chain becomes valid now.

User certificate chain

Create a Rule for Email Encryption

Let’s create a test email encryption rule for a recipient like this:

Create email encryption rule in Exchange Server

Important

IIS SMTP Service is running as Local System user, and Exchange Transport Agent is running as Network Service user, so do not put the certificate file to Desktop or user-dependent folder, S/MIME Plugin doesn’t have permission to read certificate files in these locations.

After rule is saved, click Test, current rule will be executed with a virtual email message, you can find the result in test dialog box.

If Test all rules is checked, all rules defined on current machine will be executed with a virtual email message.

Test email encrypting in Exchange Server

If the test result looks good, you can send a test email to that recipient, and validate if the received email was encrypted.


Recipient Address Pattern and Wildcard

A single email address or email address with wildcard (* and ?) can be used in Recipient Address. It indicates current rule is only enabled for matched recipient address.

An encryption certificate can be only used for a specific email address. For example, you cannot use test@emailarchitect.net's certificate to encrypt email to support@emailarchitect.net, email client would prompt an error due to address not match.

Therefore, if you used wildcard (* and ?) in recipient address, you should specify a Certificate Source that contains multiple certificates, and set Certificate Match Rule to Match certificate by recipient email address, to enable S/MIME Plugin to find the correct certificate to encrypt the email.

Note

If no certificate is identified for all recipients, the message won’t be encrypted.


Certificate Source and Certificate Match Rule

Certificate Source

Machine store, pfx file and sst file can contain multiple certificates, while cert file can only contain one certificate. Please refer to Import, Export and Generate Test Certificate to learn how to import/export certificate from certificate store.

Certificate Match Rule


Email Encryption Algorithm

Rc2, TripleDes, Aes-128, Aes-192 and Aes-256 are supported. For security reason, Aes-128, Aes-192 or Aes-256 is recommended.


AES with RSA-OAEP Hash Encryption Padding Algorithm

To comply with EDIFACT in Germany/EUROPE, please select sha256 or higher SHA algorithm in OAEP hash algorithm option and use AES-128/192/256 encryption algorithm.


Encrypt Tnef Message

Tnef is an internal email format in Exchange Server. If this option is checked, Tnef messages will first be converted to RFC822/message format by S/MIME Plugin, and then be encrypted by the specified certificate.

See Also

Email Disclaimer
Digital Signature
Journal and Troubleshooting
Appendix - Set up DomainKeys/DKIM
Appendix - Set up SPF record in DNS server

Online Tutorials:

Add Disclaimer or Signature with Embedded Images in Exchange Server 2007/2010/2013/2016/2019 - Tutorial
Add Digital Signature (S/MIME) to Email in Exchange Server 2007/2010/2013/2016/2019 - Tutorial
Encrypt Email (S/MIME) in Exchange Server 2007/2010/2013/2016/2019 - Tutorial

Add Disclaimer or Signature with Embedded Images in IIS SMTP Service and Exchange Server 2003 - Tutorial
Add Digital Signature (S/MIME) to Email in IIS SMTP Service and Exchange Server 2003 - Tutorial
Encrypt Email (S/MIME) in IIS SMTP Service and Exchange Server 2003 - Tutorial