aws ses forward email to Externel Mail Server

Amazon Simple Email Service (SES) is cloud based email service that allow to send emails from within any application. Its basically designed to support several email use cases, including transactional, marketing, or mass email communications.

Amazon SES also has support for incoming email but one of the major challenge with incoming SES emails is we need AWS SES to forward email into an external email address.

We need to implement this because AWS SES doesn’t support IMAP/POP3 protocols and IMAP/POP3 are the services supported by most of the email clients. We also need this solution if our website has contact form and we need to get email alerts when our clients contact us using the website contact form.

So in short our goal is AWS SES to forward email into an external email address like any free gmail address which supports IMAP/POP3 protocols. In this post we use Amazon cloud solutions like SES, Amazon S3, and AWS Lambda to create such solution and its heavily inspired from below two blog posts.

https://aws.amazon.com/blogs/messaging-and-targeting/forward-incoming-email-to-an-external-destination/
https://github.com/arithmetric/aws-lambda-ses-forwarder

Below is the Architecture diagram we are following and the work flow.

A external email arrives and Amazon SES handles the incoming email for your domain.
According to Amazon SES receipt rule the incoming message saves in an S3 bucket.
The Amazon SES receipt rule also triggers a Lambda function.
The Lambda function retrieves the message content from S3, and send it back to SES.
Finally Amazon SES sends the message to the forwarding email destination server.

Prerequisites

we already need a verified domain in SES with MX record points to SES provided one. Also if the Amazon SES is in sandbox mode, submit a support request to have it removed.

Create S3 Bucket and Edit Policy

Now lets get started. First create an s3 bucket from amazon s3 bucket console. Make sure you create s3 bucket in same region as your Amazon SES exists. Once created click on s3 bucket name >> Click Permissions tab >> Click Bucket policy. Add below policy to the bucket.

Copy to Clipboard

In the policy, make the following changes:

Replace <bucketName> with the name of your S3 bucket.
Replace <awsAccountId> with your AWS account ID.

Create an IAM Policy and Role

Now we are creating IAM policy and associating it to an IAM role. We assign this newly created role to the Lambda Function we create later.

Go to the IAM console in AWS console >> Click on Policies available in left side >> Click Create Policy >> Switch to JSON tab. Copy paste below content and review and create the policy by giving a name you can identify.

Copy to Clipboard

In the preceding policy, make the following changes:

Replace <bucketName> with the name of the S3 bucket that you created earlier.
Replace <region> with the name of the AWS Region that you created the bucket in. For example us-west-2
Replace <awsAccountId> with your AWS account ID.

Now lets create a new IAM role. For that Click Roles from IAM console >> click Create Role >> Select type of trusted entity as AWS Service >> Choose Lambda from it
>> Click next Permissions >> select the policy that we just created to the new role >> Click review and create by giving a Name for the Role. Make sure the role name you gave is identifiable amount other existing roles.

Create the Lambda Function

Now go to the AWS Lambda console, create a new Python 3.7 function from scratch. For that click Create Function >> Choose runtime as Python 3.7 >> In Choose or create an execution role , select use an existing role >> Select the role that we created for Lambda from drop down menu >> Gave a name and click create.

In the function code editor section gave below code.

Copy to Clipboard

# Copyright 2010-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# This file is licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License. A copy of the
# License is located at
#
# http://aws.amazon.com/apache2.0/
#
# This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
# OF ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.

import os
import boto3
import email
import re
from botocore.exceptions import ClientError
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.application import MIMEApplication

region = os.environ['Region']

def get_message_from_s3(message_id):

incoming_email_bucket = os.environ['MailS3Bucket']
    incoming_email_prefix = os.environ['MailS3Prefix']

if incoming_email_prefix:
        object_path = (incoming_email_prefix + "/" + message_id)
    else:
        object_path = message_id

object_http_path = (f"http://s3.console.aws.amazon.com/s3/object/{incoming_email_bucket}/{object_path}?region={region}")

# Create a new S3 client.
    client_s3 = boto3.client("s3")

# Get the email object from the S3 bucket.
    object_s3 = client_s3.get_object(Bucket=incoming_email_bucket,
        Key=object_path)
    # Read the content of the message.
    file = object_s3['Body'].read()

file_dict = {
        "file": file,
        "path": object_http_path
    }

return file_dict

def create_message(file_dict):

sender = os.environ['MailSender']
    recipient = os.environ['MailRecipient']

separator = ";"

# Parse the email body.
    mailobject = email.message_from_string(file_dict['file'].decode('utf-8'))

# Create a new subject line.
    subject_original = mailobject['Subject']
    subject = "FW: " + subject_original

# The body text of the email.
    body_text = ("The attached message was received from "
              + separator.join(mailobject.get_all('From'))
              + ". This message is archived at " + file_dict['path'])

# The file name to use for the attached message. Uses regex to remove all
    # non-alphanumeric characters, and appends a file extension.
    filename = re.sub('[^0-9a-zA-Z]+', '_', subject_original) + ".eml"

# Create a MIME container.
    msg = MIMEMultipart()
    # Create a MIME text part.
    text_part = MIMEText(body_text, _subtype="html")
    # Attach the text part to the MIME message.
    msg.attach(text_part)

# Add subject, from and to lines.
    msg['Subject'] = subject
    msg['From'] = sender
    msg['To'] = recipient

# Create a new MIME object.
    att = MIMEApplication(file_dict["file"], filename)
    att.add_header("Content-Disposition", 'attachment', filename=filename)

# Attach the file object to the message.
    msg.attach(att)

message = {
        "Source": sender,
        "Destinations": recipient,
        "Data": msg.as_string()
    }

return message

def send_email(message):
    aws_region = os.environ['Region']

# Create a new SES client.
    client_ses = boto3.client('ses', region)

# Send the email.
    try:
        #Provide the contents of the email.
        response = client_ses.send_raw_email(
            Source=message['Source'],
            Destinations=[
                message['Destinations']
            ],
            RawMessage={
                'Data':message['Data']
            }
        )

# Display an error if something goes wrong.
    except ClientError as e:
        output = e.response['Error']['Message']
    else:
        output = "Email sent! Message ID: " + response['MessageId']

return output

def lambda_handler(event, context):
    # Get the unique ID of the message. This corresponds to the name of the file
    # in S3.
    message_id = event['Records'][0]['ses']['mail']['messageId']
    print(f"Received message ID {message_id}")

# Retrieve the file from the S3 bucket.
    file_dict = get_message_from_s3(message_id)

# Create the message.
    message = create_message(file_dict)

# Send the email and print the result.
    result = send_email(message)
    print(result)

Now we need to create the following environment variables for the Lambda function by clicking on Edit button next to Environment variables which is available next to function code in AWS Lambda console.

Key Value
MailS3Bucket The name of the S3 bucket that you created earlier.

MailS3Prefix The path of the folder in the S3 bucket where you will store incoming email. This is the name of the sub-folder where the actual emails resides. If we don’t have any such folder path and we directly keeping the incoming emails in the main bucket folder. We don’t have to define this environment variable. Also in the lambda function code search for incoming_email_prefix variable and set the value as False. In the code it will look like below.

incoming_email_prefix = False

MailSender The incoming SES email address that we need to forward messages. This address has to be verified in SES. If the entire domain is verified, no need to verify the email specifically.

MailRecipient The destination external forwarding email address.

Region The name of the AWS Region where the SES exists.

Also from AWS Lambda Basic settings section, we can increase the code execution time out and the memory, if we are expecting large emails. The basic timeout and memory limit is enough for normal emails.

Create a Receipt Rule Set

Now go the AWS SES console and click on Rule Sets >> Click Create a New Rule Set >> Gave A name for the rule >>

On the Recipients configuration page, add our SES email addresses from which you want to forward email.

On the Actions configuration page, add an S3 action first and then an Lambda action.

For the S3 action: choose the existing S3 bucket we created . Optionally, add an object key prefix. Leave Encrypt Message unchecked and SNS Topic set to [none].

For the Lambda action: Choose the Lambda function we created . Leave Invocation Type set to Event and SNS Topic set to [none].

If you are asked by SES to add permissions to access lambda:InvokeFunction, agree to it.

Once created, it will list under rule set section. Select it and make it as Active one by clicking ” Set as Active rule set ” button.

This concludes the setting up. Now as part of testing, try to send an email to SES email address . Make sure that the email arrives in the S3 bucket. In in 5min normally , the email will forward to remote forwarded email address.

Now one of the main downside of this Lambda function is the the actual email will reach the remote destination as attachment. Not just like a regular forwarded email we usually see.

So we are going to create a new Lambda function and use that Lambda function for forwarding emails. For that go the AWS Lambda console, create a new function

Ensure Runtime is set to Node.js 12.x.

Ensure Handler is set to index.handler.

For Role, choose our old Lambda role itself from the drop down menu.

In the Function code section, copy/paste below Node.js code.

Copy to Clipboard

"use strict";

var AWS = require('aws-sdk');

console.log("AWS Lambda SES Forwarder // @arithmetric // Version 5.0.0");

// Configure the S3 bucket and key prefix for stored raw emails, and the
// mapping of email addresses to forward from and to.
//
// Expected keys/values:
//
// - fromEmail: Forwarded emails will come from this verified address
//
// - subjectPrefix: Forwarded emails subject will contain this prefix
//
// - emailBucket: S3 bucket name where SES stores emails.
//
// - emailKeyPrefix: S3 key name prefix where SES stores email. Include the
//   trailing slash.
//
// - allowPlusSign: Enables support for plus sign suffixes on email addresses.
//   If set to `true`, the username/mailbox part of an email address is parsed
//   to remove anything after a plus sign. For example, an email sent to
//   `example+test@example.com` would be treated as if it was sent to
//   `example@example.com`.
//
// - forwardMapping: Object where the key is the lowercase email address from
//   which to forward and the value is an array of email addresses to which to
//   send the message.
//
//   To match all email addresses on a domain, use a key without the name part
//   of an email address before the "at" symbol (i.e. `@example.com`).
//
//   To match a mailbox name on all domains, use a key without the "at" symbol
//   and domain part of an email address (i.e. `info`).
//
//   To match all email addresses matching no other mapping, use "@" as a key.
var defaultConfig = {
  fromEmail: "noreply@example.com",
  subjectPrefix: "",
  emailBucket: "s3-bucket-name",
  emailKeyPrefix: "emailsPrefix/",
  allowPlusSign: true,
  forwardMapping: {
    "info@example.com": [
      "yourgmailaddress@gmail.com"
    ]
    
  }
};

/**
 * Parses the SES event record provided for the `mail` and `receipients` data.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.parseEvent = function(data) {
  // Validate characteristics of a SES event record.
  if (!data.event ||
      !data.event.hasOwnProperty('Records') ||
      data.event.Records.length !== 1 ||
      !data.event.Records[0].hasOwnProperty('eventSource') ||
      data.event.Records[0].eventSource !== 'aws:ses' ||
      data.event.Records[0].eventVersion !== '1.0') {
    data.log({
      message: "parseEvent() received invalid SES message:",
      level: "error", event: JSON.stringify(data.event)
    });
    return Promise.reject(new Error('Error: Received invalid SES message.'));
  }

data.email = data.event.Records[0].ses.mail;
  data.recipients = data.event.Records[0].ses.receipt.recipients;
  return Promise.resolve(data);
};

/**
 * Transforms the original recipients to the desired forwarded destinations.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.transformRecipients = function(data) {
  var newRecipients = [];
  data.originalRecipients = data.recipients;
  data.recipients.forEach(function(origEmail) {
    var origEmailKey = origEmail.toLowerCase();
    if (data.config.allowPlusSign) {
      origEmailKey = origEmailKey.replace(/\+.*?@/, '@');
    }
    if (data.config.forwardMapping.hasOwnProperty(origEmailKey)) {
      newRecipients = newRecipients.concat(
        data.config.forwardMapping[origEmailKey]);
      data.originalRecipient = origEmail;
    } else {
      var origEmailDomain;
      var origEmailUser;
      var pos = origEmailKey.lastIndexOf("@");
      if (pos === -1) {
        origEmailUser = origEmailKey;
      } else {
        origEmailDomain = origEmailKey.slice(pos);
        origEmailUser = origEmailKey.slice(0, pos);
      }
      if (origEmailDomain &&
          data.config.forwardMapping.hasOwnProperty(origEmailDomain)) {
        newRecipients = newRecipients.concat(
          data.config.forwardMapping[origEmailDomain]);
        data.originalRecipient = origEmail;
      } else if (origEmailUser &&
        data.config.forwardMapping.hasOwnProperty(origEmailUser)) {
        newRecipients = newRecipients.concat(
          data.config.forwardMapping[origEmailUser]);
        data.originalRecipient = origEmail;
      } else if (data.config.forwardMapping.hasOwnProperty("@")) {
        newRecipients = newRecipients.concat(
          data.config.forwardMapping["@"]);
        data.originalRecipient = origEmail;
      }
    }
  });

if (!newRecipients.length) {
    data.log({
      message: "Finishing process. No new recipients found for " +
        "original destinations: " + data.originalRecipients.join(", "),
      level: "info"
    });
    return data.callback();
  }

data.recipients = newRecipients;
  return Promise.resolve(data);
};

/**
 * Fetches the message data from S3.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.fetchMessage = function(data) {
  // Copying email object to ensure read permission
  data.log({
    level: "info",
    message: "Fetching email at s3://" + data.config.emailBucket + '/' +
      data.config.emailKeyPrefix + data.email.messageId
  });
  return new Promise(function(resolve, reject) {
    data.s3.copyObject({
      Bucket: data.config.emailBucket,
      CopySource: data.config.emailBucket + '/' + data.config.emailKeyPrefix +
        data.email.messageId,
      Key: data.config.emailKeyPrefix + data.email.messageId,
      ACL: 'private',
      ContentType: 'text/plain',
      StorageClass: 'STANDARD'
    }, function(err) {
      if (err) {
        data.log({
          level: "error",
          message: "copyObject() returned error:",
          error: err,
          stack: err.stack
        });
        return reject(
          new Error("Error: Could not make readable copy of email."));
      }

// Load the raw email from S3
      data.s3.getObject({
        Bucket: data.config.emailBucket,
        Key: data.config.emailKeyPrefix + data.email.messageId
      }, function(err, result) {
        if (err) {
          data.log({
            level: "error",
            message: "getObject() returned error:",
            error: err,
            stack: err.stack
          });
          return reject(
            new Error("Error: Failed to load message body from S3."));
        }
        data.emailData = result.Body.toString();
        return resolve(data);
      });
    });
  });
};

/**
 * Processes the message data, making updates to recipients and other headers
 * before forwarding message.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.processMessage = function(data) {
  var match = data.emailData.match(/^((?:.+\r?\n)*)(\r?\n(?:.*\s+)*)/m);
  var header = match && match[1] ? match[1] : data.emailData;
  var body = match && match[2] ? match[2] : '';

// Add "Reply-To:" with the "From" address if it doesn't already exists
  if (!/^reply-to:[\t ]?/mi.test(header)) {
    match = header.match(/^from:[\t ]?(.*(?:\r?\n\s+.*)*\r?\n)/mi);
    var from = match && match[1] ? match[1] : '';
    if (from) {
      header = header + 'Reply-To: ' + from;
      data.log({
        level: "info",
        message: "Added Reply-To address of: " + from
      });
    } else {
      data.log({
        level: "info",
        message: "Reply-To address not added because From address was not " +
          "properly extracted."
      });
    }
  }

// SES does not allow sending messages from an unverified address,
  // so replace the message's "From:" header with the original
  // recipient (which is a verified domain)
  header = header.replace(
    /^from:[\t ]?(.*(?:\r?\n\s+.*)*)/mgi,
    function(match, from) {
      var fromText;
      if (data.config.fromEmail) {
        fromText = 'From: ' + from.replace(/<(.*)>/, '').trim() +
        ' <' + data.config.fromEmail + '>';
      } else {
        fromText = 'From: ' + from.replace('<', 'at ').replace('>', '') +
        ' <' + data.originalRecipient + '>';
      }
      return fromText;
    });

// Add a prefix to the Subject
  if (data.config.subjectPrefix) {
    header = header.replace(
      /^subject:[\t ]?(.*)/mgi,
      function(match, subject) {
        return 'Subject: ' + data.config.subjectPrefix + subject;
      });
  }

// Replace original 'To' header with a manually defined one
  if (data.config.toEmail) {
    header = header.replace(/^to:[\t ]?(.*)/mgi, () => 'To: ' + data.config.toEmail);
  }

// Remove the Return-Path header.
  header = header.replace(/^return-path:[\t ]?(.*)\r?\n/mgi, '');

// Remove Sender header.
  header = header.replace(/^sender:[\t ]?(.*)\r?\n/mgi, '');

// Remove Message-ID header.
  header = header.replace(/^message-id:[\t ]?(.*)\r?\n/mgi, '');

// Remove all DKIM-Signature headers to prevent triggering an
  // "InvalidParameterValue: Duplicate header 'DKIM-Signature'" error.
  // These signatures will likely be invalid anyways, since the From
  // header was modified.
  header = header.replace(/^dkim-signature:[\t ]?.*\r?\n(\s+.*\r?\n)*/mgi, '');

data.emailData = header + body;
  return Promise.resolve(data);
};

/**
 * Send email using the SES sendRawEmail command.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
exports.sendMessage = function(data) {
  var params = {
    Destinations: data.recipients,
    Source: data.originalRecipient,
    RawMessage: {
      Data: data.emailData
    }
  };
  data.log({
    level: "info",
    message: "sendMessage: Sending email via SES. Original recipients: " +
      data.originalRecipients.join(", ") + ". Transformed recipients: " +
      data.recipients.join(", ") + "."
  });
  return new Promise(function(resolve, reject) {
    data.ses.sendRawEmail(params, function(err, result) {
      if (err) {
        data.log({
          level: "error",
          message: "sendRawEmail() returned error.",
          error: err,
          stack: err.stack
        });
        return reject(new Error('Error: Email sending failed.'));
      }
      data.log({
        level: "info",
        message: "sendRawEmail() successful.",
        result: result
      });
      resolve(data);
    });
  });
};

/**
 * Handler function to be invoked by AWS Lambda with an inbound SES email as
 * the event.
 *
 * @param {object} event - Lambda event from inbound email received by AWS SES.
 * @param {object} context - Lambda context object.
 * @param {object} callback - Lambda callback object.
 * @param {object} overrides - Overrides for the default data, including the
 * configuration, SES object, and S3 object.
 */
exports.handler = function(event, context, callback, overrides) {
  var steps = overrides && overrides.steps ? overrides.steps :
    [
      exports.parseEvent,
      exports.transformRecipients,
      exports.fetchMessage,
      exports.processMessage,
      exports.sendMessage
    ];
  var data = {
    event: event,
    callback: callback,
    context: context,
    config: overrides && overrides.config ? overrides.config : defaultConfig,
    log: overrides && overrides.log ? overrides.log : console.log,
    ses: overrides && overrides.ses ? overrides.ses : new AWS.SES(),
    s3: overrides && overrides.s3 ?
      overrides.s3 : new AWS.S3({signatureVersion: 'v4'})
  };
  Promise.series(steps, data)
    .then(function(data) {
      data.log({
        level: "info",
        message: "Process finished successfully."
      });
      return data.callback();
    })
    .catch(function(err) {
      data.log({
        level: "error",
        message: "Step returned error: " + err.message,
        error: err,
        stack: err.stack
      });
      return data.callback(new Error("Error: Step returned error."));
    });
};

Promise.series = function(promises, initValue) {
  return promises.reduce(function(chain, promise) {
    if (typeof promise !== 'function') {
      return Promise.reject(new Error("Error: Invalid promise item: " +
        promise));
    }
    return chain.then(promise);
  }, Promise.resolve(initValue));
};

Now lets set Environment variables in AWS Lambda console for below two variables.

emailBucket : S3 bucket name where email stores.

fromEmail : Set a SES email address like noreply@yourdomain.com. This is the from email address which will display in the destination forwarded email box.

Also correct the variables defined in Lambda code according to our values which is under the defaultConfig section. the values we need to correct is.

fromEmail , details are already mentioned above.

emailBucket , details are already mentioned above.

emailKeyPrefix , S3 key name prefix where SES stores email. Include the trailing slash. If we don’t have any such subfolder which stored emails in s3 bucket and we are directly keeping the emails in a single folder. Just leave it as blank and it code line will look like below.

subjectPrefix: “”,

forwardMapping: , in this variable the first email address is our SES forwarding email address and the second one will be our external destination email box like our gmail address.

Now go to the SES Rule Sets we created earlier and edit it and change the lambda function into the new one we created. Test the working again.

In my case it didn’t worked in first place and no emails forwarded from s3 bucket. So I started troubleshooting. The first thing you need to is, go the AWS CloudWatch console

Click CloudWatch Logs > Log groups >> Click on our AWS Lambda function name log >> Click on latest log if there is multiple one.

Look for errors and I found below error.

level: ‘info’, message: ‘Fetching email at s3:

level: ‘error’, message: ‘copyObject() returned error:’, error: AccessDenied: Access Denied at

INFO { level: ‘error’, message: ‘Step returned error: Error: Could not make readable copy of email.’,

Basically it means that AWS Lambda function can’t read emails from s3 bucket. In order to resolve the issue, I go the IAM console >> Click roles >> choose our role created for SES forward >> Click attached policy >> add below entry under “ses:SendRawEmail”, as next line.

“s3:PutObject”

Refer aws articles if you got any syntax errors.

Once that done. Try to test our email forwarding and at this time its worked successfully and we got email in remote email inbox like regular forwarded email.

After a while we got a failure message at forwarding email address with subject as “FW: Delivery Status Notification (Failure)“. On further lookup from email header, we noticed below error code.

Diagnostic-Code: Amazon SES did not send the message to this address because it is on the suppression list for your account. For more information about removing addresses from the suppression list, see the Amazon SES Developer Guide at https://docs.aws.amazon.com/ses/latest/DeveloperGuide/sending-email-suppression-list.html
Status: 5.1.1

In order to resolve this, issue below two commands. The first one show the email address listed in account level suppression list and second is to remove that email address from the list.

Copy to Clipboard