JangoSMTP: The Stand-Alone SMTP Service with open tracking, click tracking, DKIM signing, and more.
JangoMail: The SMTP Service combined with email broadcast service.
Latest features and tips from JangoSMTP, a cutting-edge transactional email service with tracking.
Friday, December 11, 2009
SMTP service now supports large attachments
The JangoMail SMTP relay service now supports sending large file attachments in email. Before, there was a limit of approximately 2 MB in total file attachments. Now, there is NO theoretical limit, so the SMTP service can handle file attachments as large as 10 or 20 MB. Of course, email may not be the best medium to transmit such large files, but it is possible with our system.
JangoSMTP: The Stand-Alone SMTP Service with open tracking, click tracking, DKIM signing, and more.
JangoMail: The SMTP Service combined with email broadcast service.
JangoSMTP: The Stand-Alone SMTP Service with open tracking, click tracking, DKIM signing, and more.
JangoMail: The SMTP Service combined with email broadcast service.
Monday, November 23, 2009
Error handling for the SMTP Relay Service
Overview:
We've launched a new error handling system for the SMTP relay system. There are a number of reasons why a message sent through the SMTP relay may result in an error. For example:
1. If the recipient address is on your account's bounce list, then a "bounce" error will be thrown.
2. If the recipient address is on your account's unsubscribe list, then an "unsubscribe" error will be thrown.
3. If your account is over its sending limits, a "Over Account Limit" error will be thrown.
4. If a badly formed email message is passed through the relay, a "parsing" error will be thrown.
These errors are only detected after a message is received by the SMTP relay, so the error will NOT appear during the SMTP conversation with relay.jangosmtp.net.
When an error is detected, the error notification can be emailed to the Account Manager and/or the From Address on the email that resulted in the error. Additionally, the error is available for viewing under Reporting in the Transactional Emails section.
To set your error handling preferences, go to Settings --> SMTP Relay and set the two relevant checkboxes appropriately.
Recommendations:
1. If you know that you regularly attempt to relay emails to addresses on your Unsubscribe and Bounce lists, you may wish to turn off the email notifications for errors entirely, as this could result in a large volume of emails to you.
2. To decide whether or not the error should also be sent to the From Address, consider what type of transactional emails you're sending through the SMTP relay. If the From Address is an address at your organization, then it's probably okay for the From Address to also receive errors. But, if you're sending social networking "You have a new message from John" type emails, where the From Address is that of one of your users, you probably don't want your users to receive error notifications from JangoMail. In this case, uncheck the box to send errors to the From Address.
We've launched a new error handling system for the SMTP relay system. There are a number of reasons why a message sent through the SMTP relay may result in an error. For example:
1. If the recipient address is on your account's bounce list, then a "bounce" error will be thrown.
2. If the recipient address is on your account's unsubscribe list, then an "unsubscribe" error will be thrown.
3. If your account is over its sending limits, a "Over Account Limit" error will be thrown.
4. If a badly formed email message is passed through the relay, a "parsing" error will be thrown.
These errors are only detected after a message is received by the SMTP relay, so the error will NOT appear during the SMTP conversation with relay.jangosmtp.net.
When an error is detected, the error notification can be emailed to the Account Manager and/or the From Address on the email that resulted in the error. Additionally, the error is available for viewing under Reporting in the Transactional Emails section.
To set your error handling preferences, go to Settings --> SMTP Relay and set the two relevant checkboxes appropriately.
Recommendations:
1. If you know that you regularly attempt to relay emails to addresses on your Unsubscribe and Bounce lists, you may wish to turn off the email notifications for errors entirely, as this could result in a large volume of emails to you.
2. To decide whether or not the error should also be sent to the From Address, consider what type of transactional emails you're sending through the SMTP relay. If the From Address is an address at your organization, then it's probably okay for the From Address to also receive errors. But, if you're sending social networking "You have a new message from John" type emails, where the From Address is that of one of your users, you probably don't want your users to receive error notifications from JangoMail. In this case, uncheck the box to send errors to the From Address.
Friday, November 6, 2009
How we scaled up the JangoSMTP service to accommodate the ING New York City Marathon
Overview of the New York City Marathon and its Email Alerts
It was a sunny mid-September day when my head of sales informed me that the systems integrator for the 2009 New York City Marathon was looking at our SMTP service for the delivery of alert emails on runners during the marathon. Given that our SMTP relay service was relatively new at the time, we saw this as an opportunity to demonstrate the powerful tracking features and performance capabilities of our service.
Having tried other email vendors and their own internal systems in the past, the marathon was looking to avoid past issues, such as large delays in the delivery of the email alerts and email blocking issues with consumer ISPs like GMail and Hotmail.
The marathon had about 42,000 total runners. Friends and family members of each runner could "subscribe" to a runner, such that whenever the runner reached a checkpoint during the marathon, an email alert would be sent to the subscribers of that runner. This would be done electronically via a Chronotrack D-Tag. For more information on the electronic tracking of runners, see http://www.nycmarathon.org/race_scoring.htm.
My team was told to expect anywhere from 400,000 to 2,000,000 email alerts to be sent out during the race. That represents an average of 10 to 50 email alerts per runner, depending on how many individuals subscribe to a particular runner.
The Speed Problem
The JangoSMTP service has been a single-node service since its launch earlier in 2009. It was serviced by one single fault-tolerant, RAID-based, multiple-CPU, high memory Windows server located at relay.jangosmtp.net. The SMTP service works by receiving email at relay.jangosmtp.net, and then passing emails along to one of JangoMail's 40 outbound SMTP senders for delivery to the final recipient. The relay.jangosmtp.net server could receive an unlimited amount of emails and saturate our upstream Internet provider's bandwidth, but it could only process, add tracking, and transmit to the outbound SMTP senders over the Internet at a rate of 250 emails/minute. I therefore calculated:
250 emails/minute x 60 = 15,000 emails/hour
Based on past marathon results, I estimated that the fastest runners would complete the race in about 2 hours, and the slowest runners would complete the race in about 6 hours. However, the marathon would be initiated in 3 waves of 14,000 runners each, distributed over an hour the morning of the marathon. So 6 hours of running time, plus an added hour for the start of the last wave, meant 7 hours of sending email.
7 hours x 15,000 emails/hour = 105,000 emails over 7 hours
Uh oh. We weren't nearly fast enough. Even at the minimum expected volume, we weren't fast enough by a factor of 4. And at the maximum expected volume, we only had 5% of the needed capacity.
The two bottlenecks were 1) the processing of an email message, meaning the dis-assembly and re-assembly to determine what user it belonged to and add tracking mechanisms, and 2) transmitting the email to a SMTP sender. Point #2 warrants more explanation. While there is no bottleneck for relay.jangosmtp.net to receive emails from the outside world, there can be a bottleneck for relay.jangosmtp.net to transmit emails, since relay.jangosmtp.net must transmit the message to separate SMTP sender located on separate networks in separate data centers. This transmission happens over the Internet, and based on where the SMTP sender is located and the routing to it, speed can fluctuate.
Time to Scale Up
Load Balancing
We needed to scale up, and scale up fast. The initial plan was to order four more servers, and have them each serve as an additional SMTP receiver and processor for relay.jangosmtp.net. The JangoMail architecture does not employ the use of appliance-based load balancers, so I decided to handle the load balancing via our Domain Name System (DNS).
We created multiple DNS "A" records for relay.jangosmtp.net, each with a Time to Live (TTL) of 60 seconds. Five "A" records were created in total, each with a different IP address. The first of the five was the original IP for relay.jangosmtp.net, and the other four were for the four additional servers we commissioned.
By keeping the TTL at a short 60 seconds, I could make certain that each of the five servers would receive an equal load of email every minute. And since not all DNS servers on the net respect TTLs and sometimes do their own caching, I confirmed with our tech contact at the marathon that their systems would NOT cache the IPs for relay.jangosmtp.net beyond the designated 60 second TTLs.
SQL Query Caching
Every email that arrives at relay.jangosmtp.net is disassembled, tracked, re-assembled, and then passed to an outbound SMTP server for actual sending. In order to determine what email belongs to what user, and what tracking options each user has selected, the originating IP address of each email message is looked up against an IP Address table, and then once the user is determined from the IP address, the UserAccounts table in the core database is queried to determine what tracking/DomainKeys options the user has selected. These two queries combined took anywhere from 0.05 seconds to 0.2 seconds, depending on the load on the database at the time.
We shaved this time down to 0.001 seconds by caching the results of these queries and refreshing every five minutes.
Multi-threading
Our custom SMTP architecture is a multi-threaded model, allowing for the simultaneous processing and delivery of emails across user accounts. We had been informed ahead of time that the marathon would trigger email alerts from two originating IP addresses. We therefore configured two dedicated processing threads on each of the 8 servers. This gave us 16 total processing threads, and also isolated the processing/delivery of the marathon's emails from our other clients' emails during the race.
Distributed Transactional Senders
The SMTP service is part of JangoMail's transactional email platform, which also includes the SendTransactionalEmail API method. All transactional emails are sent through the email sender that is assigned for a particular user. For fault-tolerance purposes, every user has a list of transactional email senders assigned to it, such that if the first email sender is unavailable or offline, the email is passed to the second, and to the third, and so on, until the email is successfully transmitted. This approach was great for fault-tolerance and redundancy, but not for scaleability. If the first server in the list was online and available, then it would receive all the transactional emails for that account.
I therefore decided to add an internally controlled user-level setting option to randomize the list of senders. Now, if a user's list of transactional senders included:
Sender1, Sender2, Sender3, Sender4
Now the relay.jangosmtp.net servers would farm out the emails for delivery to any sender in the user's list at random, ensuring that as many transactional senders as were assigned would receive an equal load of emails to deliver, rather than having the first available sender do all the delivery. This also aided in resolving the bandwidth bottleneck with the SMTP senders mentioned earlier.
Still not fast enough - need 3 more servers
Given our SQL query optimizations, multi-threading, additional servers, the timing now looked like:
400 emails/minute/server x 5 servers = 2,000 emails/minute x 60 minutes = 120,000 emails/hour.
120,000 emails/hour x 7 hours = 840,000 emails over 7 hours
However, if the load was over 1 million emails, there would still be a delay. I decided to annex 3 additional servers that are already a part of the JangoMail network but weren't active on port 25, and turn them into 3 additional SMTP receivers. There were now 8 total DNS "A" records for relay.jangosmtp.net.
With just 48 hours before the marathon, we were informed that the expected outbound volume, based on the number of subscribers so far, would be about 750,000 emails.
And just in case all else failed...
While we've always believed in the performance and reliability of our code and architecture, we decided that for a project of this caliber, a backup plan was necessary in case our custom SMTP architecture was unable to perform. The JangoSMTP custom architecture is what allows emails that pass through the SMTP relay to be open and click-tracked, and stored in a database, so that SMTP logs can be viewed and reports can be generated based on open and click timing, domains, and geo-tracking reports based on IP addresses. However, the primary issue for the marathon, was to ensure the emails were delivered, and delivered on time. Therefore, a backup plan was put into place that could guarantee the emails would be delivered, even if we had to eliminate the tracking and logging based on our own custom architecture. If we found that JangoSMTP could not handle the load, we would replace the instances of the JangoSMTP receiver with Microsoft's built-in SMTP service (part of Internet Information Services), such that emails would be received and delivered to the final recipient, without any processing needed in between.
In order to isolate the marathon from our other clients in case this backup plan had to be put in place, we setup the domain marathon.jangosmtp.net, which mimiced the 8 A records for relay.jangosmtp.net, and we asked the system integrators for the marathon to connect to marathon.jangosmtp.net instead of relay.jangosmtp.net. This would allow us to re-direct just their email on the day of the race if needed.
The Day of the Race - November 1, 2009
I awoke at 7:30 AM EST on that Sunday, after having been out the night prior for Halloween. The emails would begin trickling in at 8:30 AM EST for runner check-ins, and the first wave of the race was set to begin at 9:40 AM EST.
I was watching the race live on TV, while at the same time monitoring the traffic flows across our 8 instances of relay.jangosmtp.net. Emails were being received, processed, and sent quickly, and there was no backlog...until about 12:25 PM EST.
All three waves had been released, and runners from all three waves were triggering a massive volume of email alerts. From 12:25 PM EST to about 12:35 PM EST, there was a 4-5 minute delay with final delivery. Thankfully, the backlog period only lasted 10 minutes. It makes sense that approximately 3 hours after the release of the first wave, that the highest volume of email was passing through, since the greatest number of runners would still be running around this time.
Additionally, at about 1:00 PM EST, we discovered that comcast.net and att.net/bellsouth.net domains were blocking one of the IPs from which marathon email was sending. While we did have a mechanism by which domain-specific routes could be used, such that we could enable all comcast.net/att.net/bellsouth.net email to go through one specific non-blocked IP address, the complexity of the randomization system we had added to accomodate the bandwidth bottleneck rendered this mechanism non-functional. I called our lead developer, who was on call, asked him to make a change to the sender-determination algorithm, and re-deployed our code across all 8 SMTP server instances. We were now able to route all comcast.net/att.net/bellsouth.net email through a separate non-blocked SMTP sender. In the end, the marathon alert emails had less than a 0.3% blocking rate.
After 1:00 PM EST, no further email deliverability or backlog issues ensued.
Conclusion
We were thrilled that we were able to pull this off for the 2009 ING New York City Marathon. Even post-marathon, we continue to make performance and feature enhancements to our transactional email platform. If you have an important project for which sending email is critical, please get in touch with us, and we'll work as hard for you as we did for the marathon.
It was a sunny mid-September day when my head of sales informed me that the systems integrator for the 2009 New York City Marathon was looking at our SMTP service for the delivery of alert emails on runners during the marathon. Given that our SMTP relay service was relatively new at the time, we saw this as an opportunity to demonstrate the powerful tracking features and performance capabilities of our service.
Having tried other email vendors and their own internal systems in the past, the marathon was looking to avoid past issues, such as large delays in the delivery of the email alerts and email blocking issues with consumer ISPs like GMail and Hotmail.
The marathon had about 42,000 total runners. Friends and family members of each runner could "subscribe" to a runner, such that whenever the runner reached a checkpoint during the marathon, an email alert would be sent to the subscribers of that runner. This would be done electronically via a Chronotrack D-Tag. For more information on the electronic tracking of runners, see http://www.nycmarathon.org/race_scoring.htm.
My team was told to expect anywhere from 400,000 to 2,000,000 email alerts to be sent out during the race. That represents an average of 10 to 50 email alerts per runner, depending on how many individuals subscribe to a particular runner.
The Speed Problem
The JangoSMTP service has been a single-node service since its launch earlier in 2009. It was serviced by one single fault-tolerant, RAID-based, multiple-CPU, high memory Windows server located at relay.jangosmtp.net. The SMTP service works by receiving email at relay.jangosmtp.net, and then passing emails along to one of JangoMail's 40 outbound SMTP senders for delivery to the final recipient. The relay.jangosmtp.net server could receive an unlimited amount of emails and saturate our upstream Internet provider's bandwidth, but it could only process, add tracking, and transmit to the outbound SMTP senders over the Internet at a rate of 250 emails/minute. I therefore calculated:
250 emails/minute x 60 = 15,000 emails/hour
Based on past marathon results, I estimated that the fastest runners would complete the race in about 2 hours, and the slowest runners would complete the race in about 6 hours. However, the marathon would be initiated in 3 waves of 14,000 runners each, distributed over an hour the morning of the marathon. So 6 hours of running time, plus an added hour for the start of the last wave, meant 7 hours of sending email.
7 hours x 15,000 emails/hour = 105,000 emails over 7 hours
Uh oh. We weren't nearly fast enough. Even at the minimum expected volume, we weren't fast enough by a factor of 4. And at the maximum expected volume, we only had 5% of the needed capacity.
The two bottlenecks were 1) the processing of an email message, meaning the dis-assembly and re-assembly to determine what user it belonged to and add tracking mechanisms, and 2) transmitting the email to a SMTP sender. Point #2 warrants more explanation. While there is no bottleneck for relay.jangosmtp.net to receive emails from the outside world, there can be a bottleneck for relay.jangosmtp.net to transmit emails, since relay.jangosmtp.net must transmit the message to separate SMTP sender located on separate networks in separate data centers. This transmission happens over the Internet, and based on where the SMTP sender is located and the routing to it, speed can fluctuate.
Time to Scale Up
Load Balancing
We needed to scale up, and scale up fast. The initial plan was to order four more servers, and have them each serve as an additional SMTP receiver and processor for relay.jangosmtp.net. The JangoMail architecture does not employ the use of appliance-based load balancers, so I decided to handle the load balancing via our Domain Name System (DNS).
We created multiple DNS "A" records for relay.jangosmtp.net, each with a Time to Live (TTL) of 60 seconds. Five "A" records were created in total, each with a different IP address. The first of the five was the original IP for relay.jangosmtp.net, and the other four were for the four additional servers we commissioned.
By keeping the TTL at a short 60 seconds, I could make certain that each of the five servers would receive an equal load of email every minute. And since not all DNS servers on the net respect TTLs and sometimes do their own caching, I confirmed with our tech contact at the marathon that their systems would NOT cache the IPs for relay.jangosmtp.net beyond the designated 60 second TTLs.
SQL Query Caching
Every email that arrives at relay.jangosmtp.net is disassembled, tracked, re-assembled, and then passed to an outbound SMTP server for actual sending. In order to determine what email belongs to what user, and what tracking options each user has selected, the originating IP address of each email message is looked up against an IP Address table, and then once the user is determined from the IP address, the UserAccounts table in the core database is queried to determine what tracking/DomainKeys options the user has selected. These two queries combined took anywhere from 0.05 seconds to 0.2 seconds, depending on the load on the database at the time.
We shaved this time down to 0.001 seconds by caching the results of these queries and refreshing every five minutes.
Multi-threading
Our custom SMTP architecture is a multi-threaded model, allowing for the simultaneous processing and delivery of emails across user accounts. We had been informed ahead of time that the marathon would trigger email alerts from two originating IP addresses. We therefore configured two dedicated processing threads on each of the 8 servers. This gave us 16 total processing threads, and also isolated the processing/delivery of the marathon's emails from our other clients' emails during the race.
Distributed Transactional Senders
The SMTP service is part of JangoMail's transactional email platform, which also includes the SendTransactionalEmail API method. All transactional emails are sent through the email sender that is assigned for a particular user. For fault-tolerance purposes, every user has a list of transactional email senders assigned to it, such that if the first email sender is unavailable or offline, the email is passed to the second, and to the third, and so on, until the email is successfully transmitted. This approach was great for fault-tolerance and redundancy, but not for scaleability. If the first server in the list was online and available, then it would receive all the transactional emails for that account.
I therefore decided to add an internally controlled user-level setting option to randomize the list of senders. Now, if a user's list of transactional senders included:
Sender1, Sender2, Sender3, Sender4
Now the relay.jangosmtp.net servers would farm out the emails for delivery to any sender in the user's list at random, ensuring that as many transactional senders as were assigned would receive an equal load of emails to deliver, rather than having the first available sender do all the delivery. This also aided in resolving the bandwidth bottleneck with the SMTP senders mentioned earlier.
Still not fast enough - need 3 more servers
Given our SQL query optimizations, multi-threading, additional servers, the timing now looked like:
400 emails/minute/server x 5 servers = 2,000 emails/minute x 60 minutes = 120,000 emails/hour.
120,000 emails/hour x 7 hours = 840,000 emails over 7 hours
However, if the load was over 1 million emails, there would still be a delay. I decided to annex 3 additional servers that are already a part of the JangoMail network but weren't active on port 25, and turn them into 3 additional SMTP receivers. There were now 8 total DNS "A" records for relay.jangosmtp.net.
With just 48 hours before the marathon, we were informed that the expected outbound volume, based on the number of subscribers so far, would be about 750,000 emails.
And just in case all else failed...
While we've always believed in the performance and reliability of our code and architecture, we decided that for a project of this caliber, a backup plan was necessary in case our custom SMTP architecture was unable to perform. The JangoSMTP custom architecture is what allows emails that pass through the SMTP relay to be open and click-tracked, and stored in a database, so that SMTP logs can be viewed and reports can be generated based on open and click timing, domains, and geo-tracking reports based on IP addresses. However, the primary issue for the marathon, was to ensure the emails were delivered, and delivered on time. Therefore, a backup plan was put into place that could guarantee the emails would be delivered, even if we had to eliminate the tracking and logging based on our own custom architecture. If we found that JangoSMTP could not handle the load, we would replace the instances of the JangoSMTP receiver with Microsoft's built-in SMTP service (part of Internet Information Services), such that emails would be received and delivered to the final recipient, without any processing needed in between.
In order to isolate the marathon from our other clients in case this backup plan had to be put in place, we setup the domain marathon.jangosmtp.net, which mimiced the 8 A records for relay.jangosmtp.net, and we asked the system integrators for the marathon to connect to marathon.jangosmtp.net instead of relay.jangosmtp.net. This would allow us to re-direct just their email on the day of the race if needed.
The Day of the Race - November 1, 2009
I awoke at 7:30 AM EST on that Sunday, after having been out the night prior for Halloween. The emails would begin trickling in at 8:30 AM EST for runner check-ins, and the first wave of the race was set to begin at 9:40 AM EST.
I was watching the race live on TV, while at the same time monitoring the traffic flows across our 8 instances of relay.jangosmtp.net. Emails were being received, processed, and sent quickly, and there was no backlog...until about 12:25 PM EST.
All three waves had been released, and runners from all three waves were triggering a massive volume of email alerts. From 12:25 PM EST to about 12:35 PM EST, there was a 4-5 minute delay with final delivery. Thankfully, the backlog period only lasted 10 minutes. It makes sense that approximately 3 hours after the release of the first wave, that the highest volume of email was passing through, since the greatest number of runners would still be running around this time.
Additionally, at about 1:00 PM EST, we discovered that comcast.net and att.net/bellsouth.net domains were blocking one of the IPs from which marathon email was sending. While we did have a mechanism by which domain-specific routes could be used, such that we could enable all comcast.net/att.net/bellsouth.net email to go through one specific non-blocked IP address, the complexity of the randomization system we had added to accomodate the bandwidth bottleneck rendered this mechanism non-functional. I called our lead developer, who was on call, asked him to make a change to the sender-determination algorithm, and re-deployed our code across all 8 SMTP server instances. We were now able to route all comcast.net/att.net/bellsouth.net email through a separate non-blocked SMTP sender. In the end, the marathon alert emails had less than a 0.3% blocking rate.
After 1:00 PM EST, no further email deliverability or backlog issues ensued.
Conclusion
We were thrilled that we were able to pull this off for the 2009 ING New York City Marathon. Even post-marathon, we continue to make performance and feature enhancements to our transactional email platform. If you have an important project for which sending email is critical, please get in touch with us, and we'll work as hard for you as we did for the marathon.
Tuesday, October 27, 2009
Official Launch of JangoSMTP.com, Our Stand-Alone SMTP Relay Service
We have just launched JangoSMTP.com, a new stand-alone SMTP relay service and the first designed for email marketers.
Why use it?
JangoSMTP is the first SMTP relay service to offer open and click tracking. It also comes with a variety of features that contribute to its extreme deliverability:
Who is it for?
JangoSMTP can be utilized by users of desktop email clients (Outlook, Thunderbird, etc.), Gmail users, and web programmers who send transactional emails through their own corporate servers.
Where can I learn more?
Go to http://www.jangosmtp.com/ for more information or Contact Us directly.
Why use it?
JangoSMTP is the first SMTP relay service to offer open and click tracking. It also comes with a variety of features that contribute to its extreme deliverability:
- DomainKeys/DKIM Signing
- SPF/SenderID Authentication
- Sender Score Certification
- Feedback Loops with ISPs
Who is it for?
JangoSMTP can be utilized by users of desktop email clients (Outlook, Thunderbird, etc.), Gmail users, and web programmers who send transactional emails through their own corporate servers.
Where can I learn more?
Go to http://www.jangosmtp.com/ for more information or Contact Us directly.
Monday, October 26, 2009
New API Method: AddTransactionalGroup
We just released a new API method for the JangoMail API which allows you to add a Transactional Group to your account:
*AddTransactionalGroup
Adds a new Transactional Group to your account. Returns a string.
We also recently released 3 other transactional email API methods. These methods allow you to retrieve opens, clicks, unsubscribes, bounces, and complaint statistics for a single transactional email in your account.
Visit api.jangomail.com for a full list of our API methods.
*AddTransactionalGroup
Adds a new Transactional Group to your account. Returns a string.
We also recently released 3 other transactional email API methods. These methods allow you to retrieve opens, clicks, unsubscribes, bounces, and complaint statistics for a single transactional email in your account.
Visit api.jangomail.com for a full list of our API methods.
Wednesday, October 21, 2009
Two SMTP Service Bug Fixes
Tonight we've deployed two bug fixes to the SMTP relay service:
Those wishing to read about the allowance of folded lines as stated in the SMTP specification can do so here: http://www.faqs.org/rfcs/rfc822.html. Specifically, read section:
3.1.1. LONG HEADER FIELDS
- Previously, emails sent with very long Subject lines, such that the Subject line folded onto the next line, were improperly processed, and this could result in headers being visible in the email body. This is now fixed.
- Previously, emails sent with very long From lines, such that the From line folded onto the next line, were improperly processed, and this could result in the email not being transmitted to the final recipient at all. This is now fixed.
- Previously, encoded subject lines, such as:
=?utf-8?Q?You've_received_=E2=82=A8229.64_($2.00_USD)_in_your_Surveyhead_?= =?utf-8?Q?account?=
would not have their encoding preserved. This is also now fixed.
Those wishing to read about the allowance of folded lines as stated in the SMTP specification can do so here: http://www.faqs.org/rfcs/rfc822.html. Specifically, read section:
3.1.1. LONG HEADER FIELDS
Monday, October 19, 2009
New Transactional Email API Methods
We have just released 3 new Transactional Email API Methods for the JangoSMTP Email Marketing API.
These new methods allow you to retrieve opens, clicks, unsubscribes, bounces, and complaint statistics for a single transactional email in your account:
*Reports_Transactional_GetSingleEmailStats_Dataset
Retrieves statistics for a transactional email. Returns a .NET DataSet.
*Reports_Transactional_GetSingleEmailStats_String
Retrieves statistics for a transactional email. Returns a string.
*Reports_Transactional_GetSingleEmailStats_XML
Retrieves statistics for a transactional email. Returns an XML document.
These new methods allow you to retrieve opens, clicks, unsubscribes, bounces, and complaint statistics for a single transactional email in your account:
*Reports_Transactional_GetSingleEmailStats_Dataset
Retrieves statistics for a transactional email. Returns a .NET DataSet.
*Reports_Transactional_GetSingleEmailStats_String
Retrieves statistics for a transactional email. Returns a string.
*Reports_Transactional_GetSingleEmailStats_XML
Retrieves statistics for a transactional email. Returns an XML document.
Thursday, August 13, 2009
Behind the scenes of our SMTP Service
The last couple of days have been a challenging and learning experience for the team in charge of our SMTP Relay service. As the volume of email that passes through the relay grows, so do the problems associated with performance and scaleability.
The first version of the relay service operated in a single-threaded linear fashion. Meaning, all emails sent to relay.jangosmtp.net, are processed and sent one at a time. Whenever an email message arrives at relay.jangosmtp.net, the following steps are executed:
1. The originating IP address and From Address are examined to determine what JangoSMTP user account the email message belongs to.
2. The tracking options for that particular user are loaded.
3. The email message is disassembled, the tracking mechanisms are added, and then the email message is re-assembled.
4. The email message is transmitted to the appropriate JangoSMTP sending server, based on whether the user account is enrolled in the Sender Score Certified program or based on the domain of the recipient email address.
5. The sending email server receives the message, signs the message with DomainKeys and DKIM, and then transmits the email message to the final destination email server based on the recipient domain.
A traditional SMTP service is much simpler and only need incorporate step 1 and a part of step 5. It can take our processes anywhere from 1/10th of a second to 3 seconds to process and deliver a single email message. The amount of time depends on the size of the email, the encoding of the email, and the tracking options selected.
Tuesday morning, a larger client submitted 50,000 transactional emails to relay.jangosmtp.net. The emails arrived at the relay.jangosmtp.net server over the course of 5 hours. Our linear process went to work, processing and transmitting each email message individually. As a result, email messages from other, smaller customers were delayed, some for serveral hours. We were alerted to the problem very soon after the backlog of emails began to build. Since all JangoMail staff members use the JangoMail SMTP relay with their desktop email clients, it was easy for them to tell something was wrong -- emails my staff was sending to customers, prospects, and themselves weren't being received. We realized that having a single process, processing each email one by one, just isn't going to cut it for scenarios where a customer transmits a large amount of email messages at once.
Within 8 hours, we re-architected the entire process so that multiple threads could operate on groups of email messages simultaneously. We now have the ability to assign a dedicated processing thread to any large customer. Since all threads operate simultaneously, a large amount of email from one customer will now not hold up a small amount of email from another customer.
After we deployed the new code last night, we thought our problems were solved. We awoke Wednesday morning to discover that the multi-threaded process was crashing, due to I/O concurrency issues. The constant crashing of the new code resulted in another backlog of undelivered email messages. Within 10 minutes, we discovered the issue was that all threads were attempting to log their activities to the same log file, and this was resulting in I/O disk errors. Within 60 minutes, we corrected the issue and deployed the new code, and since then (1:15 PM EST Wednesday), there have been no errors and no backlogs.
We appreciate our SMTP service customers' support over the last couple days, especially those that were adversely affected by the backlog of emails. One of the factors that I think makes JangoMail a different type of company is the awesome skill and dedication of our developers. When an architectural or functional issue is discovered, our programmers work until the problem is solved. We react swiftly, coming up with creative solutions to difficult problems within minutes -- because if we don't, the problems get worse: backlogs build, emails stop sending, the system could grind to a halt. The types of development problems we face are generally not the type that can be solved with a Google search. This is because much of JangoMail's technology is unique. We've done things that very few companies in our industry have done, including writing our own Email Rendering Tool, writing our own SMTP sending engine from scratch, and now, introducing the world's first SMTP service with tracking.
I hope this post sheds some light on the inner workings of the JangoMail operation, my staff's committment to delivering an amazing service, and a little bit of our secret sauce on how the magic happens. If anyone has any questions about the SMTP service, email me at ajay AT us dot jangomail dot com.
The first version of the relay service operated in a single-threaded linear fashion. Meaning, all emails sent to relay.jangosmtp.net, are processed and sent one at a time. Whenever an email message arrives at relay.jangosmtp.net, the following steps are executed:
1. The originating IP address and From Address are examined to determine what JangoSMTP user account the email message belongs to.
2. The tracking options for that particular user are loaded.
3. The email message is disassembled, the tracking mechanisms are added, and then the email message is re-assembled.
4. The email message is transmitted to the appropriate JangoSMTP sending server, based on whether the user account is enrolled in the Sender Score Certified program or based on the domain of the recipient email address.
5. The sending email server receives the message, signs the message with DomainKeys and DKIM, and then transmits the email message to the final destination email server based on the recipient domain.
A traditional SMTP service is much simpler and only need incorporate step 1 and a part of step 5. It can take our processes anywhere from 1/10th of a second to 3 seconds to process and deliver a single email message. The amount of time depends on the size of the email, the encoding of the email, and the tracking options selected.
Tuesday morning, a larger client submitted 50,000 transactional emails to relay.jangosmtp.net. The emails arrived at the relay.jangosmtp.net server over the course of 5 hours. Our linear process went to work, processing and transmitting each email message individually. As a result, email messages from other, smaller customers were delayed, some for serveral hours. We were alerted to the problem very soon after the backlog of emails began to build. Since all JangoMail staff members use the JangoMail SMTP relay with their desktop email clients, it was easy for them to tell something was wrong -- emails my staff was sending to customers, prospects, and themselves weren't being received. We realized that having a single process, processing each email one by one, just isn't going to cut it for scenarios where a customer transmits a large amount of email messages at once.
Within 8 hours, we re-architected the entire process so that multiple threads could operate on groups of email messages simultaneously. We now have the ability to assign a dedicated processing thread to any large customer. Since all threads operate simultaneously, a large amount of email from one customer will now not hold up a small amount of email from another customer.
After we deployed the new code last night, we thought our problems were solved. We awoke Wednesday morning to discover that the multi-threaded process was crashing, due to I/O concurrency issues. The constant crashing of the new code resulted in another backlog of undelivered email messages. Within 10 minutes, we discovered the issue was that all threads were attempting to log their activities to the same log file, and this was resulting in I/O disk errors. Within 60 minutes, we corrected the issue and deployed the new code, and since then (1:15 PM EST Wednesday), there have been no errors and no backlogs.
We appreciate our SMTP service customers' support over the last couple days, especially those that were adversely affected by the backlog of emails. One of the factors that I think makes JangoMail a different type of company is the awesome skill and dedication of our developers. When an architectural or functional issue is discovered, our programmers work until the problem is solved. We react swiftly, coming up with creative solutions to difficult problems within minutes -- because if we don't, the problems get worse: backlogs build, emails stop sending, the system could grind to a halt. The types of development problems we face are generally not the type that can be solved with a Google search. This is because much of JangoMail's technology is unique. We've done things that very few companies in our industry have done, including writing our own Email Rendering Tool, writing our own SMTP sending engine from scratch, and now, introducing the world's first SMTP service with tracking.
I hope this post sheds some light on the inner workings of the JangoMail operation, my staff's committment to delivering an amazing service, and a little bit of our secret sauce on how the magic happens. If anyone has any questions about the SMTP service, email me at ajay AT us dot jangomail dot com.
Sunday, August 2, 2009
You Can Now Use the JangoSMTP Relay with Gmail
Gmail allows users to use a custom From Address to send emails, but some recipients would see both the From Address and the Gmail Sender Address. Messages would appear to recipients as "From username@gmail.com On Behalf Of username@customdomain.com". Gmail announced last week that users can now send with their custom domain through their company's email servers, and that will eliminate the "On Behalf of" message.
Configuring Your Gmail Account
Set up Gmail to use the JangoSMTP Relay.
1. Sign into your Gmail account.
2. Go to Settings in the upper right corner.
3. Click on the Accounts tab.
4. In the Send mail as: section, click edit info next to the account that you want to change. You should already have your business email address set up to send through Gmail. If you don't, click the Add Another Email Address option and follow the steps provided.
5. Hit the Next Step button.
6. Choose the Send through [your domain] SMTP servers option.
7. For SMTP Server, erase the default address and type in relay.jangosmtp.net
8. For Port, choose 25.
9. Type in your JangoSMTP Username and Password.
8. Save Changes
Gmail will now send your emails via the JangoSMTP Relay instead of its own servers. Enjoy the Open Tracking, Click Tracking, and all the other benefits of our SMTP service.
This means that you can now use the JangoMail SMTP Service with Gmail to eliminate the "On Behalf of" message and give you the added benefits of Open Tracking, Click Tracking, SMTP logging, and participation in the Sender Score Certified Program.
*Please note: This is only available for people using a custom From Address (NON-Gmail Address) in Gmail.
*Please note: This is only available for people using a custom From Address (NON-Gmail Address) in Gmail.
Configuring Your Gmail Account
Set up Gmail to use the JangoSMTP Relay.
1. Sign into your Gmail account.
2. Go to Settings in the upper right corner.
3. Click on the Accounts tab.
4. In the Send mail as: section, click edit info next to the account that you want to change. You should already have your business email address set up to send through Gmail. If you don't, click the Add Another Email Address option and follow the steps provided.
5. Hit the Next Step button.6. Choose the Send through [your domain] SMTP servers option.
7. For SMTP Server, erase the default address and type in relay.jangosmtp.net
8. For Port, choose 25.
9. Type in your JangoSMTP Username and Password.
8. Save Changes
Gmail will now send your emails via the JangoSMTP Relay instead of its own servers. Enjoy the Open Tracking, Click Tracking, and all the other benefits of our SMTP service.
To view Gmail's announcement of this new feature, visit:
http://gmailblog.blogspot.com/2009/07/send-mail-from-another-address-without.html
http://gmailblog.blogspot.com/2009/07/send-mail-from-another-address-without.html
Friday, July 31, 2009
Video Tutorials Now Available on Trackable SMTP Relay Configuration
JangoMail has two new Video Tutorials on configuring our SMTP Relay Service:
1. Using the JangoSMTP Relay with a Desktop Email Client
2. Using the JangoSMTP Relay on a Web Server
Watch these to learn how to configure the JangoSMTP Relay Service to work with a Desktop Email Client like Outlook, Thunderbird, and Lotus Notes and how to configure it for use on a Web Server. This new service improves your sending capabilities by offering these benefits:
- Open Tracking
- Click Tracking
- DomainKeys/DKIM Signing
- Easy web-based access to SMTP Logs
- Categorization of different types of transactional emails
- API methods to retrieve Reporting data
Thursday, July 16, 2009
Bug fix: Using the unsubscribe link in transactional emails
We have fixed a bug that prevented the unsubscribe link from working properly in transactional emails, including emails sent with the trackable SMTP relay.
To insert an unsubscribe link into transactional emails, do so the same way as you would with a regular email broadcast campaign, using a snipped of HTML code:
Just link to this URL:
http://x.jango9.com/u.z?***uniqueid****
and be sure to replace x.jango9.com with your account's actual Tracking Domain. The system default Tracking Domain is currently x.jango9.com.
If you link to the above URL in a transactional email, the ***uniqueid**** will be replaced by the JangoMail email engine with a string unique to your recipient. Then, then the URL is clicked, the recipient will be added to your account's unsubscribe list, preventing said person from receiving future emails from your account.
To insert an unsubscribe link into transactional emails, do so the same way as you would with a regular email broadcast campaign, using a snipped of HTML code:
Just link to this URL:
http://x.jango9.com/u.z?***uniqueid****
and be sure to replace x.jango9.com with your account's actual Tracking Domain. The system default Tracking Domain is currently x.jango9.com.
If you link to the above URL in a transactional email, the ***uniqueid**** will be replaced by the JangoMail email engine with a string unique to your recipient. Then, then the URL is clicked, the recipient will be added to your account's unsubscribe list, preventing said person from receiving future emails from your account.
Sunday, July 5, 2009
Frequently Asked Questions (FAQ) about the SMTP relay service
Q. How are bounces handled?
A. If you send an email message to an address that is on your account's Bounce list, then that email message won't be sent, and instead, a notification will be sent back to the sender with this error message:
Web Service Exception BouncedEmailAddressException: Email address is on account's Bounce list. System.Exception: JangoMailNamespace.BouncedEmailAddressException: Web Service Exception BouncedEmailAddressException: Email address is on account's Bounce list.
If you send an email message to an address that is not yet on your account's bounce list, but the recipient address is invalid, then that address will then be added to your account's Bounce list. If you wish to receive an email notification of the bounce-back, like you would with a "regular" SMTP service, then set the appropriate options under My Options and Settings --> Bounce Handling.
Q. How are unsubscribes handled?A. If you send an email message to an address on your account's Unsubscribe list, the email message will not be sent and instead an error notification containing the following text will be sent back to the From Address on the email message:
Web Service Exception UnsubscribedEmailAddressException: Email address is on account's Unsubscribe list. System.Exception: Namespace.UnsubscribedEmailAddressException: Web Service Exception UnsubscribedEmailAddressException: Email address is on account's Unsubscribe list.
You have the option to have the SMTP relay bypass your account's unsubscribe list and not perform this check. To do so, check the appropriate box under My Options and Settings --> SMTP Relay.
Q. How is word wrapping handled?
By default, the SMTP relay is set to wrap lines at 76 characters. Many of Microsoft's email clients wrap lines at 76 characters, so we chose this as the default. You can, however, customize this setting under My Options and Settings --> SMTP Relay. Most desktop email clients, like Outlook, Thunderbird, and Eudora already transmit the email to the SMTP server with proper word wrapping in place. Many programming-based components, like PHP, ASP, and ASP.Net email components that send email programatically via an SMTP server, may not do any word wrapping at all. It is for this reason that the SMTP relay will do the word wrapping at 76 characters by default. To prevent the SMTP relay from doing any word wrapping, set the Wrapping option to 0 under My Options and Settings --> SMTP Relay
Q. Can I use the standard JangoMail unsubscribe link in my email messages sent via the SMTP relay?
A. Yes, you can. Just link to:
http://x.jango8.com/u.z?***uniqueid****
x.jango8.com can be replaced with the Tracking Domain specific to your account.
Q. My email client sends emails encoded in base 64 or quoted-printable. Will there be any issues with this?
A. Everything will work fine. JangoMail's SMTP relay will not only preserve your email client's original encoding scheme, but it will also still be able to add the open-tracking and click-tracking entities to your email message.
Q. What are the costs to use the SMTP relay?
A. The pricing is based on the number and size of the emails you send per month. Pricing is at http://www.jangosmtp.com/Pricing.asp.
Q. What happens if I attempt to send more emails through the SMTP relay than my account allows?
A. If your account is marked as an "overage allowed" account, then you will never encounter this issue. If your account does have set limits, as most do unless you've requested otherwise, then when your emails exceed your account limits, the sender will get return notifications indicating that the account is over limit and that the email was not delivered.
Q. Is my email message delivered to my recipient instantly through the SMTP relay?
A. Your email message is delivered within 3-5 seconds after it arrives at relay.jangosmtp.net. That 3-5 seconds is required for processing, such as adding the open-tracking and click-tracking mechanisms, signing with DomainKeys (and DKIM), and checking against your account's Bounce list.
Q. Can click-tracking affect phishing filters?
A. Yes, depending on the email program that the recipient is using. Since click-tracking alters your original URL, spam filters may detect this as a phishing attempt. The smarter email clients get though, the less of an issue this will become over time. Additionally, if your display text itself isn't a URL, then there won't be an issue. For example:
This is the best web site. (good way to link)
versus
http://www.google.com/ is the best web site. (bad way to link)
Q. What happens if I send email from either an IP address or a From Address that I have not yet authorized under My Options and Settings --> SMTP Relay?
A. An error notification will be sent to the Sender of the email message with the IP and From Address infomation:
Relaying denied for [IP Address] / "Joe Smith"
A. If you send an email message to an address that is on your account's Bounce list, then that email message won't be sent, and instead, a notification will be sent back to the sender with this error message:
Web Service Exception BouncedEmailAddressException: Email address is on account's Bounce list. System.Exception: JangoMailNamespace.BouncedEmailAddressException: Web Service Exception BouncedEmailAddressException: Email address is on account's Bounce list.
If you send an email message to an address that is not yet on your account's bounce list, but the recipient address is invalid, then that address will then be added to your account's Bounce list. If you wish to receive an email notification of the bounce-back, like you would with a "regular" SMTP service, then set the appropriate options under My Options and Settings --> Bounce Handling.
Q. How are unsubscribes handled?A. If you send an email message to an address on your account's Unsubscribe list, the email message will not be sent and instead an error notification containing the following text will be sent back to the From Address on the email message:
Web Service Exception UnsubscribedEmailAddressException: Email address is on account's Unsubscribe list. System.Exception: Namespace.UnsubscribedEmailAddressException: Web Service Exception UnsubscribedEmailAddressException: Email address is on account's Unsubscribe list.
You have the option to have the SMTP relay bypass your account's unsubscribe list and not perform this check. To do so, check the appropriate box under My Options and Settings --> SMTP Relay.
Q. How is word wrapping handled?
By default, the SMTP relay is set to wrap lines at 76 characters. Many of Microsoft's email clients wrap lines at 76 characters, so we chose this as the default. You can, however, customize this setting under My Options and Settings --> SMTP Relay. Most desktop email clients, like Outlook, Thunderbird, and Eudora already transmit the email to the SMTP server with proper word wrapping in place. Many programming-based components, like PHP, ASP, and ASP.Net email components that send email programatically via an SMTP server, may not do any word wrapping at all. It is for this reason that the SMTP relay will do the word wrapping at 76 characters by default. To prevent the SMTP relay from doing any word wrapping, set the Wrapping option to 0 under My Options and Settings --> SMTP Relay
Q. Can I use the standard JangoMail unsubscribe link in my email messages sent via the SMTP relay?
A. Yes, you can. Just link to:
http://x.jango8.com/u.z?***uniqueid****
x.jango8.com can be replaced with the Tracking Domain specific to your account.
Q. My email client sends emails encoded in base 64 or quoted-printable. Will there be any issues with this?
A. Everything will work fine. JangoMail's SMTP relay will not only preserve your email client's original encoding scheme, but it will also still be able to add the open-tracking and click-tracking entities to your email message.
Q. What are the costs to use the SMTP relay?
A. The pricing is based on the number and size of the emails you send per month. Pricing is at http://www.jangosmtp.com/Pricing.asp.
Q. What happens if I attempt to send more emails through the SMTP relay than my account allows?
A. If your account is marked as an "overage allowed" account, then you will never encounter this issue. If your account does have set limits, as most do unless you've requested otherwise, then when your emails exceed your account limits, the sender will get return notifications indicating that the account is over limit and that the email was not delivered.
Q. Is my email message delivered to my recipient instantly through the SMTP relay?
A. Your email message is delivered within 3-5 seconds after it arrives at relay.jangosmtp.net. That 3-5 seconds is required for processing, such as adding the open-tracking and click-tracking mechanisms, signing with DomainKeys (and DKIM), and checking against your account's Bounce list.
Q. Can click-tracking affect phishing filters?
A. Yes, depending on the email program that the recipient is using. Since click-tracking alters your original URL, spam filters may detect this as a phishing attempt. The smarter email clients get though, the less of an issue this will become over time. Additionally, if your display text itself isn't a URL, then there won't be an issue. For example:
This is the best web site. (good way to link)
versus
http://www.google.com/ is the best web site. (bad way to link)
Q. What happens if I send email from either an IP address or a From Address that I have not yet authorized under My Options and Settings --> SMTP Relay?
A. An error notification will be sent to the Sender of the email message with the IP and From Address infomation:
Relaying denied for [IP Address] / "Joe Smith"
Tuesday, June 16, 2009
PDF Document now available on Trackable SMTP Relay feature
The official PDF guide is now available:
http://www.jangosmtp.com/JangoSMTP-Tutorial-SMTP-Relay-Tracking.pdf
Read this step by step guide to learn how to track every single person to person email that you send. The new SMTP service has lots of benefits over your existing SMTP service:
http://www.jangosmtp.com/JangoSMTP-Tutorial-SMTP-Relay-Tracking.pdf
Read this step by step guide to learn how to track every single person to person email that you send. The new SMTP service has lots of benefits over your existing SMTP service:
- Open Tracking
- Click Tracking
- DomainKeys/DKIM Signing
- Easy web-based access to SMTP Logs
- Categorization of different types of transactional emails
- API methods to retrieve Reporting data
Friday, June 5, 2009
New Feature: Trackable SMTP relay now in beta
Today we have launched a feature into beta, that no other email marketing service provider has -- an SMTP relay service built specifically for email marketers. Any JangoMail active account holder can test it right now.
You can now send all of your person-to-person and transactional emails through the JangoMail SMTP relay at relay.jangosmtp.net.
The benefits of relaying your individual email messages through relay.jangosmtp.net instead of your corporate email server or your ISP email server are:
1. Login to your account and go to My Options and Settings --> SMTP Relay.
2. Review the settings and setup an authentication scheme, either by IP Address or by From Address.
3. If you choose to authenticate by From Address, then your email system will have to authenticate into the SMTP relay service with your JangoMail account username/password. Additionally, all of your emails must come from the From Address(es) that you specify under My Options and Settings --> SMTP Relay --> From Addresses.
4. After you've setup your authentication and tracking preferences, start relaying your email through relay.jangosmtp.net. You can connect on port 25 or port 2525.
5. All Reporting is in real-time under Reporting --> Transactional Emails.
------------------------------------------------------
Update on 6/15/09: Official PDF document now available at
http://www.jangosmtp.com/JangoSMTP-Tutorial-SMTP-Relay-Tracking.pdf
You can now send all of your person-to-person and transactional emails through the JangoMail SMTP relay at relay.jangosmtp.net.
The benefits of relaying your individual email messages through relay.jangosmtp.net instead of your corporate email server or your ISP email server are:
- Open Tracking
- Click Tracking
- SMTP Logging
- DomainKeys/DKIM signing
- Automatic plain text and HTML message generation
- Grouping of different types of emails into different categories for ease of reporting
1. Login to your account and go to My Options and Settings --> SMTP Relay.
2. Review the settings and setup an authentication scheme, either by IP Address or by From Address.
3. If you choose to authenticate by From Address, then your email system will have to authenticate into the SMTP relay service with your JangoMail account username/password. Additionally, all of your emails must come from the From Address(es) that you specify under My Options and Settings --> SMTP Relay --> From Addresses.
4. After you've setup your authentication and tracking preferences, start relaying your email through relay.jangosmtp.net. You can connect on port 25 or port 2525.
5. All Reporting is in real-time under Reporting --> Transactional Emails.
------------------------------------------------------
Update on 6/15/09: Official PDF document now available at
http://www.jangosmtp.com/JangoSMTP-Tutorial-SMTP-Relay-Tracking.pdf
Subscribe to:
Posts (Atom)