Rewriting 30,000 lines of code in a friendly way

Wednesday, Dec 7th, 2016

Mixmax is a communications platform that brings email into the 21st century. We believe everything you do today on the web should be possible in any email.

This blog post is part of the Mixmax 2016 Advent Calendar . The previous post on December 6 th was about Improving Elasticsearch querytimes .

Sequences is one of our flagship features on Mixmax. Sequences enables users to create outbounds campaigns that can be configured to the last detail, with granular customizations perrecipient.

Sequences evolved from the Mail Merge feature. Originally, Mail Merges were a subset of the functionality currently provided by Sequences. It allowed to create a single email campaign with user variables that were filled with a CSV uploaded by the user. Eventually, Mail Merges evolved to allow sending multiple emails in a campaign, and more recently, we allowed for adding recipients after the initial set of recipients, as well as customizing stages once they were sent and we plan to implement many more features in the comingmonths.

Everything that powered Mail Merge was rebuilt on a period of about 2 months. During this time, there was over 30,000 lines of code removed, reimplementing the whole features under the Sequences concept in a fraction of that code. Also, hundreds of thousands of documents were migrated and amended to match the new collection design and modulesinterfaces.

It was an incredibly daunting task. Sequences being a stable and well known feature for our valuable users, it had to remain stable during the whole process of the refactor. Mail Merges were actually becoming more unstable as we added more features due the high amount of complexity. We even had dedicated engineers to support the mail merge infrastructure while we were rewriting into Sequences. It was really a race against time that we eventuallybeat.

A quick look into MailMerge

We use MongoDB as our database, as such, we designed our collection similar tothis:

{ name: 'My Campaign' subject: 'Introduction', body: 'Hello world!', variables: ['name'], scheduledAt: '2016-11-05T18:00:00.000Z', sentAt: '2016-11-05T18:00:00.000Z', messages: ['messageId_1', 'messageId_2', 'messageId_3'], recipients: [ { email: 'foo@example.com', variables: { name: 'Foo' } }, { email: 'bar@example.com', variables: { name: 'Bar' } } ] }

The Mail Merge sending flow is specialized for its use case (hundreds of recipients in a campaign), a lot of code supported this specialized use case and, with time, organically grew to become a very daunting component in oursystem.

Once we decided to expand Mail Merges so it could send multiple stages at the time, we needed to do so in the least disruptive way and taking care not to break existing interfaces and components, as such, we created a new collection MailMergeStage that looks exactly like the MailMerge with a few moreadditions:

// The same as `MailMerge` but with the following added properties: { trigger: 'notRead', // The stage will be sent as long as it hasn't been read offset: 259200000 // the stage will be sent after this amount of time (in millis) passes after the previous stage. }

In addition, the MailMerge collection received a new field: stages which is an array of the stage ids belonging to the mail merge campaign. Additionally, message documents receive an additional metadata field: mailMergeId if the message belongs to a MailMerge or mailMergeStageId if the message was sent from a MailMergeStage document.

Since a MailMergeStage is almost exactly the same as a MailMerge document, all interfaces recognized these objects without any changes, since stages are sent conditionally, we did build some code that worked exclusively with MailMergeStages but overall, the impact on the existing code was minimal. However it already showed signs of code smell :

Message documents would either have mailMergeId or mailMergeStageId to reference the parent campaign document. So we needed to query both fields because functions would receive either a MailMerge or a MailMergeStage document. As the codebase evolved, we needed to distinguish between a MailMergeStage and a MailMerge more and moreoften. The first stage of a mail merge campaign was always explicitly handled in a special way because the first stage happened to be the MailMerge document itself, while the remaining stages were MailMergeStages documents. From its conception, a mail merge was designed to run as a monolithic campaign that runs once. Adding recipients after the initial set users uploaded from the CSV was verydifficult

Also, more often than not, we needed information specific to a Message , but we only kept the Message id field, so if we wanted to simply check the recipients in a given campaign, we needed to run a second query on the Message collection.

Processing a stage could become a slow process, mainly due the high amount of logic involved during a messagecreation.

We had many ideas to improve our Mail Merge product but we were more and more slowed down by the existing design of the Mail Merge collections, and the lack of modularity. While we had a powerful Inbox Syncing feature and scalable and fast infrastructure for our Live Feed, Mail Merges could not make use of these features due tech debt and the highly specific workflows that were in place for MailMerge.

Refactoring Mail Merges intoSequences

The first step on our refactor effort was to determine what is the central actor in a Sequence. The more and more we analyzed the use cases, UX , mocks and future features, we saw how the "Recipient" is the central actor in thefeature:

A sequence is sent to arecipient A recipient has individual analytics, independent of other recipients in thecampaign. Every recipient responds differently to thecampaign. Campaigns should be totally customizable. While they have general configurations, such as message content, subject and other settings, it was clear that users want to have the option to tailor the campaign for each recipient ifnecessary. A campaign should be a living entity that can be modified, extended and that can have new recipients added to it at any point intime.

Under this line of thinking, a Sequence then became a sort of "blueprint" which is carried over for eachrecipient.

While with Mail Merge the central actor was the Stage, with Sequences the central actor is the recipient. The document structure for the recipient is similar tothis:

// SequenceRecipient { email: 'foo@example.com', sequence: 'sequence_id', stages: [ { id: 'stage_id', scheduledAt: '2016-11-05T18:00:00.000Z', sentAt: '2016-11-05T18:00:00.000Z', body: 'customized body', message: { id: 'message_id' } } ], variables: { name: 'Foo' } } // Sequ