How we Scaled our Notification System at MUNCH:ON

how we scaled

Notifications are very important triggers for any product but even more so for us at MUNCH:ON. Food delivery services, like ours, require a lot of notifications to be sent out to users. We send out emails, SMS’s, and pushes throughout the day when users place and receive orders, a new menu is scheduled, and for a variety of other reasons. We, therefore, needed a system that is independent and scalable in order to handle the load at peak hours. It’s important to note that notifications are not being built or released every day so we wanted to keep them separate. Single monolithic applications were unable to serve this purpose for us, so we had to create our own solution. We, therefore, moved to microservices and built a separate notifications system that can be scaled and maintained, as and when needed.

What we were doing:

  • Notification workers in the same application project. Code of email or SMS notifications were being done in the same application that serves APIs to clients. 
  • Replicated the same workers in different application projects for the same notification and for the same purpose. 
  • Extra load on the server while making 3rd party calls to SMS, push, and email clients.
  • Queue processing on the same server using Supervisor.
  • Difficulty while migrating to microservices where APIs and functionalities are built and served from different application services. 
  • Increased server load and cost at peak hours, when users are placing orders on our application and transaction notifications are being sent out.

What we are doing now:

  • We built each worker for our notifications in NodeJS in a separate microservice.
  • When we need to send a notification to our customer or a transaction is made on our application, we make a request to AWS sqs or sns depending on the requirement.
  • Our consumers on the other side in NodeJS are listening and receiving messages from AWS SQS.
  • Messages sent from Producers contain only 2 Values, ID and Type. ID can be Order ID or User ID and Type contains the type of notification to be prepared in Preparation Service. Here, each notification data is prepared, i.e message for SMS or substitution tags for emails.
  • Once the notification data is prepared, it is then sent to another queue and the consumer of that queue is in Sender Service.
  • Sender service then sends notifications to respective clients i.e email, SMS or push and saves logs in DynamoDB.

What have we achieved?

Plug & Play: We can easily plug code into any service or application that we want to send notifications to on any action or transaction. We just need to add a AWS sqs/sns connection and to send a message to sqs/sns. AWS sqs/sns connection and sdks are available for many languages including php, java(spring), nodeJS. We simply add a few lines of code from wherever we want to send a notification. It saves a lot of time for different teams across the company.

Java – Spring Boot

amazonSQS.sendMessage(queueUrl, objectMapper.writeValueAsString(data)) 

amazonSNS.publish(topicArn, objectMapper.writeValueAsString(data))

Php – Laravel

$this->sns->publish([

   ‘TopicArn’ => config(‘awsSnsARN’).’:’.$topic,

   ‘Message’ => json_encode($data)

])

$params = [

   ‘DelaySeconds’ => 0,

   ‘MessageBody’ => json_encode($data),

   ‘QueueUrl’ => config(‘awsConfig.awsQueueUrl’).config(‘awsConfig.awsAccountId’).’/’.$queue

];

   $this->sqs->sendMessage($params);

NodeJS

     const params = {

       MessageBody: JSON.stringify(data),

       QueueUrl: `${config.AWS.SQSUrl}${config.AWS.accountId}/${queName}`,

       DelaySeconds: 0,

     };

     sqs.sendMessage(params, (error, result) => {

       if (error) {

         console.log(error, error.stack);

       } else {

         console.log(result);

       }

     });

So there’s no need to write any markup or build any notification or substitute tag inside a service, application, or APIserving response to clients. We just ask our Preparation & Sender services to do this for us.

  • Saving Network Calls: AWS sns gives us the ability to save network calls. For example, where three types of notifications need to be sent, i.e email, SMS, or push on a certain action, we just send a notification to our SNS topic which is subscribed by multiple SQS queues. So, instead of making three network calls, we only make one and only make direct calls to SQS when needed. AWS calls it ‘fan out’.
  • Highly Scalable & Decoupled: Our whole notification system is very decoupled and highly scalable. For instance, if there are more users than expected at peak times and notifications become delayed in queues waiting for workers or even when nodeJS processes begin to take more time, we can easily increase and multiply our workers at the consumer-end or replicate preparation service instances to handle a greater load. As we have separate service Sender for making 3rd party Http calls to email, SMS and push clients, Preparation service frees itself by sending messages to Sender and begins to consume the next notification in the queue.
  • Upgradable: This architecture provides us the ability to easily change or upgrade as well. For example, SQS can be replaced with any other queue brokers like Kafka or RabbitMQ. Currently, we are using SendGrid as an email client, but now that can be changed to any other client if we need to. We would just need to update in Sender Service. This applies for SMS and push clients as well.

Written by: Usman Riaz, Product Engineer at MUNCH:ON. Special thanks to the MUNCH:ON microservices team; Awn Ali, Rohaan Hussain, and Hassan Ejaz.

Leave a Reply

Your email address will not be published. Required fields are marked *