7. Serverless conditional future task

Date: 2020-05-21
Driver: Wisen Tanasa

Status

Accepted

Context

We are required to send reminder emails. Here are the scenarios:

1. 24 hours before viewing date to seekers (viewing reminder)
2. 24 hours after viewing creation to support (following up)

We have run a spike previously on AWS Step Function, and we have understood that we can leverage the service to schedule a future task. The complexity comes when a viewing is rescheduled on scenario 1. This reminder email must not be sent when a viewing is rescheduled as that email will contain wrong information.

We could cancel the currently running execution, so that the reminder email is not sent when it is being rescheduled. This approach however will not work on the scenario 2. In scenario 2, we do not want to send a follow up reminder when a viewing status is no longer set to Pending. When the providers have confirmed a viewing - status is Confirmed, we can't cancel the execution, as all of the subsequent tasks will be cancelled as well. It's not possible to just cancel 1 single task within an execution.

We could extract all the reminder tasks to become an independent Step Function. We think this solution might work, but might render too complex for our current size of workflow.

The alternative to leveraging Step Function is to run the reminders as cron jobs or batch. These cron jobs will be running every minute or day. The script that's being run by the cron then will be able to query the viewings which require reminders. This will solve all the problems described in the previous paragraphs. There are two problems with the batch approach:

1. It will not scale. Eventually when we have too many viewings, the batch might not be able to catch up, and we'll need to design it to work better concurrently.
2. We'll lose the tracability of the workflow as this script is running out of the main workflow, which is managed by Step Function

Decision

We will be sticking with Serverless solution to send the reminders. To mitigate the change of viewing status or date, we will make sure that the current state of the viewing is still correct before we send the reminders out:

• On the viewing reminder scenario, before we send the emails out, we will check if the current date is 24 hours before the viewing date and if the status is Confirmed.
• On the following up scenario, before we send the emails out, we will check if the viewing status is Pending.

Consequences

With this approach, our solutions will scale with the cloud. We wouldn't need to worry when our batch is hitting is limit, or not able to keep up with the scale.

The conditional checks that we do before we send the emails could also be considered as business logic, therefore we will be able to test these logics in unit tests. Testing execution cancellation in contrast will be much harder.

The downside of this decision is the state of the Step Function will not reflect the actual task run. When a task is named as Send Follow up Reminder, and Step Function is showing that this task has run, we don't actually know whether the reminder were sent or not. We will also be unnecessarily paying Lambda compute when the task is not actually going to do do anything.