10. SOM synthetic monitoring

Date: 2020-07-08
Driver: Harsh S. Kulshrestha
Approver: Wisen Tanasa

Status

Accepted

Context

As part of operational readiness post go live, we would like to create synthetic transactions on our app on a regular basis. The transactions are expected provide maximum coverage over the app, run on a scheduled basis, and raise alerts about any potential problems with the app.

There are two possible ways we can achieve synthetic monitoring on SOM:

1. Using AWS lambdas responsible for running automation suites on the app on a scheduled basis. This would mean that we will have to bundle automation suites like puppeteer with the deployment package, and handle every aspect of monitoring ourselves. For example, creating suitable cloudwatch metrics, managing lambda layers, keeping deployment size under permissible limits etc. While this is achievable, it would be time consuming and would require a good amount of effort.
2. Leveraging synthetic monitoring suite introduced by AWS recently, i.e., AWS Canaries. These are highly similar to the automated lambdas discussed above, but are managed by AWS. Canaries come bundled with puppeteer, basic node libraries (with runtime support for node 10.x), and are easy to integrate with cloudwatch alarms. With the use of canaries, we expect only the automation script would require most of our effort, the rest should be easily handled by AWS.

Primary reasons why the above two approaches are the best suited ones in this case is because from any suite that runs synthetic monitoring, we would have certain minimum expectations such as ability to run on a scheduled basis, metrics, cost effectiveness (since this is practically going to run 24*7 at regular intervals), and low infrastructual maintainance. Lambdas fulfill almost every aspect of our requirements. Moreover, there might be scenarios in which we might need the suite to directly interact with our underlying AWS infrastructure (for example, invoking another lambda and bypassing api restrictions). This could easily be achieved with the above two approaches.

Decision

We will go ahead with approach#2. This is primarily because since it's a managed service, it should be quick to setup as compared to creating custom lambdas.

Consequences

By incorporating synthetic monitoring in our app, we will be able to gain a very high degree of confidence in the app. We can rely on synthetic monitoring in making sure that our production environment is functioning as expected.

By using AWS Managed Canaries to create our synthetic monitoring suite, we're gaining up on minimising the effort required to setup our own synthetic monitoring suite and also the time taken will be less.

A drawback of using such an AWS service is that we will have limited control on how we can tweak the complete setup to make best use of it. For example, it doesn't support environment variables as of now, and accepts inline scripts (as a string)/scripts uploaded to S3 to be run as part of the suite. As a result, we have to patch through the way in which we can interpolate variables in the script. To overcome this, we are using es6-templated-strings, which is capable of interpolating variables within a file. Moreover, since it a recently introduced service, it's still maturing. For example, there is no guraantee that once you upload a new script via cdk, the changes will be reflected in the next run. It might have something to do with how lambdas cache their code, but there's no documentation to support it. We also don't have any control over the lambdas that canaries create. While this is not a must have, having such capabilites could help optimise the complete suite better.

While AWS canaries satisfy our basic expectations, we do acknowledge the fact that it is a fairly new service offered by AWS and is in its initial stages. We are taking a risk by adopting canaries and might have to rethink our approach if we need more extensive and flexible monitoring suite, and if canaries fail to provide that in future.