2020-07-30 Webapp Build Once Postmortem

Date: 2020-07-30

Summary: Upmo was down for 1 hour due to a change in how we deploy our web application

Impact:

• Google Maps and Autocomplete are not functional Book
• viewing can't be completed (not tested) because it's pointing to the wrong Auth0 endpoint
• Users can't login as it's pointing to the wrong Auth0 endpoint
• upmo.com was not navigetable for 5 - 10 minutes

Root causes: Non-production URLs are hardcoded in the webapp build, resulting in the production webapp trying to talk to non-prod resources.

Resolution: Code revert

Detection: Manual + synthetic monitoring

Action items:

Action item	Type
Clear CloudFront cache on every deploy	mitigate
Create smoke testing suite	mitigate
Disallow use of process.env via Lint	prevent
Reduce the amount of shared nonprod resources	prevent
Check on production after prod deploy	process

Lessons Learned

What went well

• Synthetic monitoring did fail
• We spend minimum time in production to learn about the issue
• The revert is a successful

What went wrong

• There are too many issues happening at once: auth0, google maps, etc.
• Misunderstanding of Nuxt Runtime configs
• Acceptance tests raised no errors
• Issues were not reproducible in staging environments

Where we got lucky

• We discovered this issue before the synthetic monitoring is triggered, because the developer is conscious to make sure that the code change is working in production
• Not a lot of active users

Timeline

2020-07-30 (all times UTC)

• 17:48 CircleCI deploys the latest change to production

• 17:51 Wisen notices that our website is not navigatable, some of the pages are blank

  • This is the error in the browser console: GET https://upmo.com/_nuxt/app.76279fd.js net::ERR_ABORTED 404
  • Hard reloading didn't seem to help

• 17:53 The blank page issue is resolved

• 17:53 Google Autocomplete and Maps are showing error in the browser console

  Google Maps JavaScript API error: RefererNotAllowedMapError
  https://developers.google.com/maps/documentation/javascript/error-messages#referer-not-allowed-map-error
  Your site URL to be authorized:
  https://upmo.com/offices?numOfPeople=20&minPrice=0&maxPrice=20000
  

• 17:53 Observes another error in the browser console related to Auth0: VM1040:1 GET https://accept-upmo.eu.auth0.com/userinfo 401

  • Our production seems to point to the wrong runtime configuration

• 17:55 Verified that the latest runtime config contains the correct Google Maps API Key in __NUXT__.config

• 17:59 Synthetic monitoring is failing

• 17:58 Found the culprit for Auth0 issues in nuxt.config.js, where values are not taken from runtime config (hardcoded to accept environment)

• 18:01 Code reverted and pushed

• 18:09 Found the culprit for Google maps in nuxt.config.js, where the API key is not taken from the runtime config (hardcoded to accept environment)

  • This is discovered from searching the non-prod API key in the browser's console

• 18:40 Production is reverted. Incident resolved.

• 18:47 Synthetic monitoring failure is resolved