# 2020-07-30 Webapp Build Once Postmortem
Date: 2020-07-30
Summary: Upmo was down for 1 hour due to a change in how we deploy our web application
Impact:
- Google Maps and Autocomplete are not functional Book
- viewing can't be completed (not tested) because it's pointing to the wrong Auth0 endpoint
- Users can't login as it's pointing to the wrong Auth0 endpoint
- upmo.com was not navigetable for 5 - 10 minutes
Root causes: Non-production URLs are hardcoded in the webapp build, resulting in the production webapp trying to talk to non-prod resources.
Resolution: Code revert
Detection: Manual + synthetic monitoring
Action items:
Action item | Type |
---|---|
Clear CloudFront cache on every deploy | mitigate |
Create smoke testing suite | mitigate |
Disallow use of process.env via Lint | prevent |
Reduce the amount of shared nonprod resources | prevent |
Check on production after prod deploy | process |
# Lessons Learned
# What went well
- Synthetic monitoring did fail
- We spend minimum time in production to learn about the issue
- The revert is a successful
# What went wrong
- There are too many issues happening at once: auth0, google maps, etc.
- Misunderstanding of Nuxt Runtime configs
- Acceptance tests raised no errors
- Issues were not reproducible in staging environments
# Where we got lucky
- We discovered this issue before the synthetic monitoring is triggered, because the developer is conscious to make sure that the code change is working in production
- Not a lot of active users
# Timeline
2020-07-30 (all times UTC)
17:48 CircleCI deploys the latest change to production
17:51 Wisen notices that our website is not navigatable, some of the pages are blank
- This is the error in the browser console:
GET https://upmo.com/_nuxt/app.76279fd.js net::ERR_ABORTED 404
- Hard reloading didn't seem to help
- This is the error in the browser console:
17:53 The blank page issue is resolved
17:53 Google Autocomplete and Maps are showing error in the browser console
Google Maps JavaScript API error: RefererNotAllowedMapError https://developers.google.com/maps/documentation/javascript/error-messages#referer-not-allowed-map-error Your site URL to be authorized: https://upmo.com/offices?numOfPeople=20&minPrice=0&maxPrice=20000
17:53 Observes another error in the browser console related to Auth0:
VM1040:1 GET https://accept-upmo.eu.auth0.com/userinfo 401
- Our production seems to point to the wrong runtime configuration
17:55 Verified that the latest runtime config contains the correct Google Maps API Key in
__NUXT__.config
17:59 Synthetic monitoring is failing
17:58 Found the culprit for Auth0 issues in
nuxt.config.js
, where values are not taken from runtime config (hardcoded to accept environment)18:01 Code reverted and pushed
18:09 Found the culprit for Google maps in
nuxt.config.js
, where the API key is not taken from the runtime config (hardcoded to accept environment)- This is discovered from searching the non-prod API key in the browser's console
18:40 Production is reverted. Incident resolved.
18:47 Synthetic monitoring failure is resolved