# 2021-09-20 Google maps api failure postmortem
Date: 2021-09-20
Summary: Upmo's location search and map was unusable for an hour due to incorrect usage of an environment variable.
Impact:
- Google autocomplete was unusable on our website.
- Google maps couldn't load on our website.
- Other google maps api based services, such as directions were unusable too.
Root causes: Nonprod environment variables getting injected in production (this relates back to Webapp Build Once Postmortem )
Resolution: Code revert
Detection: Manual + Synthetic monitoring suite
Action items:
| Action item | Type |
|---|---|
| Be more diligent while modifying environment variables used within nuxt config | process |
| Outline a quicker rollback strategy | mitigate |
| Avoid using production page in blog post assertion | process |
# Lessons Learned
# What went well
- Harsh manually verified things on prod immediately after deployment and realised things weren't right.
- We reverted our code immediately after manual verification.
- Our synthetic monitoring suite alarmed us as an additional verification measure.
# What went wrong
- Reverting our code broke tests, as some tests depended on production page verification.
- This issue couldn't be caught in nonprod, as it directly affects a specific environment.
# Where we got lucky
- We manually verified things immediately after deployment.
- We knew exactly what had gone wrong and how to fix it in time with surety.
# Timeline
2021-09-20 (all times UTC, unless explicity mentioned)
- 12:03 Our google maps script refactoring is deployed on production.
- 12:03 Harsh does a basic sanity check on production and discovers the incident.
- 12:04 Harsh investigates the root cause as an incorrect api key being configured in webapp.
- 12:05 Harsh pushes the reverted commit to resolve the issue.
- 12:36 Our automation test fails due to an assertion against an already broken production environment.
- 12:40 Harsh pushes the modified buildscript to temporarily skip our tests.
- 13:13 The reverted commit is deployed to production.
- 13:13 Harsh and Deen verify everything is working normally in production.