# 2021-05-14 Price range not getting updated postmortem
Date: 2021-05-14
Summary: Price range was updating erratically on production environment
Impact:
- Users could not view the price range given a search criteria
- Users were under the false impression that price range was still loading, even though it had failed to update.
Root causes: AWS Appsync was down globally for an hour.
Resolution: Monitor AWS updates about App Sync.
Detection: Manual
Action items:
| Action item | Type |
|---|---|
| Verify Price range in synthetic monitoring | process |
| Update cloudwatch alarms | process |
# Lessons Learned
# What went well
- Ali identified the issue on production early on.
- We spent minimum time to understand what actually went wrong.
- Our acceptance test suite gave us the confidence that code wasn't broken.
# What went wrong
- Synthetic monitoring did not catch this.
# Where we got lucky
- AWS is quite prompt at resolving issues with their infrastructure.
- Not a lot of active users
# Timeline
2021-05-14 (all times UTC, unless explicity mentioned)
- 16:05 Ali notifies price range is loading intermittently on prod.
- 16:06 Harsh verifies and confirms he sees the same consistently on prod. The price range text was stuck at Loading... and the browser console showed 500 errors.
- 16:06 Harsh navigates to graphql lambda logs and doesn't find any. This confirms our service wasn't getting hit at all. Potential cause could be CORS errors from the API.
- 16:07 Harsh tries again on staging and it works.
- 16:07 Harsh tries again on production and it works.
- 16:08 Harsh navigates to AWS notification console and finds a notification from AWS:
Event- Appsync API issue - 16:08 Harsh notifies this to the team and waits for further notifications form AWS.
- 16:09 Harsh randomly verifies prod again and finds it's still working. The issue is still open with AWS.
- 20:30 (PDT) - AWS confirms the issue is resolved from their end