# 2020-09-23 Webapp Blank Building Page Postmortem
Date: 2020-09-23
Summary: The building page was inaccessible/blank for 1 hour & 2 minutes due to change introduced to support virtual tour for a building
Impact:
- User's can't view a building as the building page was unable to render
- Users can't proceed to book a viewing as the building page was inaccessible.
Root causes: In order to support virtual tour for a building, our webapp expects a key from algolia document. Due to that missing key, the building page got broken.
Resolution: Ran the refresh all records script to update the algolia document.
Detection: Manual + synthetic monitoring
Action items:
| Action item | Type |
|---|---|
| Fail build if there is any error while building static page | prevent |
| Deploy the latest data first and don't break backward compatibility before making UI changes | mitigate |
| Look for incident failure on priority | process |
| Establish rotation plan in pager duty to fix flakiness | process |
# Lessons Learned
# What went well
- Synthetic monitoring did fail & raised alert on our incident slack channel.
- We spend minimum time in production to learn about the issue
- Algolia refresh script is very fast and took very minimal time.
# What went wrong
- No sanity on staging & prod after deployment
- Acceptance tests raised no errors
- Build did not fail while building pages although it had type error in the console.
- Not actively looking into the incidents channel
- Harsh going to production
# Where we got lucky
- Not a lot of active users
- Algolia refresh didn't required a rebuild of webapp. Don't know why ?
# Timeline
2020-09-23 (all times UTC)
- 11:02 CircleCI deploys the latest change to production
- 11:06 Synthetic monitoring raised an alert
- 11:56 Harsh notifed on slack channel that prod is down
- Harsh shared a screenshot in which building page was blank.
- This is the error in the browser console:
Cannot read property 'path' of undefined - Hard reloading didn't seem to help
- 11:57 Aman replied on the thread by Harsh and investigated the error.
- use of path was introduced for virtual tour, hence the following solutions
came around. Either of them can be used to solve the problem:
- Revert the code change on the building page for virtual tour
- Run the algolia refresh
- use of path was introduced for virtual tour, hence the following solutions
came around. Either of them can be used to solve the problem:
- 12:01 Aman ran agolia refresh for staging & prod
- 12:03 Aman did the sanity on staging & prod and verifed that everyhting is working.
- 12:06 Incident & synthetic monitoring failure is resolved