# 2020-12-09 Missing virtual tour rendering postmortem
Date: 2020-12-09
Summary: For some building that did not have a virtual tour, the webapp showed a blank section in the virtual tour area.
Impact:
- Some buildings/spaces that did not have a virtual tour showed blank virtual tours on the buildings page
- Some buildings/spaces that did not have a virtual tour showed a virtual tour icon on the offices page.
Root causes: Having more than one way to define non existing virtual tours, using undefined, and empty virtual tour path.
Resolution: Code fix to use a singular method of identification for non existing virtual tours.
Detection: Manual
Action items:
| Action item | Type |
|---|---|
| Fix the way we identify missing virtual tours in provider portal and som | prevent |
| Automated identification of cases where virtual tours fail to load. | mitigate |
# Lessons Learned
# What went well
- We were able to quickly identify such an incident.
- We knew what had gone wrong quite soon.
# What went wrong
- There is no way to identify such incidents using automated tests.
- While the fix was quick, it took us hours to push it to prod because of deployment issues (open street maps server was down)
# Where we got lucky
- We just happened to verify one such space manually after bulk import.
- Not a lot of active users
- The number of buildings impacted was low.
# Timeline
2020-12-09 (all times UTC)
- 16:30 Rosie imported spaces using bulk import functionality in pro. Certain buildings that were newly added/edited in pro started showing up in som.
- 16:37 Rosie and Ali noticed a space which showed blank virtual tour.
- 16:39 Harsh confirmed that the virtual tour path for that space was null/"".
- 16:39 Rosie notified another such building to Harsh
- 16:39 Harsh confirmed the same for the building that Rosie notified about.
- 16:46 Ali notified Wisen and Nick on the channel about the issue.
- 17:05 Harsh verifies the fix on local and checks with Wisen if he can revert a commit to unblock the build.
- 17:08 Wisen pushes a fix to unblock build.
- 17:20 Harsh pushed a fix for the issue.
- 17:30 19:20 - OSM keeps on stalling the build (OSM Endpoints kept throwing timeouts and [503 Service Unavailable]
- 19:30 Accept test passed
- 19:39 Fix gets deployed to prod