# 2020-12-09 Missing virtual tour rendering postmortem

Date: 2020-12-09

Summary: For some building that did not have a virtual tour, the webapp showed a blank section in the virtual tour area.

Impact:

  • Some buildings/spaces that did not have a virtual tour showed blank virtual tours on the buildings page
  • Some buildings/spaces that did not have a virtual tour showed a virtual tour icon on the offices page.

Root causes: Having more than one way to define non existing virtual tours, using undefined, and empty virtual tour path.

Resolution: Code fix to use a singular method of identification for non existing virtual tours.

Detection: Manual

Action items:

Action item Type
Fix the way we identify missing virtual tours in provider portal and som prevent
Automated identification of cases where virtual tours fail to load. mitigate

# Lessons Learned

# What went well

  • We were able to quickly identify such an incident.
  • We knew what had gone wrong quite soon.

# What went wrong

  • There is no way to identify such incidents using automated tests.
  • While the fix was quick, it took us hours to push it to prod because of deployment issues (open street maps server was down)

# Where we got lucky

  • We just happened to verify one such space manually after bulk import.
  • Not a lot of active users
  • The number of buildings impacted was low.

# Timeline

2020-12-09 (all times UTC)

  1. 16:30 Rosie imported spaces using bulk import functionality in pro. Certain buildings that were newly added/edited in pro started showing up in som.
  2. 16:37 Rosie and Ali noticed a space which showed blank virtual tour.
  3. 16:39 Harsh confirmed that the virtual tour path for that space was null/"".
  4. 16:39 Rosie notified another such building to Harsh
  5. 16:39 Harsh confirmed the same for the building that Rosie notified about.
  6. 16:46 Ali notified Wisen and Nick on the channel about the issue.
  7. 17:05 Harsh verifies the fix on local and checks with Wisen if he can revert a commit to unblock the build.
  8. 17:08 Wisen pushes a fix to unblock build.
  9. 17:20 Harsh pushed a fix for the issue.
  10. 17:30 19:20 - OSM keeps on stalling the build (OSM Endpoints kept throwing timeouts and [503 Service Unavailable]
  11. 19:30 Accept test passed
  12. 19:39 Fix gets deployed to prod