Summary
On 9th February around 12PM, we were informed by several clients that they were not able to login to Interprefy EMI. The team started investigating the issue immediately.
Cause
After investigation, we have concluded that a database view was timing out the requests. Due to this, the clients were receiving an “Undefined error”. The team had executed a planned upgrade on the same database view before and therefore the root cause was immediately identified.
What have we done
After identification of root cause, we have immediately rolled back the database view changes. After this action, we have re-tested the service successfully and informed clients that the service was resumed.
Preventive actions
As preventive action, we have immediately rolled back the changes on the database.
Corrective action
As corrective action, we have improved our development process regarding database views. A different development and rollout strategy is defined to create and rename database views making sure that these issues do not re-occur.
Resolution date
On 9th February around 2PM, we have considered this issue closed.