Degradation in performance of GiveGab API
Incident Report for GiveGab
Resolved
On Friday November 22, 2019 from 7:53 PM ET until 10:41 PM ET, the GiveGab API experienced degraded performance which impacted both the core GiveGab year-round platform (www.givegab.com) as well as our giving day platform (e.g. www.extragive.org).

During this time, users may have seen slow response times and request timeouts as certain pieces of content such as donation stats as well as some pages, such as the GiveGab admin dashboard may not have loaded. Some users experience failure errors when donating, which may have led some donors to try multiple times, which may have resulted in multiple unintended charges to them.

At roughly 10:41 PM ET all systems fully recovered with normal response times.

There were several factors that contributed to this performance degradation. The main issue was that our high-performance caching mechanism failed for a critical API endpoint. This overloaded the API and as we auto-scaled exacerbated the issue as the API could not efficiently access the database.

This caching issue has since been addressed. For those users that may have donated multiple times out of error, the team is currently analyzing and processing refunds for those cases.

Additionally, the team is undertaking a number of other activities to ensure that this does not happen again. At this time we are still investigating the abrupt nature of the degradation in performance to see what triggered this when our monitoring should have caught a ramp up to the issue and automatically alerted us.
Posted Nov 22, 2019 - 19:30 EST