Database performance degradation — March 30–31, 2026
Resolved
Mar 31, 2026 at 7:29pm UTC
Resolved. All problematic database queries have been identified and replaced with optimized alternatives. Enhanced monitoring has been deployed to detect performance degradation earlier. No further action required.
Affected services
Created
Mar 31, 2026 at 7:29pm UTC
On March 30 and March 31, 2026, absentify experienced two periods of degraded performance.
Incident 1: March 30, approximately 07:00–10:00 UTC (~3 hours)
Incident 2: March 31, approximately 07:00–08:00 UTC (~1 hour)
What caused it
A recent software update introduced a database query that turned out to be significantly more resource-intensive than expected due to a bug in a third-party library we use. Under peak morning traffic (European business hours), this query consumed excessive database resources, causing slow response times across all application features.
After resolving the first incident, we re-introduced a limited version of the problematic query, believing the reduced scope would be safe. This assumption proved incorrect, and the issue recurred the following morning under similar traffic conditions.
Was any data lost?
No. No data was lost or compromised during either incident. The issue was purely related to performance and availability.
What we did to fix it
- Immediate fix: We identified and removed the problematic query within hours of each incident, restoring normal performance.
- Comprehensive review: We conducted a full review of our codebase and replaced all instances of the problematic query pattern with optimized alternatives.
- Improved monitoring: We enhanced our alerting system to detect similar issues earlier and ensure notifications are properly delivered to our operations team.
- Prevention measures: We implemented safeguards to prevent this type of query from being reintroduced in future updates.
Our commitment
We sincerely apologize for the inconvenience caused by these disruptions. We understand that you rely on absentify for your day-to-day operations, and we take service availability very seriously.
The measures we have implemented address both the immediate cause and the systemic factors that allowed this issue to occur. We are confident these changes will prevent similar incidents in the future.
Questions or concerns?
If you have any questions about this incident or need further information, please contact our support team at support@absentify.com
Affected services