Partial stats outage
Incident Report for Wistia
Resolved
We have finished processing the backlog of stats data. At this point, stats should be current for all customers. Thank you for your patience.

We will continue to monitor our systems to ensure that stats are functioning properly.
Posted Oct 01, 2021 - 12:43 EDT
Update
We continue to make progress on the backlog of stats data. We had processed about 50% of the backlog at 12pm (ET) today, and we expect to see full recovery sometime on Friday, Oct 1.

All customers should be able to use the stats API, but please be aware that stats data may not be available for date ranges starting on 9/26 until the system fully recovers.

We will update the ETA for full recovery as we gather more information.
Posted Sep 30, 2021 - 17:16 EDT
Update
Recovery continues. We expect to see full recovery of stats data for all customers by no later than Saturday, October 2. In the meantime, all customers should be able to use the stats API, but please be aware that stats data may not be available for date ranges starting on 9/26 until the system fully recovers.

We will update the ETA for full recovery as we gather more information.
Posted Sep 29, 2021 - 17:31 EDT
Monitoring
We are continuing to work towards restored stats service. A new database has been provisioned, and we have begun processing the backlog of stats data since Sunday night (9/26). We will provide an ETA for full recovery later today.

Customers who have been receiving stats API errors should now be able to use the API, but please be aware that stats data may not be available for date ranges starting on 9/26 until the system fully recovers.

We are also working on a fix that will allow all customers to use the Media Stats page while the recovery process continues.
Posted Sep 29, 2021 - 13:13 EDT
Update
We are continuing to work on a fix for this issue. Thank you for your patience. At this time, incoming stats data is still being collected for all customers, and we do not believe that any data has been lost.

Last night, an issue with our cloud provider caused one of our stats databases to fail. While we have backup copies of its data, restoring the database itself will take time. In the meantime, customers whose stats data is housed on that database will receive an error if they try to use the stats API. We apologize for any inconvenience this is causing.

CORRECTION: This update incorrectly stated that customers whose stats data is housed on other databases were not affected. Those customers may have been unable to access the Media Stats page.

We will continue to provide updates and will provide an ETA for restored stats service for affected customers when we can. The earliest that at ETA will be provided is tomorrow, Sept 28.
Posted Sep 27, 2021 - 17:06 EDT
Identified
The issue has been identified and a fix is being implemented.
Posted Sep 27, 2021 - 11:32 EDT
Investigating
Due to an incident with AWS, some customers may be unable to reach our stats endpoints. Stats processing has also been delayed. We're continuing to monitor the AWS incident and will update accordingly.
Posted Sep 27, 2021 - 00:26 EDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Sep 27, 2021 - 00:25 EDT
Investigating
Due to an incident with AWS, some customers may be unable to reach our stats endpoints. Stats processing has also been delayed. We're continuing to monitor the AWS incident and will update accordingly.
Posted Sep 27, 2021 - 00:25 EDT
This incident affected: Stats.