Amazon Web Services (AWS) - Ongoing Issues

Incident Report for Trinity High School

Postmortem

From Amazon/AWS: Between 11:49 PM PDT on October 19 and 2:24 AM PDT on October 20, we experienced increased error rates and latencies for AWS Services in the US-EAST-1 Region. Additionally, services or features that rely on US-EAST-1 endpoints such as IAM and DynamoDB Global Tables also experienced issues during this time. At 12:26 AM on October 20, we identified the trigger of the event as DNS resolution issues for the regional DynamoDB service endpoints. After resolving the DynamoDB DNS issue at 2:24 AM, services began recovering but we had a subsequent impairment in the internal subsystem of EC2 that is responsible for launching EC2 instances due to its dependency on DynamoDB. As we continued to work through EC2 instance launch impairments, Network Load Balancer health checks also became impaired, resulting in network connectivity issues in multiple services such as Lambda, DynamoDB, and CloudWatch. We recovered the Network Load Balancer health checks at 9:38 AM. As part of the recovery effort, we temporarily throttled some operations such as EC2 instance launches, processing of SQS queues via Lambda Event Source Mappings, and asynchronous Lambda invocations. Over time we reduced throttling of operations and worked in parallel to resolve network connectivity issues until the services fully recovered. By 3:01 PM, all AWS services returned to normal operations. Some services such as AWS Config, Redshift, and Connect continue to have a backlog of messages that they will finish processing over the next few hours.

Posted Oct 20, 2025 - 19:35 EDT

Resolved

The issues with AWS have been resolved. PowerSchool reports that Schoology is functioning normally again.

Although Amazon has fixed the issues on their end with AWS, note that some sites, services, and apps may continue to have lingering issues as the individual companies perform their own recoveries. We would expect that, generally speaking, services across the internet will be fully functional by Tuesday morning (21 October), but there could be exceptions.
Posted Oct 20, 2025 - 19:34 EDT

Update

An update from PowerSchool regarding the issues with Schoology:

"We are continuing to monitor the situation closely as our cloud provider, AWS, works through mitigation efforts. While there are signs of progress, full recovery is expected to take additional time. As a result, users may continue to experience disruptions across impacted services until the issue is fully resolved. We appreciate your continued patience and will provide further updates as they become available."
Posted Oct 20, 2025 - 15:11 EDT

Identified

AWS has begun deploying mitigations to the affected systems. It is likely this will take some time (hours).
Posted Oct 20, 2025 - 14:11 EDT

Investigating

AWS continues to have widespread issues affecting many online services, including Schoology (Rockspace). It appears things took another turn for the worse, as PowerSchool has again marked Schoology as "degraded" instead of "operational" as it was earlier in the day. Many other services are affected as well, including Canva, Adobe Creative Cloud, and hundreds/thousands of other sites and services.
Posted Oct 20, 2025 - 13:46 EDT
This incident affected: Websites (Rockspace).