Like most SaaS companies, ProntoForms relies on third party services like Amazon AWS for data storage needs. This has tremendous benefits for us and our customers – the growing Amazon S3 service is built on robust and reliable infrastructure, offering great performance and good value.
For many years, AWS has offered its customer impressive uptime rates, with only very rare and minor glitches in data availability or performance. However, on February 28 this year AWS experienced a significant service outage that took down many large internet sites around the world for several hours.
According to media reports, the company said in a blog post that “…one of its employees was debugging an issue with the billing system and accidentally took more servers offline than intended. That error started a domino effect that took down two other server subsystems and so on and so on.”
The same Amazon blog post stated: “Removing a significant portion of the capacity caused each of these systems to require a full restart. While these subsystems were being restarted, S3 was unable to service requests.”
Like most internet-based services in North-America, ProntoForms was affected by the outage, and as a result our customers were not able to successfully submit and access mobile form data.
Shortly after we noticed an issue the technical support team saw a sudden spike in support tickets from users getting error messages in the field.
“We saw a large number Pronto customers who wanted answers quickly on February 28,” said Scott Daly, Support Manager at ProntoForms.
“But replying to each ticket individually is inefficient and time consuming. I recommend that all ProntoForms customers subscribe to our Status Page at http://status.prontoforms.com. This is where we’ll post all updates with regards to the status of our service, including known technical issues,” he continued.
The Status Page was updated quickly after the AWS S3 outage, and customers who subscribed to the notification system were notified in minutes after the service went down.
Signing up for the notifications couldn’t be easier – just enter your email address and you’re done. (We will obviously never share your information with anybody else.)
In the Silver Lining department, ProntoForms customers saw no-to-minimal long-term disruptions from the Amazon outage. From a technical perspective, the ProntoForms solution was able to handle the situation gracefully:
- All data submissions were kept in Outboxes
- Once AWS S3 came back online, all submissions were automatically routed to their respective destination
- No data was lost
In the end, the robustness of our solution minimized the effects of the outage, so kudos to our R&D team!
