Animoto Video
Lessons learned First of all, when you scale 10x and then 10x again to run on thousands of servers every little problem turns into a large one. So suddenly its not something you can ignore anymore. An example for this was having exponential backoff for uploads to S3 when using curl, but forgetting that the 5th retry exceeds the S3 connection timeout. Once we changed this to a constant retry timeout it all went smooth again.
Sources:
- blog.rightscale.com
- www.boxoftricks.net
- www.garagespin.com
- www.cottontimer.com
- www.the-newsroom.com
- jasoncypret.com
Tags: animoto, exponential backoff, servers, uploads