We’ve recently completed development of a hosted version of our SCORM Engine. In the coming weeks we will be transitioning TestTrack over to using the hosted Engine to enable much greater scalability than the single server install can currently provide. This project will involved liberal use of several of the Amazon Web Services, due largely to their ease of use, low cost and high scalability. Using the Elastic Compute Cloud (EC2) was an obvious choice for us, as we can easily create and destroy machines as our load fluctuates. Less obvious, however, was how we should go about storing content files. These were our requirements for a storage device:
1) easy ability to upload large files using standard FTP clients, and possibly other commonly available protocols. Files over 1 GB aren’t uncommon, and larger files than that are certainly possible.
2) real-time file updates (i.e. when I upload a new version of a file and then immediately request it, there should be no chance that I get back an older version)
3) small amount of storage for now (so it’s cheap) with the ability to grow to a few TB or more as demand requires it
4) storage should all be accessible at a single root location. We currently have files spread over two drives, which requires a small bit of one-off code to determine which files from which users go on which drive.
5) ability to access the content directly from any one of (potentially) several web servers
6) some form of backup and/or redundancy to prevent data loss
Amazon currently provides two different mechanisms for persistent storage: Simple Storage Service (S3) and Elastic Block Store (EBS). Each of these storage methods has its advantages, but at first look, neither will fulfill all of our requirements straight out of the box.