Amazon SimpleDBThis excerpt is from the book, 'A Developer's Guide to Amazon SimpleDB', authored by Mocky Habeeb, published by Pearson/Addison-Wesley Professional, Aug. 2010, ISBN 0321623630, copyright 2011 Pearson Education, Inc."
Amazon has been offering its customers computing infrastructure via Amazon Web Services (AWS) since 2006.AWS aims to use its own infrastructure to provide the building blocks for other organizations to use.The Elastic Compute Cloud (EC2) is an AWS offering that enables you to spin up virtual servers as you need the computing power and shut them off when you are done.Amazon Simple Storage Service (S3) provides fast and unlimited file storage for the web.Amazon SimpleDB is a service designed to complement EC2 and S3, but the concept is not as easy to grasp as “extra servers” and “extra storage.” This chapter will cover the concepts behind SimpleDB and discuss how it compares to other services.
What Is SimpleDB?SimpleDB is a web service providing structured data storage in the cloud and backed by clusters of Amazon-managed database servers.The data requires no schema and is stored securely in the cloud.There is a query function, and all the data values you store are fully indexed. In keeping with Amazon’s other web services, there is no minimum charge, and you are only billed for your actual usage.
What SimpleDB Is NotThe name “SimpleDB” might lead you to believe that it is just like relational database management systems (RDBMS), only simpler to use. In some respects, this is true, but it is not just about making simplistic database usage simpler. SimpleDB aims to simplify the much harder task of creating and managing a database cluster that is fault-tolerant in the face of multiple failures, replicated across data centers, and delivers high levels of availability.
One misconception that seems to be very common among people just learning about SimpleDB is the idea that migrating from an RDBMS to SimpleDB will automatically solve your database performance problems. Performance certainly is an important part of 2 Chapter 1 Introducing Amazon SimpleDB the equation when you seek to evaluate databases. Unfortunately, for some people, speed is the beginning and the end of the thought process. It can be tempting to view any of the new hosted database services as a silver bullet when offered by a mega-company like Microsoft,Amazon, or Google. But the fact is that SimpleDB is not going to solve your existing speed issues.The service exists to solve an entirely different set of problems. Reads and writes are not blazingly fast.They are meant to be “fast enough.” It is entirely possible that AWS may increase performance of the service over time, based on user feedback. But SimpleDB is never going to be as speedy as a standalone database running on fast hardware. SimpleDB has a different purpose.
Robust database clusters replicating data across multiple data centers is not a data storage solution that is typically easy to throw together. It is a time consuming and costly undertaking. Even in organizations that have the database administrator (DBA) expertise and are using multiple data centers, it is still time consuming. It is costly enough that you would not do it unless there was a quantifiable business need for it. SimpleDB offers data storage with these features on a pay-as-you-go basis.
Of course, taking advantage of these features is not without a downside. SimpleDB is a moderately restrictive environment, and it is not suitable for many types of applications. There are various restrictions and limitations on how much data can be stored and transferred and how much network bandwidth you can consume.
Schema-Less DataSimpleDB differs from relational databases where you must define a schema for each database table before you can use it and where you must explicitly change that schema before you can store your data differently. In SimpleDB, there is no schema requirement. Although you still have to consider the format of your data, this approach has the benefit of freeing you from the time it takes to manage schema modifications. The lack of schema means that there are no data types; all data values are treated as variable length character data. As a result, there is literally nothing extra to do if you want to add a new field to an existing database.You just add the new field to whichever data items require it.There is no rule that forces every data item to have the same fields. The drawbacks of a schema-less database include the lack of automatic integrity checking in the database and an increased burden on the application to handle formatting and type conversions. Detailed coverage of the impact of schema-less data on queries appears in Chapter 4,“A Closer Look at Select,” along with a discussion of the formatting issues.
Stored Securely in the CloudThe data that you store in SimpleDB is available both from the Internet and (with less latency) from EC2.The security of that data is of great importance for many applications, while the security of the underlying web services account should be important to all users.
To protect that data, all access to SimpleDB, whether read or write, is protected by your account credentials. Every request must bear the correct and authorized digital signature or else it is rejected with an error code. Security of the account, data transmission, and data storage is the subject of Chapter 8,“Security in SimpleDB-Based Applications.”
Billed Only for Actual UsageIn keeping with the AWS philosophy of pay-as-you-go, SimpleDB has a pricing structure that includes charges for data storage, data transfer, and processor usage.There are no base fees and there are no minimums.At the time of this writing,Amazon’s monthly billing for SimpleDB has a free usage tier that covers the first gigabyte (GB) of data storage, the first GB of data transfer, and the first 25 hours of processor usage each month. Data transfer costs beyond the free tier have historically been on par with S3 pricing, whereas storage costs have always been somewhat higher. Consult the AWS website at for current pricing information.
Domains, Items, and Attribute PairsThe top level of data storage in SimpleDB is the domain.A domain is roughly analogous to a database table.You can create and delete domains as needed.There are no configuration options to set on a domain; the only parameter you can set is the name of the domain.
All the data stored in a SimpleDB domain takes the form of name-value attribute pairs. Each attribute pair is associated with an item, which plays the role of a table row. The attribute name is similar to a database column name but unlike database rows that must all have identical columns, SimpleDB items can each contain different attribute names.This gives you the freedom to store different data in some items without changing the layout of other items that do not have that data. It also allows the painless addition of new data fields in the future.
Multi-Valued AttributesIt is possible for each attribute to have not just one value, but an array of values. For example, an application that allows user tagging can use a single attribute named “tags” to hold as many or as few tags as needed for each item.You do not need to change a schema definition to enable multi-valued attributes. All you need to do is add another attribute to an item and use the same attribute name with a different value.This provides you with flexibility in how you store your data.
QueriesSimpleDB is primarily a key-value store, but it also has useful query functionality.A SQLstyle query language is used to issue queries over the scope of a single domain.A subset of the SQL select syntax is recognized.The following is an example SimpleDB select statement:
You put a domain name—in this case, products—in the FROM clause where a table name would normally be.The WHERE clause recognizes a dozen or so comparison operators, but an attribute name must always be on the left side of the operator and a literal value must always be on the right.There is no relational comparison between attributes allowed here. So, the following is not valid:
All the data stored in SimpleDB is treated as plain string data.There are no explicit indexes to maintain; each value is automatically indexed as you add it.
High AvailabilityHigh availability is an important benefit of using SimpleDB.There are many types of failures that can occur with a database solution that will affect the availability of your application. When you run your own database servers, there is a spectrum of different configurations you can employ.
To help quantify the availability benefits that you get automatically with SimpleDB, let’s consider how you might achieve the same results using replication for your own database servers.At the easier end of the spectrum is a master-slave database replication scheme, where the master database accepts client updates and a second database acts as a slave and pulls all the updates from the master.This eliminates the single point of failure. If the master goes down, the slave can take over. Managing these failures (when not using SimpleDB) requires some additional work for swapping IP addresses or domain name entries, but it is not very difficult.
Moving toward the more difficult end of the self-managed replication spectrum allows you to maintain availability during failure that involves more than a single server.There is more work to be done if you are going to handle two servers going down in a short period, or a server problem and a network outage, or a problem that affects the whole data center. Creating a database solution that maintains uptime during these more severe failures requires a certain level of expertise. It can be simplified with cloud computing services like EC2 that make it easy to start and manage servers in different geographical locations. However, when there are many moving parts, the task remains time consuming. It can also be expensive.
When you use SimpleDB, you get high availability with your data replicated to different geographic locations automatically.You do not need to do any extra work or become an expert on high availability or the specifics of replication techniques for one vendor’s database product.This is a huge benefit not because that level of expertise is not worth attaining, but because there is a whole class of applications that previously could not justify that effort.
Database ConsistencyOne of the consequences of replicating database updates across multiple servers and data centers is the need to decide what kind of consistency guarantees will be maintained.A database running on a single server can easily maintain strong consistency.With strong consistency, after an update occurs, every subsequent database access by every client reflects the change and the previous state of the database is never seen.
This can be a problem for a database cluster if the purpose of the cluster is to improve availability. If there is a master database replicating updates to slave databases, strong consistency requires the slaves to accept the update at the same time as the master. All access to the database would then be strongly consistent. However, in the case of a problem preventing communication between the master and a slave, the master would be unable to accept updates because doing so out of sync with a slave would break the consistency guarantee. If the database rejects updates during even simple problem scenarios, it defeats the availability. In practice, replication is often not done this way.A common solution to this problem is to allow only the master database to accept updates and do so without direct contact with any slave databases. After the master commits each transaction, slaves are sent the update in near real-time.This amounts to a relaxing of the consistency guarantee. If clients only connect to the slave when the master goes down, then the weakened consistency only applies to this scenario.
SimpleDB sports the option of either eventual consistency or strong consistency for each read request.With eventual consistency, when you submit an update to SimpleDB, the database server handling your request will forward the update to the other database servers where that domain is replicated.The full update of all replicas does not happen before your update request returns.The replication continues in the background while other requests are handled.The period of time it takes for all replicas to be updated is called the eventual consistency window.The eventual consistency window is usually small.AWS does not offer any guarantees about this window, but it is frequently less than one second.
A couple things can make the consistency window larger. One is a high request load. If the servers hosting a given SimpleDB domain are under heavy load, the time it takes for full replication is increased.Additionally a network or server failure can block replication until it is resolved. Consider a network outage between data centers hosting your data. If the SimpleDB load-balancer is able to successfully route your requests to both data centers, your updates will be accepted at both locations. However, replication will fail between the two locations.The data you fetch from one will not be consistent with updates you have applied to the other. Once the problem is fixed, SimpleDB will complete the replication automatically.
Using a consistent read eliminates the consistency window for that request.The results of a consistent read will reflect all previous writes. In the normal case, a consistent read is no slower than an eventually consistent read. However, it is possible for consistent read requests to display higher latency and lower bandwidth on occasion.
Sizing Up the SimpleDB Feature SetThe SimpleDB API exposes a limited set of features. Here is a list of what you get:
Benefits of Using SimpleDBWhen you use SimpleDB, you give up some features you might otherwise have, but as a trade-off, you gain some important benefits, as follows:
Database Features SimpleDB Doesn’t HaveThere are a number of common database features noticeably absent from Amazon SimpleDB. Programs based on relational database products typically rely on these features.You should be aware of what you will not find in SimpleDB, as follows:
Higher-Level Framework FunctionalityThis simplicity of what SimpleDB offers on the server side is matched by the simplicity of what AWS provides in officially supported SimpleDB clients.There is a one-to-one mapping of service features to client calls.There is a lot of functionality that can be built atop the basic SimpleDB primitives. In addition, the inclusion of these advance features has already begun with a number of third-party SimpleDB clients. Popular persistence frameworks used as an abstraction layer above relational databases are prime candidates for this.
Some features normally included within the database server can be written into SimpleDB clients for automatic handling.Third-party client software is constantly improving, and some of the following features may be present already or you may have to write it for yourself:
Service LimitsThere are quite a few limitations on what you are allowed to do with SimpleDB. Most of these are size and quantity restrictions.There is an underlying philosophy that small and quickly serviced units of work provide the greatest opportunity for load balancing and maintaining uniform service levels. AWS maintains a current listing of the service limitations within the latest online SimpleDB Developer Guide at the AWS website. At the time of this writing, the limits are as follows:
These limits may seem restrictive when compared to the unlimited nature of data sizes you can store in other database offerings. However, there are two things to keep in mind about these limits. First, SimpleDB is not a general-purpose data store suitable for everything. It is specifically designed for storing small chunks of data. For larger data objects that you want to store in the cloud, you are advised to use Amazon S3. Secondly, consider the steps that need to be taken with a relational database at higher loads when performance begins to degrade.Typical recommendations often include offloading processing from the database, reducing long-running queries, and applying selective de-normalization of the data.These limits are what help enable efficient and automatic background replication and high concurrency and availability. Some of these limits can be worked around to a degree, but no workarounds exist for you to make SimpleDB universally appropriate for all data storage needs.
Abandoning the Relational Model?There have been many recent products and services offering data storage but rejecting the relational model.This trend has been dubbed by some as the NoSQL movement.There is a fair amount of enthusiasm both for and against this trend.A few of those in the “against” column argue that databases without schemas, type checking, normalization, and so on are throwing away 40 years of database progress. Likewise, some proponents are quick to dispense the hype about how a given NoSQL solution will solve your problems. The aim of this section is to present a case for the value of a service like SimpleDB that addresses legitimate criticism and avoids hype and exaggeration.
A Database Without a SchemaOne of the primary areas of contention around SimpleDB and other NoSQL solutions centers on the lack of a database schema. Database schemas turn out to be very important in the relational model.The formalism of predefining your data model into a schema provides a number of specific benefits, but it also imposes restrictions.
SimpleDB has no notion of a schema at all. Many of the structures defined in a typical database schema do not even exist in SimpleDB.This includes things such as stored procedures, triggers, relationships, and views. Other elements of a database schema like fields and types do exist in SimpleDB but are flexible and are not enforced on the server. Still other features, like indexes, require no formal definition because the SimpleDB service creates and manages them behind the scenes.
However, the lack of a schema requirement in SimpleDB does not prevent you from gaining the benefits of a schema.You can create your own schema for whatever portion of your data model that is appropriate.This allows you to cherry-pick the benefits that are helpful to your application without the unneeded restrictions.
One of the most important things you gain from codifying your data layout is a separation between it and the application.This is an enabling feature for tools and application plug-ins.Third-party tools can query your data, convert your data from one format to another, and analyze and report on your data based solely on the schema definition.The alternative is less attractive.Tools and extensions are more limited in what they can do without knowledge of the formats. For example, you cannot compute the sum of values in a numeric column if you do not know the format of that column. In the degenerate case, developers must search through your source code to infer data types.
In SimpleDB, many of the most common database features are not available. Query, however, is one important feature that is present and has some bearing on your data formatting. Because all the data you store in SimpleDB is variable length character data, you must apply padding to numeric data in order for queries to work properly. For example, if you want to store an attribute named “price” with a value of “269.94,” you must first add leading zeros to make it “00000269.94.”This is required because greater-than and lessthan comparisons within SimpleDB compare each character from left to right. Padding with zeros allows you to line up the decimal point so the comparisons will be correct for all possible values of that attribute. Relational database products handle this for you behind the scenes when you declare a column type is a numeric type like int.
This is a case in SimpleDB where a schema is beneficial.The code that initially imports records into SimpleDB, the code that writes records as your app runs, and any code that uses a numeric attribute in a query all need to use the exact same format. Explicitly storing the schema externally is a much less error-prone approach than implicitly defining the format in duplicated code across various modules. Another benefit of the predefined schema in the relational model is that it forces you to think through the data relationships and make unambiguous decisions about your data layout. Sometimes, however, the data is simple, there are no relationships, and creating a data model is overkill. Sometimes you may still be in the process of defining the data model. SimpleDB can be used as part of the prototyping process, enabling you to evolve your schema dynamically as issues surface that may not otherwise have become known so quickly.You may be migrating from a different database with an existing data model.The important thing to remember is that SimpleDB is simple by design. It can be useful in a variety of situations and does not prevent you from creating your own schema external to SimpleDB.
Areas Where Relational Databases StruggleRelational databases have been around for some time.There are many robust and mature products available. Modern database products offer a multitude of features and a host of configuration options.
One area where difficulty arises is with database features that you do not need or that you should not use for a particular application. Applications that have simple data storage requirements do not benefit from the myriad of available options. In fact, it can be detrimental in a couple different ways. If you need to learn the intricacies of a particular database product before you can make good use of it, the time spent learning takes away from time you could have spent on your application. Knowledge of how database products work is good to have. It would be hard to argue that you wasted your time by learning it because that information could serve you well far into the future. Similarly, if there is a much simpler solution that meets your needs, you could choose that instead. If you had no immediate requirement to gain product specific database expertise, it would be hard to insist that you made the wrong choice. It is a tough sell to argue that the more time-consuming, yet educational, route is always better than the simple and direct route.This is a challenge faced by databases today, when the simple problems are not met with simple solutions.
Another pain point with relational databases is horizontal scaling. It is easy to scale a database vertically by beefing up your server because memory and disk drives are inexpensive. However, scaling a database across multiple servers can be extremely difficult. There is a whole spectrum of options available for horizontal scaling that includes basic master-slave replication as well as complicated sharding strategies.These solutions each require a different, and sometimes considerable, amount of expertise.Nevertheless, they all have one thing in common when compared to vertical scaling solutions. On top of the implementation difficulty, each additional server results in an additional increase in ongoing maintenance responsibility. Moreover, it is not merely the additional server maintenance of having more servers. I am referring to the actual database administration tasks of managing additional replicas, backups, and log shipping. It also includes the tasks of rolling out schema changes and new indexes to all servers in the cluster.
If you are in a situation where you want a simple database solution or you want horizontal scaling, SimpleDB is definitely a service to consider.However, you may need to be prepared to defend your decision.
Scalability Isn’t Your ProblemAround every corner, you can find people who will challenge your efforts to scale horizontally. Beyond the cost and difficulty, there is a degree of resistance to products and services that seek to solve these problems.
The typical, and now clichéd, advice tends to be that scalability is not your problem, and trying to solve scalability at the outset is a case of premature optimization.This is followed by a discussion of how many daily page views a single high-performance database server can support. Finally, it ends by noting that it is really just a problem for when you reach the scale of Google or Amazon.
The premise of the argument is actually solid, although not applicable to all situations. The premise is that when you are building a site or service that nobody has heard of yet, you are more concerned about handling loads of people than about making the site remarkable. It is good advice for these situations. Moreover, it is especially timely considering that there is a small but religious segment of Internet commentators who eagerly chime,“X doesn’t scale,” where X is any alternative to the solution the commenter uses. Among programmers, there is a general preoccupation with performance optimization that seems somewhat out of balance.
The fact is that for many projects, scalability really is not your problem, but availability can be.Distributing your data store across servers from the outset is not a premature optimization when you can quantify the cost of down time. If a couple hours of downtime will have an impact on your business, then availability is something worth thinking about. For the IT department delivering a mission-critical application, availability is important. Even if only 20 users will use it during normal business hours, when it provides a competitive advantage, it is important to maintain availability through expected outages. When you have a product launch, and your credibility is at stake as much as your revenue, you are not putting the cart before the horse when you protect yourself against hardware failures.
There are many situations where availability is an important system quality. Look at how common it is for a multi-server web cluster to host one website. Before you can add a second web server, you must first solve a small set of known problems. User sessions have to be managed properly; load balancing has to be in place and routing around unresponsive servers.However,web server clusters are useful for more than high-traffic load handling.They are also beneficial because we know that hardware will fail, and we want to maintain service during the failure.We can add another web server because it is neither costly nor difficult, and it improves the availability.With the advent of systems designed to provide higher database availability that are not costly nor hard, availability becomes worth pursuing for less-critical projects.
Avoiding the SimpleDB HypeThere are many different application scenarios where SimpleDB is an interesting option. That said, some people have overstated the benefits of using SimpleDB specifically and hosted NoSQL databases in general.The reasoning seems to be that services running on the infrastructure of companies like Amazon, Google, or Microsoft will undoubtedly have nearly unlimited automatic scalability.Although there is nothing wrong with enthusiasm for products and services that you like, it is good to base that enthusiasm on reality. Do not be fooled into thinking that any of these new databases is going to be a panacea. Make sure you educate yourself about the pros and cons of each solution as you evaluate it.The majority of services in this space have a free usage tier, and all the opensource alternatives are completely free to use.Take advantage of it, and try them out for yourself.We live in an amazing time in history where the quantity of information available at our fingertips is unprecedented.Access to web-based services and open-source projects is a huge opportunity.The tragedy is that in a time when it has never been easier to gain personal experience with new technology, all too often we are tempted to adopt the opinions of others instead of taking the time to form our own opinions. Do not believe the hype—find out for yourself.
Putting the DBA Out of WorkOne of the stated goals of SimpleDB is allowing customers to outsource the time and effort associated with managing a web-scale database. Managing the database is traditionally the world of the DBA. Some people have assumed that advocating the use of SimpleDB amounts to advocating a world where the DBA diminishes in importance.However, this is not the case at all.
One of the things that have come about from the widespread popularity of EC2 has been a change in the role of system administrators.What we have found is that managing EC2 virtual instances is less work than managing a physical server instance.However, the result has not been a rash of system administrator firings. Instead, the result has been that system administrators are able to become more productive by managing larger numbers of servers than they otherwise could.The ease of acquisition and the low cost to acquire and release the computing power have led, in many cases, to a greater and more dynamic use of the servers. In other words, organizations are using more server instances because the various levels of the organization can handle it, from a cost, risk, and labor standpoint.
SimpleDB and its cohorts seem to facilitate a similar change but on a smaller scale. First, SimpleDB has less general applicability than EC2. It is a suitable solution for a much smaller set of problems.AWS fully advocates the use of existing relational database products. SimpleDB is an additional option, not a replacement. Moreover, SimpleDB finds good usage in some areas where a relational database might not normally be used, as in the case of storing web user session data. In addition, for those projects that choose to use SimpleDB instead of, or along with, a relational database, it does not mean that there is no role for the DBA. Some tasks remain similar to EC2, which can result in a greater capacity for IT departments to create solutions.
Dodging Copies of C.J. DateThere are database purists who wholeheartedly try to dissuade people from using any type of non-relational database on principle alone. Not only that, but they also go to great lengths to advocate the proper use of relational databases and lament the fact that no current database products correctly implement the relational model. Having found the one-true data storage paradigm, they believe that the relational model is “right” and is the only one that will last.The purists are not wrong in their appreciation for the relational model and for SQL.The relational model is the cornerstone of the database field, and more than that, an invaluable contribution to the world of computing. It is one of the two best things to come out of 1969. Invented by a mathematician and considered a branch of mathematics itself, there is a solid theoretical rigor that underlies its principles. Even though it is not a complete or finished branch, the work to date has been sound.
The world of mathematics and academic research is an interesting place.When you have spent large quantities of your life and career there, you are highly qualified to make authoritative comments on topics like correctness and provability.Nevertheless, being either a relational model expert or merely someone who holds them in high regard does not say anything about your ability to deliver value to users. It is clearly true that modeling your data “correctly” can provide measurable benefits and that making mistakes in your model can lead to certain classes of problems. However, you can still provide significant user value with a flawed model, and correctness is no guarantee of success.
It is like perfectly generated XHTML that always validates. It is like programming with a functional style (in any programming language) that lets you prove your programs are correct. It is like maintaining unit tests that provide 100% test coverage for every line of code you write.There is nothing inherently bad you can say about these things. In fact, there are plenty of good things to say about them.The problem is not a technical problem— it is a people problem.The problem is when people become hyper-focused on narrow technological aspects to the exclusion of the broader issues of the application’s purpose.
The people conducting database research and the ones who take the time to help educate the computing industry deserve our respect. If you have a degree in computer science, chances are you studied C.J. Date’s work in your database class.Among professional programmers, there is no good excuse for not knowing data and relational fundamentals. However, the person in the next row of cubicles who is only contributing condescending criticism to your project is no C.J. Date. In addition, the user with 50 times your stackoverflow.com reputation who ridicules the premise of your questions without providing useful suggestions is no E.F. Codd. Understanding the theory is of great importance. Knowing how to deliver value to your users is of greater importance. In the end, avoid vociferous ignorance and don’t let anyone kick copies of C.J. Date in your face.
Other Pieces of the PuzzleIn the world of cloud computing, there are a growing number of companies and services from which to choose. Each service provider seeks to align its offerings with a broader strategy.With Amazon, that strategy includes providing very basic infrastructure building blocks for users to assemble customized solutions.AWS tries to get you to use more than one service offering by making the different services useful with each other and by offering fast and free data transfer between services in the same region.This section describes three other Amazon Web Services, along with some ways you might find them to be useful in conjunction with SimpleDB.
Adding Compute Power with Amazon EC2AWS sells computing power by the hour via the Amazon Elastic Compute Cloud (Amazon EC2).This computing power takes the form of virtual server instances running on top of physical servers within Amazon data centers.These server instances come in varying amounts of processor horsepower and memory, depending on your needs and budget. What makes this compute cloud elastic is the fact that users can start up, and shut down, dozens of virtual instances at a moment’s notice.
These general-purpose servers can fulfill the role of just about any server. Some of the popular choices include web server, database server, batch-processing server, and media server.The use of EC2 can result in a large reduction in ongoing infrastructure maintenance when compared to managing private in-house servers. Another big benefit is the elimination of up-front capital expenditures on hardware in favor of paying for only the compute power that is used.
The sweet spot between SimpleDB and EC2 comes for high-data bandwidth applications. For those apps that need fast access to high volumes of data in SimpleDB, EC2 is the platform of choice.The free same region data transfer can add up to a sizable cost savings for large data sets, but the biggest win comes from the consistently low latency.AWS does not guarantee any particular latency numbers but typically, round-tripping times are in the neighborhood of 2 to 7 milliseconds between EC2 instances and SimpleDB in the same region.These numbers are on par with the latencies others have reported between EC2 instances. For contrast, additional latencies of 50 to 200 milliseconds or more are common when using SimpleDB across the open Internet.When you need fast SimpleDB, EC2 has a lot to offer.
Storing Large Objects with Amazon S3Amazon Simple Storage Service (Amazon S3) is a web service that enables you to store an unlimited number of files and charges you (low) fees for the actual storage space you use and the data transfer you use.As you might expect, data transfer between S3 and other Amazon Web Services is fast and free. S3 is easy to understand, easy to use, and has a multitude of great uses.You can keep the files you store in S3 private, but you can also make them publicly available from the web. Many websites are using S3 as a media-hosting service to reduce the load on web servers.
EC2 virtual machine images are stored and loaded from S3. EC2 copies storage volumes to and loads storage volumes from S3.The Amazon CloudFront content delivery network can serve frequently accessed web files in S3.The Amazon Elastic MapReduce service runs MapReduce jobs stored in S3. Publicly visible files in S3 can be served up via the BitTorrent peer-to-peer protocol.The list of uses goes on and on.... S3 is really a common denominator cloud service.
SimpleDB users can also find good uses for S3. Because of the high speed within the Amazon cloud, S3 is an obvious storage location choice for SimpleDB import and export data. It is also a solid location to place SimpleDB backup files.
Queuing Up Tasks with Amazon SQSAmazon Simple Queue Service (Amazon SQS) is a web service that reliably stores messages between distributed computers. Placing a robust queue between the computers allows them to work independently. It also opens the door to dynamically scaling the number of machines that push messages and the number that retrieve messages. Although there is no direct connection between SQS and SimpleDB, SQS does have some complementary features that can be useful in SimpleDB-based applications.The semantics of reliable messaging can make it easier to coordinate multiple concurrent clients than when using SimpleDB alone. In cases where there are multiple SimpleDB clients, you can coordinate clients using a reliable SQS queue. For example, you might have multiple servers that are encoding video files and storing information about those videos in SimpleDB. SimpleDB makes a great place to store that data, but it could be cumbersome for use in telling each server which file to process next.The reliable message delivery of SQS would be much more appropriate for that task.
Comparing SimpleDB to Other Products and ServicesNumerous new types of products and services are now available or will soon be available in the database/data service space. Some of these are similar to SimpleDB, and others are tangential.A few of them are listed here, along with a brief description and comparison to SimpleDB.
Windows Azure PlatformTheWindows Azure Platform is Microsoft’s entry into the cloud-computing fray.Azure defines a raft of service offerings that includes virtual computing, cloud storage, and reliable message queuing. Most of these services are counterparts to Amazon services.At the time of this writing, the Azure services are available as a CommunityTechnology Preview. To date, Microsoft has been struggling to gain its footing in the cloud services arena.
There have been numerous, somewhat confusing, changes in product direction and product naming.Although Microsoft’s cloud platform has been lagging behind AWS a bit, it seems that customer feedback is driving the recent Azure changes.There is every reason to suspect that once Azure becomes generally available, it will be a solid alternative to AWS. Among the services falling under the Azure umbrella, there is one (currently) named Windows AzureTable.AzureTable is a distributed key-value store with explicit support for partitioning across storage nodes. It is designed for scalability and is in many ways similar to SimpleDB.The following is a list of similarities between AzureTable and SimpleDB:
Google App EngineApp Engine is a service offered by Google that lets you run web applications, written in Java or Python, on Google’s infrastructure. As an application-hosting platform,App Engine includes many non-database functions, but the App Engine data store has similarities to SimpleDB.The non-database functions include a number of different services, all of which are available via API calls.The APIs include service calls to Memcached, email, XMPP, and URL fetching.
App Engine includes an API for data storage based on Google Big Table and in some ways is comparable to SimpleDB.Although Big Table is not directly accessible to App Engine applications, there is support in the data store API for a number of features not available in SimpleDB.These features include data relations, object mapping, transactions, and a user-defined index for each query.
App Engine also has a number of restrictions, some of which are similar to SimpleDB restrictions, like query run time. By default, the App Engine data store is strongly consistent. Once a transaction commits, all subsequent reads will reflect the changes in that transaction. It also means that if the primary storage node you are using goes down,App Engine will fail any update attempts you make until a suitable replacement takes over.To alleviate this issue, App Engine has recently added support for the same type of eventual consistency that SimpleDB has had all along.This move in the direction of SimpleDB gives App Engine apps the same ability as SimpleDB apps to run with strong consistency with option to fall back on eventual consistency to continue with a degraded level of service.
CouchDB is an open-source product that you install and manage yourself. It allows distributed replication among peer servers and has full support for robust clustering. CouchDB was designed from the start to handle high levels of concurrency and to maintain high levels of availability. It seeks to solve many of the same problems as SimpleDB, but from the standpoint of an open-source product offering rather than a pay-as-you-go service.
Dynamo-Like ProductsAmazon Dynamo is a data store used internally within Amazon that is not available to the public.Amazon has published information about Dynamo that includes design goals, runtime characteristics, and examples of how it is used. From the published information,we know that SimpleDB has some things in common with Dynamo, most notably the eventual consistency.
Since the publication of Dynamo information, a number of distributed key-value stores have been developed that are in the same vein as Dynamo.Three open-source products that fit into this category are Project Voldemort, Dynomite, and Cassandra. Each of these projects takes a different approach to the technology, but when you compare them to SimpleDB, they generally fall into the same category.They give you a chance to have highly available key-value access distributed across machines.You get more control over the servers and the implementation that comes with the maintenance cost of managing the setup and the machines. If you are looking for something in this class of data storage, SimpleDB is a likely touch-free hosted option, and these projects are hands-on self-hosted alternatives.
Compelling Use Cases for SimpleDBSimpleDB is not a replacement for relational databases.You need to give careful consideration to the type of data storage solution that is appropriate for a given application.This section includes a discussion of some of the use cases that match up well with SimpleDB. Web Services for Connected Systems
IT departments in the enterprise are tasked with delivering business value and support in an efficient way. In recent years, there has been movement toward both service orientation and cloud computing. One of the driving forces behind service orientation is a desire to make more effective use of existing applications. Simple Object Access Protocol (SOAP) has emerged as an important standard for message passing between these connected systems as a means of enabling forward compatibility. For new services deployed in the cloud, SimpleDB is a compelling data storage option.
Data transfer between EC2 instances and the SimpleDB endpoint in the same region is fast and free.The consistent speed and high availability of SimpleDB are helpful when defining a Service Level Agreement (SLA) between IT and business units. All this meshes with the ability of EC2 to scale out additional instances on demand.
Low-Usage ApplicationThere are applications in the enterprise and on the open web that do not see a consistent heavy load.They can be low usage in general with periodic or seasonal spikes—for instance, at the end of the month or during the holidays. Sometimes there are few users at all times by design or simply by lack of popularity.
For these types of applications, it can be difficult to justify an entire database server for the one application.The typical answer in organizations with sufficient infrastructure is to host multiple databases on the same server.This can work well but may not be an option for small organizations or for individuals. Shared database hosting is available from hosting companies, but service levels are notoriously unpredictable.With SimpleDB, low-usage applications can run within the free tier of service while maintaining the ability to scale up to large request volumes when necessary.This can be an attractive option even when database-sharing options are available.
Clustered Databases Without the Time SinkClustering databases for scalability or for availability is no easy task. If you already have the heavy data access load or if you have the quantifiable need for uptime, it is obviously a task worth taking on. Moreover, if you already have the expertise to deploy and manage clusters of replicated databases, SimpleDB may not be something you need. However, if you do have the experience, you know many other things as well: you know the cost to roll the clusters into production, to roll out schema updates, and to handle outages.This information can actually make it easier to decide whether new applications will provide enough revenue or business value to merit the time and cost.You also have a great knowledge base to make comparisons between in-house solutions and SimpleDB for the features it provides.
You may have a real need for scalability or uptime but not the expertise. In this case, SimpleDB can enable you to outsource the potentially expensive ongoing database maintenance costs.
Dynamic Data ApplicationRigid and highly structured data models serve as the foundation of many applications, while others need to be more dynamic. It is becoming much more important for new applications to include some sort of social component than it was in the past.Along with these social aspects, there are requirements to support various types of user input and customization, like tagging, voting, and sharing. Many types of social applications require community building, and can benefit from a platform, which allows data to be stored in new ways, without breaking the old data. Customer-facing applications, even those without a social component, need to be attentive to user feedback.
Whether it is dynamic data coming from users or dynamic changes made in response to user feedback, a flexible data store can enable faster innovation.
Amazon S3 Content SearchAmazon S3 has become a popular solution for storing web-accessible media files.Applications that deal with audio, video, or images can access the media files from EC2 with no transfer costs and allow end users to download or stream them on a large scale without needing to handle the additional load.When there are a large number of files in S3, and there is a need to search the content along various attributes, SimpleDB can be an excellent solution.
It is easy to store attributes in SimpleDB, along with pointers to where the media is stored in S3. SimpleDB creates an index for every attribute for quick searching. Different file types can have different attributes in the same SimpleDB domain. New file types or new attributes on existing file types can be added at any time without requiring existing records to be updated.
Empowering the Power UsersFor a long time, databases have been just beyond the edge of what highly technical users can effectively reach. Many business analysts, managers, and information workers have technical aptitude but not the skills of a developer or DBA.These power users make use of tools like spreadsheet software and desktop databases to solve problems. Unfortunately, these tools work best on a single workstation, and attempts at sharing or concurrent use frequently cause difficulty and frustration; enterprise-capable database software requires a level of expertise and time commitment beyond what these users are willing to spend.
The flexibility and scalability of SimpleDB can be a great boon to a new class of applications designed for power users. SimpleDB itself still requires programming on the client and is not itself directly usable by power users. However, the ability to store data directly without a predefined schema and create queries is an enabling feature. For applications that seek to empower the power users, by creating simple, open-ended applications with dynamic capabilities, SimpleDB can make a great back end.
Existing AWS CustomersThis chapter pointed out earlier the benefits of using EC2 for high-bandwidth applications. However, if you are already using one or more of the Amazon Web Services, SimpleDB can be a strong candidate for queryable data storage across a wide range of applications. Of course, running a relational database on an EC2 instance is also a viable and popular choice. Moreover, you would do well to consider both options. SimpleDB requires you to make certain trade-offs, but if the choices provide a net benefit to your application, you will have gained some great features from AWS that are difficult and time consuming to develop on your own.
SummaryAmazon SimpleDB is a web service that enables you to store semi-structured data within Amazon’s data centers.The service provides automatic, geographically diverse data replication and internal routing around failed storage nodes. It offers high availability and enables horizontal scalability.The service allows you to offload hardware maintenance and database management tasks.
You can use SimpleDB as a distributed key-value store using the GetAttributes,
PutAttributes, and DeleteAttributes API calls.You also have the option to query for
your data along any of its attributes using the Select API call. SimpleDB is not a relational
database, so there are no joins, foreign keys, schema definitions, or relational constraints
that you can specify. SimpleDB also has limited support for transactions, and updates
propagate between replicas in the background. SimpleDB supports strong consistency,
where read operations immediately reflect the results of all completed and eventual consistency,
where storage nodes are updated asynchronously in the background.