dm-adapter-simpledb

dm-adapter-simpledb copied to clipboard

= dm-adapter-simpledb

== What

A DataMapper adapter for Amazon's SimpleDB service.

Features:

• Uses the RightAWS gem for efficient SimpleDB operations.
• Full set of CRUD operations
• Supports nearly all DataMapper query predicates.
• Full support for complex nested union, intersaction, and negation in queries
• Migrations
• DataMapper identity map support for record caching
• Lazy-loaded attributes
• DataMapper Serial property support via UUIDs.
• Array properties
• Basic aggregation support (Model.count("..."))
• String "chunking" permits attributes to exceed the 1024-byte limit
• Support for efficient :limit and :offset, for result set paging
• Robust quoting of names in values in selects

Note: as of version 1.0.0, this gem supports supports the DataMapper 0.10.* series and breaks backwards compatibility with DataMapper 0.9.*.

== Who

Originally written by Jeremy Boles.

Contributers: Edward Ocampo-Gooding (edward) Dan Mayer (danmayer) Thomas Olausson (latompa) Avdi Grimm (avdi)

== Where

dm-adapter-simpledb is currently maintained by the Devver team and lives at: http://github.com/devver/dm-adapter-simpledb/

== TODO

• Handle exclusive ranges natively Implement as inclusive range + filter step
• Tests for associations
• Option for smart lexicographical storage for numbers
  • Zero-pad integers
  • Store floats using exponential notation
• Option to store Date/Time/DateTime as ISO8601
• Full aggregate support (min/max/etc)
• Option to use libxml if available
• Parallelized queries for increased throughput
• Support of normalized 1:1 table:domain schemes that works with associations
• Sharding
• Support BatchPutAttributes
• Silence SSL warnings See http://pivotallabs.com/users/carl/blog/articles/1079-standup-blog-11-24-2009-model-validations-without-backing-store-associations-to-array-and-ssl-with-aws
• Token cache for reduced requests when given an offset
• Optimize key queries

== Usage

There is a functional example in examples/ in this project.

A example Rails on SimpleDB project is maintained here: http://github.com/danmayer/datamapper-simpledb-rails-example A example Sinatra on SimpleDB project is maintained here: http://github.com/danmayer/sinatra-simpledb-example

=== Standalone

require 'rubygems'
require 'dm-core'
require 'dm-adapter-simpledb'

DataMapper.setup(:default, 
  :adapter       => 'simpledb',
  :access_key    => "ACCESS_KEY",
  :secret_key    => "SECRET_KEY",
  :domain        => "DOMAIN",
)

[Same as the following, but skip the database.yml]

=== In a Merb application See sample Merb application using Merb-Auth and protected resources on SimpleDB: http://github.com/danmayer/merb-simpledb-dm_example/tree/master

Setup database.yml with the SimpleDB DataMapper adapter:

  adapter:    simpledb
  access_key: (a 20-character, alphanumeric sequence)
  secret_key: (a 40-character sequence)
  domain:     'my_amazon_sdb_domain'

Create a model

  class Tree
    include DataMapper::Resource
  
    storage_name[:default] = "trees"
  
    property :id,   Serial
    property :name, String, :nullable => false
  end

Use interactively (with merb -i)

$ merb -i

  maple = Tree.new
  maple.name = "Acer rubrum"
  maple.save

  all_trees = Tree.all() # calls #read_all
  a_tree = Tree.first(:name => "Acer rubrum")
  yanked_tree = Tree.remote(:name => "Acer rubrum")

== Running the tests Add these two lines to your .bash_profile as the spec_helper relies on them

 $ export AMAZON_ACCESS_KEY_ID='YOUR_ACCESS_KEY'
 $ export AMAZON_SECRET_ACCESS_KEY='YOUR_SECRET_ACCESS_KEY'

Configure the domain to use for integration tests. THIS DOMAIN WILL BE DELETED AND RECREATED BY THE TESTS, so do not choose a domain which contains data you care about. Configure the domain by creating a file named THROW_AWAY_SDB_DOMAIN in the projet root:

$ echo dm_simpledb_adapter_test > THROW_AWAY_SDB_DOMAIN

Run the tests:

 rake spec

NOTE: While every attempt has been made to make the tests robust, Amazon SimpleDB is by it's nature an unreliable service. Sometimes it can take a very long time for updates to be reflected by queries, and sometimes calls just time out. If the tests fail, try them again a few times before reporting it as a bug. Also try running the spec files individually.

== Bibliography

Relating to Amazon SimpleDB http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1292&ref=featured Approaching SimpleDB from a relational database background

Active Record Persistence with Amazon SimpleDB http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1367&categoryID=152

Building for Performance and Reliability with Amazon SimpleDB http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1394&categoryID=152

Query 101: Building Amazon SimpleDB Queries http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1231&categoryID=152

Query 201: Tips & Tricks for Amazon SimpleDB Query http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1232&categoryID=152 Latter portion describes parallelization advantages of normalized domains – the downside being the added complexity at the application layer (this library’s).

Using SimpleDB and Rails in No Time with ActiveResource http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1242&categoryID=152 Exemplifies using the Single Table Inheritance pattern within a single SimpleDB domain by storing the model type in an attribute called '_resource' and using a “SHA512 hash function on the request body combined with a timestamp and a configurable salt” for the id.

RightScale Ruby library to access Amazon EC2, S3, SQS, and SDB http://developer.amazonwebservices.com/connect/entry!default.jspa?categoryID=140&externalID=1014&fromSearchPage=true