Amazon Dynamo for token management

[akbarahmed]

This is a feature proposal to add an Amazon Dynamo DAO for token management.

We are actively working on this feature and will issue a PR when ready.

[avinashZato]

Hi Akbar,

I couldn't find material on DynomiteDB or on other forums for the following questions:

1. How are the tokens re-calculated when a node joins/ leaves the cluster?
2. How is the data rebalanced after the tokens are re-calculated? Is this even a Dynomite concern?

Any pointers will really help.

[akbarahmed]

@avinashZato

• Node tokens are calculated when the cluster is first created. Then, when a node is replaced (ex. if a node fails), the new node queries the list of node tokens to find a previously failed token. It allocates the previously failed token to itself, and begins an automated recovery process. Once the node is full healed, it rejoins the cluster automatically.
• When a node leaves the cluster (ex. when it fails), the Dyno client will be notified of the failure due to failed health checks and will direct traffic away from the failed node...until it is brought back online.
• If you are adding a single node, say with a minor scale up, then this is currently not an automated process. There is ongoing work to allow automated scale up with rebalancing by doubling the cluster size. Please add a watch to the project on GH to be notified when this PR is submitted.
• Automated data rebalancing is not currently implemented.

Typically, you'll deploy a Dynomite cluster with a somewhat fixed size. However, as mentioned above, there is active work to automate the scale up/down process.

[avinashZato]

Appreciate your quick response here @akbarahmed ! Our deployment falls in category three that you've listed above where we incrementally add nodes.

To give you some context, we typically have an on-premise deployment where the customer would typically start with N nodes each with a capacity of C. Depending on the load a new Node N1 will be added with the same capacity C.

For approach three (the manual process), we'll probably follow the steps recommended by @Minh in #173 .

If it isn't too much to ask do you have a tentative date for when the PR would be submitted?

[akbarahmed]

There's no ETA for when the PR will be ready.

[ipapapa]

@avinashZato I am not following why Dynomite needs to follow the diurnal pattern of the client. In fact, if you plan to scale up then you need to scale down, and generally speaking stateful systems must not follow that pattern because data can get out of sync.

If you are using Dynomite as a cache, then it means that you set a short TTL on the data. Hence the Dyno dual write functionality would probably solve your problem. In other words, you will bring a new Dynomite cluster that has some extra capacity, dual write to it, then take the older cluster down once the data have expired. The same way can work for scaling down.

[ipapapa]

@akbarahmed I like a lot having the DynamoDB DAO because it supports cross-region replication and you do not need another system outside of AWS. In case of region outage, we can move everything to another region and still have the tokens to come back to the failed region. This was the idea behind the Cassandra DAO. @diegopacheco was also considering changing the Astyanax to the Datastax Java Driver, which would make the DAO a little more standardized, but I am not sure how far he got with it.

[diegopacheco]

@ipapapa not much far yet.

[diegopacheco]

@ipapapa This is working in progress. You can follow progress here https://github.com/diegopacheco/dynomite-manager-1/tree/dev-datastax-javadriver I will fire a PR once I'm done. :-)

[ipapapa]

DynamoDB does not actually support bi-directional cross-region replication (only one way). More info. Hence at this point, I do not see any reason to pursue the DynamoDB DAO. We can revisit this one bi-directional replication is added.

[posix4e]

This claims it is full multi region , can we reopen this?https://aws.amazon.com/dynamodb/global-tables/

[ipapapa]

@posix4e Done. It would be really nice to see the DAO being added.

[posix4e]

dao?

[ipapapa]

DAO = Data Access Object. It is a Java design pattern. More information: https://www.oracle.com/technetwork/java/dataaccessobject-138824.html

[srinivas-c]

@ipapapa , @akbarahmed Any update on supporting DynamoDB for token management. We will be eager to use this :)

Thanks in advance

[srinivas-c]

Hi,

I submitted a pull request for DynamoDB support, the only piece missing is integrating dynamodb with Florida. Can you review and share comments if any https://github.com/Netflix/dynomite-manager/pull/114

Thanks