Feature Request: DynamoDB support

[codefromthecrypt]

From @cemo on May 26, 2017 10:59

I would like to see DynamoDB support for zipkin. I usually let AWS services to store data and I deal rest of the services. DynamoDB seems a good and cheap alternative for Zipkin. Is it possible to support it as well?

Copied from original issue: openzipkin/zipkin#1599

[codefromthecrypt]

From @cemo on June 5, 2017 10:20

@adriancole is this attractive only for me? DynamoDB based on storage means less cost and a better scalability. Am I missing something at here? Or Should I open this issue on zipkin-aws?

[devinsba]

I might be interested here, we aren't ElasticSearch users elsewhere yet so we don't have a ton of operational knowledge

[dgomesbr]

@cemo and @adriancole as I know integration directly to dynamodb would be easier and straightforward, we could still address that via lambda consumers to dynamodb, whereas we continue to leverage the initial sqs support, but also gives freedom to consume it as we wanted and even store on S3 for later analytics, dashboarding etc

[codefromthecrypt]

Thanks for the context, folks:

there are three concerns to consider (at least)

• user interest and "right fit"
• developer interest to write test and document this (storage is the most complex component in zipkin, more than collector)
• developer+user interest to maintain this (ex keep up when new features are added to zipkin, someone to review those changes)

I think interest and right fit may have been made. We need to see who can help keep this alive once started. Ex I'd offer to help in design and reviewing the component, and I usually do maintenance. However, we need multiple parties on board with the work involved, particularly the less glamorous work (ex when we do zipkin v2 or add fields helping with any changes)

[jhonatanforero]

Hi Everyone, Elasticsearch storage It's a expensive service in Aws, this is a good Idea for keep cheap the zipkin infra in the Aws cloud. @adriancole

[codefromthecrypt]

there was an assumption this would be cheap, but that was contested by one of the users.. that the query side of actually doing this would be not cheap.

The bigger problem is that the effort is a huge amount of code to maintain, and even completing it didn't happen despite heroism: https://github.com/openzipkin/zipkin-aws/pull/125

I would suggest that unless someone is willing to pick this up and complete it, other options, like direct storage into X-Ray (without agent even), might be alternatives to consider.

[jhonatanforero]

Ok , This is the context --> Today, We have Kinesis -> Ec2 (collector) -> E.S 2 shards of Kinesis Cost 60 USD per Month, E.S cost per GB storage, so this infra is expensive, We are migrating Kinesis, and we want to use SQS. We want to be Agnostic, so We don't want to use x-ray, We like a lot the zipkin Lens UI, I'm searching for new storage like Cassandra service management in Aws maybe It would be cheaper. How much can I reduce the cost for zipkin infra in Aws? This is the big question. If the collector (sqs) was developed in Spring cloud function as Aws Lambda (Java) but where can I install the Zipkin UI? This a topic for discussion.

[codefromthecrypt]

Thanks for all the context. this is great. in fact we have a sites section which maybe you can peruse or look at for insight into other deployments. I know there are amazon sites running all of our storage options

https://github.com/openzipkin/openzipkin.github.io/wiki/Sites

in fact having a doc on your site would be fantastic and easier for others to consider also.

For now, this issue is about dynamodb and i think it is fair to say this feature would need an owner who can drive the code to completion and until that time it isnt an option.

meanwhile I think your specific setup as mentioned could be served by multiple options (ps we love lens too ;)) rather than discuss on the dynamodb thread.. would you mind moving to chat?

https://gitter.im/openzipkin/zipkin

On Tue, Feb 18, 2020, 8:45 AM Jhonatan Forero Rosas < [email protected]> wrote:

Ok , This is the context --> Today, We have Kinesis -> Ec2 (collector) -> E.S 2 shards of Kinesis Cost 60 USD per Month, E.S cost per GB storage, so this infra is expensive, We are migrating Kinesis, and we want to use SQS. We want to be Agnostic, so We don't want to use x-ray, We like a lot the zipkin Lens UI, I'm searching for new storage like Cassandra service management in Aws maybe It would be cheaper. How much can I reduce the cost for zipkin infra in Aws? This is the big question. If the collector (sqs) was developed in Spring cloud function as Aws Lambda (Java) but where can I install the Zipkin UI? This a topic for discussion.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openzipkin/zipkin-aws/issues/45?email_source=notifications&email_token=AAAPVV2K44VQKOTVI6AKOBTRDMVSTA5CNFSM4DN7Q6L2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMAEMUQ#issuecomment-587220562, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAPVVYNSK7X66VQOVNJMD3RDMVSTANCNFSM4DN7Q6LQ .