go-hbase icon indicating copy to clipboard operation
go-hbase copied to clipboard

General usage questions

Open rgcooper53 opened this issue 8 years ago • 2 comments

I started with go-hbase from lazyshot, ran into problems. He eventually referred me to your version. I got yours working, loading a file with 129k rows, 16 columns into HBase. I expected somewhat slower performance running in HBase standalone mode and testing on a Linux VM on VMWare, but was still surprised at how much slower it was -- 250 updates/second, With an eventual target of at least 500k/second in fully distributed mode with proper configuration, I was looking for at least indicative standalone rates say in the 10k range or at the very least 1k-10k.

  1. Would you rate this package as "ready for primetime," meaning I could use it with sufficient testing in a live mission critical application?

  2. the lazyshot version had methods for "Gets" and "Puts" which I understood were to support batching of updates and requests, your version does not have these. Is there a reason they were removed other than just the scope of your effort and available time? Is there any reason I couldn't work to add them back in if I needed them for the performance boost?
    Thanks.

rgcooper53 avatar Dec 01 '16 16:12 rgcooper53

Thank you for your feedback!

  1. I use this package for https://github.com/pingcap/go-themis which is a ACID transaction library for HBase, I benched this library, performance is OK, so may I take a look at your test code and test environment, so that I can find out the cause. BTW since go-themis have been discard, so I'm afraid it's not been tested in production environment.

  2. According to the use case in go-themis, I don't need to port lazyshot's Gets function, and I think his implementation of Gets/Puts is buggy, I create my implementation in go-themis using coprocessor.

c4pt0r avatar Dec 02 '16 00:12 c4pt0r

I remember I did a simple benchmark, something like https://github.com/pingcap/go-themis/blob/master/benchmark/main.go ,but with raw go-hbase API, and I got ~30k qps in read. How many goroutine did you use?

c4pt0r avatar Dec 02 '16 01:12 c4pt0r