pdk icon indicating copy to clipboard operation
pdk copied to clipboard

Can I use pdk to ingest data from other database like Hive?

Open BigRantLing opened this issue 3 years ago • 0 comments

Hi guys: I'm a freshman of Pilosa and go. I read some docs in your web site, and I am not clear with Pilosa and it's data model so far. For example, some records of relational database like below:

name age
A 18
B 19

It will be transformed to bitmap index:

Field name

ID 1 2
A 1 1 0
B 2 0 1

Field age (BSI)

age 18 19
comp 0 1 0
comp 2 0 0
comp 3 1 1
comp 4 1 1
comp 5 0 0
not_null 1 1

Is that right?

If I understand the above correctly. I have the following confusion:

  1. Pilosa store row and column ids in field, how can I transform the string values to int ids while ingestion. For example Field name, how can I transform A to Row id 1 while ingestion. The map A to 1 need to be maintained in other server of user's system or pilosa will do it automatically?If it's latter, can I get the value A while query Row(name=1)?
  2. If I want to ingest data from hive, what should I do ? Is there any DML like batch query or it need users to set Rows of field one by one?

Excepting to hear from you guys and thanks for helping me to use and understand Pilosa better !

BigRantLing avatar Jun 28 '21 04:06 BigRantLing