pdk
pdk copied to clipboard
Can I use pdk to ingest data from other database like Hive?
Hi guys: I'm a freshman of Pilosa and go. I read some docs in your web site, and I am not clear with Pilosa and it's data model so far. For example, some records of relational database like below:
name | age |
---|---|
A | 18 |
B | 19 |
It will be transformed to bitmap index:
Field name
ID | 1 | 2 | |
---|---|---|---|
A | 1 | 1 | 0 |
B | 2 | 0 | 1 |
Field age (BSI)
age | 18 | 19 |
---|---|---|
comp 0 | 1 | 0 |
comp 2 | 0 | 0 |
comp 3 | 1 | 1 |
comp 4 | 1 | 1 |
comp 5 | 0 | 0 |
not_null | 1 | 1 |
Is that right?
If I understand the above correctly. I have the following confusion:
- Pilosa store row and column ids in field, how can I transform the string values to int ids while ingestion. For example
Field name
, how can I transformA
to Row id 1 while ingestion. The map A to 1 need to be maintained in other server of user's system or pilosa will do it automatically?If it's latter, can I get the valueA
while query Row(name=1)? - If I want to ingest data from hive, what should I do ? Is there any DML like batch query or it need users to set Rows of field one by one?
Excepting to hear from you guys and thanks for helping me to use and understand Pilosa better !