How to import sst-file to tikv cluster

[liujp]

Question

How to import sst-file to tikv cluster @3pointer

[3pointer]

basically, there is no natural way to import sst files. you need to build SSTMeta requests in your code and use ingest method to import sst files.

[liujp]

Is there any examples to guide the usage of sst-importer client in kvproto in https://github.com/pingcap/kvproto/blob/master/proto/import_sstpb.proto and the instruction of this repo

[3pointer]

There is no guide for these code. and sst-importer is not a independent component to import sst files. It's a part of tidb-lightning before.

And the most code has migrated to TiDB repo

[liujp]

ok, thx

[liujp]

There is no guide for these code. and sst-importer is not a independent component to import sst files. It's a part of tidb-lightning before.

And the most code has migrated to TiDB repo

I have read the introduction of the tidb lightning, I think I need a tool like lightning, input the csv-file and encode the values to the key-value in my own method. So I think the tikv-importer fit me; so I have some questions about this tool:

1. if the use C++ to implement the client of the tikv-importer, which proto file to be use?
2. the input of this tool is the kvs from the client, and output is SST file, is that right?

[lance6716]

input the csv-file and encode the values to the key-value

This is done at lightning side. Lightning is a golang program and uses the encoding functionality of TiDB which also a golang program.

this tool tikv-importer mainly does https://github.com/tikv/importer/blob/351ed5a142a61a2141bf06512085e6f8eb88feb4/src/import/kv_service.rs#L59-L76

Yes it can receive the KV paris from client. The expected output is not SST files, but using RPC to directly send the internal SST files to TiKV and ingest them.

And I must say that this tool is no longer maintained.

[liujp]

ok, thanks.But actually I only want to import k-v Paris to tikv cluster. As far as I know, the lightning is a tool for SQL struct data file, that is to say, It need a schema to parse the data format. which is different from me; So, mu question is that:

1. this tool is no longer maintained. Can I use this client(as below) access the tiki-cluster directly, like assessing importer-tool
2. How can I using lightning to import k-v Paris to the tikv-cluster If it can;
3. Or some solutions you give

[lance6716]

@liujp If you already have KV pairs, yes lightning is not suitable for this use case. I think you can directly implement the RPC client of service ImportSST https://github.com/pingcap/kvproto/blob/ecdbf1f8c130089392a9bb5f86f7577deddfbed5/proto/import_sstpb.proto#L34C1-L34C18

And use Write/RawWrite, Ingest RPC methods. You can refer to lightning's code https://github.com/pingcap/tidb/blob/19e888f1fecc72f1e4eac284db583eff15941dac/br/pkg/lightning/backend/local/local.go#L1361-L1375

this tikv-importer may have API compatibility break, we do not recommend to use it.