carbondata icon indicating copy to clipboard operation
carbondata copied to clipboard

Multi hosts write to one hdfs file?

Open BestP2P opened this issue 3 years ago • 1 comments

if I use hdfs system, and the using sdk program running on multi hosts, how can i let them write to one hdfs file? thank you

BestP2P avatar Jun 04 '21 06:06 BestP2P

I suggest using SDK to write data into the stage area and using insert into the stage to add it to the table.

https://github.com/apache/carbondata/blob/master/docs/flink-integration-guide.md

Another way is using add segment function to support multiple formats in a table, but it has many limitations.

https://github.com/apache/carbondata/blob/8740016917168777f1514ef4de0615f83b13c6d3/docs/addsegment-guide.md

QiangCai avatar Jun 28 '21 01:06 QiangCai