webhdfs
webhdfs copied to clipboard
can't upload large size file
Hi, i've try to upload small size of file and it succeed. but when i try to upload 130Mb file, it always failed. is there any options configurations that i must set first to do this operation?
sometimes i got this error msg.
Error: BP-1312390296-192.168.1.228-1545800077608:blk_1073838522_97940 does not exist or is not under Constructionnull
i've try to increase the buffer size like below this but it still failed :
var localFileStream = fs.createReadStream('/tmp/test.jar', { highWaterMark: 1024 * 1024 });
var remoteFileStream = hdfs.createWriteStream('/user/training/upload/test.jar', { highWaterMark: 1024 * 1024 });
Have you found any workaround for this?