webhdfs icon indicating copy to clipboard operation
webhdfs copied to clipboard

can't upload large size file

Open y0uzha opened this issue 4 years ago • 1 comments

Hi, i've try to upload small size of file and it succeed. but when i try to upload 130Mb file, it always failed. is there any options configurations that i must set first to do this operation?

sometimes i got this error msg.

Error: BP-1312390296-192.168.1.228-1545800077608:blk_1073838522_97940 does not exist or is not under Constructionnull

i've try to increase the buffer size like below this but it still failed :

var localFileStream = fs.createReadStream('/tmp/test.jar', { highWaterMark: 1024 * 1024 });
var remoteFileStream = hdfs.createWriteStream('/user/training/upload/test.jar', { highWaterMark: 1024 * 1024 });

y0uzha avatar Nov 01 '19 04:11 y0uzha

Have you found any workaround for this?

maziyarpanahi avatar Dec 25 '19 09:12 maziyarpanahi