alluxio icon indicating copy to clipboard operation
alluxio copied to clipboard

Add folder check for load job

Open Haoning-Sun opened this issue 1 year ago • 0 comments

What changes are proposed in this pull request?

Add folder check for load job.

Why are the changes needed?

Adding 'partial-listing' when loading a file instead of a directory will cause the loading task to fail, as it results in duplicate blocks of the file being loaded.

./bin/alluxio fs load /test.txt --submit --partial-listing
./bin/alluxio fs load /test.txt  --progress
Progress for loading path '/test.txt':
	Settings:	bandwidth: unlimited	verify: false
	Job State: FAILED (alluxio.exception.runtime.InternalRuntimeException: Job failed because it's not healthy.)
	Files Processed: 5
	Bytes Loaded: 2783.32MB
	Throughput: 927.77MB/s
	Block load failure rate: 5.02%
	Files Failed: 1

Haoning-Sun avatar Dec 20 '23 10:12 Haoning-Sun