alluxio icon indicating copy to clipboard operation
alluxio copied to clipboard

Published 20 hours ago • verified publisher icon Alluxio

Fix Alluxio/task-mediums#9472 Add verbose mode to fs mount

Open xichen01 opened this issue 2 years ago • 8 comments

What changes are proposed in this pull request?

Add verbose info to fs mount when execution throws an exception

Why are the changes needed?

task-mediums: https://github.com/Alluxio/alluxio/issues/9472

Example: mount failed because the version does not support it.

[root@centos /root/alluxio]$ bin/alluxio fs mount /hdfs hdfs://192.168.0.1:9000/
mount command:  alluxio fs mount /hdfs hdfs://192.168.0.1:9000/ option MountContext{ProtoOptions=readOnly: false
shared: false
commonOptions {
  syncIntervalMs: -1
  ttl: -1
  ttlAction: DELETE
}
detail: true
}
createMountId 2422744136847795522
Loading 18 factory core jars from /root/alluxio/lib
Loading extension jars from /root/alluxio/extensions
The total number of loaded factory core jars is 18
alluxio.underfs.version is set by user,  target version is 9.99.999
Check eligible of OzoneUnderFileSystemFactory version 1.2.1 for hdfs://192.168.0.1:9000/
Factory implementation OzoneUnderFileSystemFactory version 1.2.1 isn't eligible for path hdfs://192.168.0.1:9000/

Check eligible of HdfsUnderFileSystemFactory version 3.3.1 for hdfs://192.168.0.1:9000/
Factory implementation HdfsUnderFileSystemFactory version 3.3.1 isn't eligible for path hdfs://192.168.0.1:9000/


//...


No factory implementation supports the path hdfs://192.168.0.1:9000/
Versions [3.3.1,3.3.1] are supported for path hdfs://192.168.0.1:9000/ but you have configured version: 9.99.999
Mount failed, No Under File System Factory found for: hdfs://192.168.0.1:9000/
[root@centos /root/alluxio]$

mount failed because the UFS connect failed.

[root@centos /root/alluxio]$ bin/alluxio fs mount /hdfs hdfs://192.168.0.1:9000/
mount command:  alluxio fs mount /hdfs hdfs://192.168.0.1:9000/ option MountContext{ProtoOptions=readOnly: false
shared: false
commonOptions {
  syncIntervalMs: -1
  ttl: -1
  ttlAction: DELETE
}
detail: true
}
createMountId 5657492373465276373
Loading 18 factory core jars from /root/alluxio/lib
Loading extension jars from /root/alluxio/extensions
The total number of loaded factory core jars is 18
alluxio.underfs.version is not set by user
Check eligible of OzoneUnderFileSystemFactory version 1.2.1 for hdfs://192.168.0.1:9000/
Factory implementation OzoneUnderFileSystemFactory version 1.2.1 isn't eligible for path hdfs://192.168.0.1:9000/

Check eligible of HdfsUnderFileSystemFactory version 3.3.1 for hdfs://192.168.0.1:9000/
Successfully! Factory implementation HdfsUnderFileSystemFactory version 3.3.1 is eligible for path hdfs://192.168.0.1:9000/

//...

Under File System Factory HdfsUnderFileSystemFactory version 3.3.1 found for: hdfs://192.168.0.1:9000/
Load Under File System HdfsUnderFileSystemFactory with ClassLoader ExtensionsClassLoader
Load Under File System HdfsUnderFileSystemFactory successfully
Mount failed, Call From VM-219-220-centos/192.168.0.1 to VM-219-220-centos:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

remount

[root@centos /root/alluxio]$ bin/alluxio fs mount /hdfs hdfs://192.168.0.1:9000/
mount command:  detail:  alluxio fs mount /hdfs hdfs://192.168.0.1:9000/ option MountContext{ProtoOptions=readOnly: false
shared: false
commonOptions {
  syncIntervalMs: -1
  ttl: -1
  ttlAction: DELETE
}
detail: true
}
detail:  createMountId 4507953386737830466
detail:  Key{authority=192.168.0.1:9000, scheme=hdfs, properties=null} UFS UnderFileSystemWithLogging already exists in the cache, use cached UFS
detail:  Create mount directory /hdfs successfully
Mounted hdfs://192.168.0.1:9000/ at /hdfs
[root@centos /root/alluxio]$

xichen01 avatar Apr 27 '22 16:04 xichen01

Automated checks report:

  • Commits associated with Github account: PASS
  • PR title follows the conventions: FAIL
    • The title of the PR does not pass all the checks. Please fix the following issues:
      • First word of title ("Fixes") is not an imperative verb. Please use one of the valid words

Some checks failed. Please fix the reported issues and reply 'alluxio-bot, check this please' to re-run checks.

alluxio-bot avatar Apr 27 '22 16:04 alluxio-bot

Automated checks report:

  • Commits associated with Github account: PASS
  • PR title follows the conventions: PASS

All checks passed!

alluxio-bot avatar Apr 27 '22 16:04 alluxio-bot

@xichen01 An interesting feature you did! Please take care of the failed UTs.

maobaolong avatar Apr 28 '22 06:04 maobaolong

failed UTs.

Error: 6.147 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project alluxio-table-base: Compilation failure: Compilation failure: 
Error: 6.150 [ERROR] /usr/src/alluxio/table/base/src/main/java/alluxio/table/ProtoUtils.java:[14,26] package alluxio.grpc.table does not exist
Error: 6.150 [ERROR] /usr/src/alluxio/table/base/src/main/java/alluxio/table/ProtoUtils.java:[15,26] package alluxio.grpc.table does not exist
Error: 6.150 [ERROR] /usr/src/alluxio/table/base/src/main/java/alluxio/table/ProtoUtils.java:[16,26] package alluxio.grpc.table does not exist
Error: 6.150 [ERROR] /usr/src/alluxio/table/base/src/main/java/alluxio/table/ProtoUtils.java:[17,38] package alluxio.grpc.table.layout.hive does not exist
Error: 6.151 [ERROR] /usr/src/alluxio/table/base/src/main/java/alluxio/table/ProtoUtils.java:[19,27] package com.google.protobuf does not exist
Error: 6.151 [ERROR] /usr/src/alluxio/table/base/src/main/java/alluxio/table/ProtoUtils.java:[32,39] cannot find symbol

Error: 8.545 [ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.198 s <<< FAILURE! - in alluxio.master.transport.GrpcMessagingTransportTest
Error: 8.549 [ERROR] alluxio.master.transport.GrpcMessagingTransportTest.testServerClosed  Time elapsed: 0.937 s  <<< FAILURE!
java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:87)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertTrue(Assert.java:53)
	at alluxio.master.transport.GrpcMessagingTransportTest.testServerClosed(GrpcMessagingTransportTest.java:153)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:364)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:237)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:158)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)

xichen01 avatar Apr 28 '22 10:04 xichen01

@xichen01 I triggered to restart the failed test, lets wait for the result.

maobaolong avatar Apr 28 '22 23:04 maobaolong

@dbw9580 Remove verbose output when mount is executed normally. Print execution process only when mount fails. Does not return execution information via RPC. Instead, it returns via exception message when mount fails

xichen01 avatar May 19 '22 12:05 xichen01

@dbw9580 PTAL

xichen01 avatar Jun 07 '22 06:06 xichen01

@dbw9580 Remove Recorder from UnderFileSystemConfiguration, pass Recorder via method parameter now

xichen01 avatar Sep 06 '22 14:09 xichen01

@dbw9580 PTAL thanks!

jiacheliu3 avatar Sep 30 '22 03:09 jiacheliu3

@dbw9580 PTAL thanks!

xichen01 avatar Oct 25 '22 11:10 xichen01

alluxio-bot, merge this please

jiacheliu3 avatar Dec 05 '22 07:12 jiacheliu3