alluxio icon indicating copy to clipboard operation
alluxio copied to clipboard
verified publisher icon Alluxio

Cache status after getFingerprint

Open kuszz opened this issue 3 years ago • 7 comments

What changes are proposed in this pull request?

#16444

kuszz avatar Nov 01 '22 11:11 kuszz

@kuszz Thanks for this performance improvement, please fix the style issue.

maobaolong avatar Nov 01 '22 13:11 maobaolong

@kuszz Please care about the failed tests, I help to copy one failed test callstack, it seems broke by this PR.

java.lang.IllegalArgumentException: path name  does not match ufs status name /tmp/junit4471494242978627946/junit7460544120945017143
	at alluxio.underfs.UfsStatusCache.addStatus(UfsStatusCache.java:100)
	at alluxio.master.file.InodeSyncStream.syncExistingInodeMetadata(InodeSyncStream.java:820)
	at alluxio.master.file.InodeSyncStream.syncInodeMetadata(InodeSyncStream.java:730)
	at alluxio.master.file.InodeSyncStream.syncInternal(InodeSyncStream.java:474)
	at alluxio.master.file.InodeSyncStream.sync(InodeSyncStream.java:404)
	at alluxio.master.file.FileSystemMasterSyncMetadataFlushJournalTest.run(FileSystemMasterSyncMetadataFlushJournalTest.java:139)
	at alluxio.master.file.FileSystemMasterSyncMetadataFlushJournalTest.flatDirectory(FileSystemMasterSyncMetadataFlushJournalTest.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:68)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:326)
	at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:89)
	at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:97)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.executeTest(PowerMockJUnit44RunnerDelegateImpl.java:310)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTestInSuper(PowerMockJUnit47RunnerDelegateImpl.java:131)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.access$100(PowerMockJUnit47RunnerDelegateImpl.java:59)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner$TestExecutorStatement.evaluate(PowerMockJUnit47RunnerDelegateImpl.java:147)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.evaluateStatement(PowerMockJUnit47RunnerDelegateImpl.java:107)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:82)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:298)
	at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:87)
	at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:50)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.invokeTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:218)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.runMethods(PowerMockJUnit44RunnerDelegateImpl.java:160)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$1.run(PowerMockJUnit44RunnerDelegateImpl.java:134)
	at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
	at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.run(PowerMockJUnit44RunnerDelegateImpl.java:136)
	at org.powermock.modules.junit4.common.internal.impl.JUnit4TestSuiteChunkerImpl.run(JUnit4TestSuiteChunkerImpl.java:117)
	at org.powermock.modules.junit4.common.internal.impl.AbstractCommonPowerMockRunner.run(AbstractCommonPowerMockRunner.java:57)
	at org.powermock.modules.junit4.PowerMockRunner.run(PowerMockRunner.java:59)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:364)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:237)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:158)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)

maobaolong avatar Nov 02 '22 01:11 maobaolong

@kuszz Please care about the failed tests, I help to copy one failed test callstack, it seems broke by this PR.

java.lang.IllegalArgumentException: path name  does not match ufs status name /tmp/junit4471494242978627946/junit7460544120945017143
	at alluxio.underfs.UfsStatusCache.addStatus(UfsStatusCache.java:100)
	at alluxio.master.file.InodeSyncStream.syncExistingInodeMetadata(InodeSyncStream.java:820)
	at alluxio.master.file.InodeSyncStream.syncInodeMetadata(InodeSyncStream.java:730)
	at alluxio.master.file.InodeSyncStream.syncInternal(InodeSyncStream.java:474)
	at alluxio.master.file.InodeSyncStream.sync(InodeSyncStream.java:404)
	at alluxio.master.file.FileSystemMasterSyncMetadataFlushJournalTest.run(FileSystemMasterSyncMetadataFlushJournalTest.java:139)
	at alluxio.master.file.FileSystemMasterSyncMetadataFlushJournalTest.flatDirectory(FileSystemMasterSyncMetadataFlushJournalTest.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:68)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:326)
	at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:89)
	at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:97)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.executeTest(PowerMockJUnit44RunnerDelegateImpl.java:310)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTestInSuper(PowerMockJUnit47RunnerDelegateImpl.java:131)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.access$100(PowerMockJUnit47RunnerDelegateImpl.java:59)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner$TestExecutorStatement.evaluate(PowerMockJUnit47RunnerDelegateImpl.java:147)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.evaluateStatement(PowerMockJUnit47RunnerDelegateImpl.java:107)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:82)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:298)
	at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:87)
	at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:50)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.invokeTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:218)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.runMethods(PowerMockJUnit44RunnerDelegateImpl.java:160)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$1.run(PowerMockJUnit44RunnerDelegateImpl.java:134)
	at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
	at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.run(PowerMockJUnit44RunnerDelegateImpl.java:136)
	at org.powermock.modules.junit4.common.internal.impl.JUnit4TestSuiteChunkerImpl.run(JUnit4TestSuiteChunkerImpl.java:117)
	at org.powermock.modules.junit4.common.internal.impl.AbstractCommonPowerMockRunner.run(AbstractCommonPowerMockRunner.java:57)
	at org.powermock.modules.junit4.PowerMockRunner.run(PowerMockRunner.java:59)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:364)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:237)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:158)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)

@maobaolong Thanks, I have modified the code.

kuszz avatar Nov 02 '22 11:11 kuszz

@kuszz Left a minor comment.

Otherwise, would you please give more codes or information to prove why and when this can reduce call to ufs.

InodeSyncStream.syncInodeMetadata(line 730) -> syncExistingInodeMetadata(InodeSyncStream line 818, getParsedFingerprint -> BaseUnderFileSystem.getFingerprint -> getStatus) -> mStatusCache.prefetchChildren(InodeSyncStream line 916)

kuszz avatar Nov 03 '22 08:11 kuszz

@kuszz Is there any update?

maobaolong avatar Nov 24 '22 00:11 maobaolong

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Jun 05 '23 15:06 github-actions[bot]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Aug 10 '24 15:08 github-actions[bot]