nextflow
nextflow copied to clipboard
Azure, WARN: Failed to cleanup work dir
Hi NextFlow devs/community,
I'm running into an error when I try to clean my working directories (running on Microsoft Azure). Specifically, in my nextflow.config
, I have (as the last line)
cleanup = true
The error I receive is:
WARN: Unable to start plugin 'nf-azure' required by az://<my_container_name>/work/2e/8cd...
WARN: Failed to cleanup work dir: /my_container_name/work
Here is my log for the run: nextflow.log
WIth the exception of not cleaning the working directory, the job runs as expected (files are read from and created in my Azure blob containers, so I don't think
blob permissions the problem).
Any help with this error is greatly appreciated! Please let me know if I can provide any additional information that might be helpful.
Best, Derek
Hi @des2037 ,
Thanks for letting us know about this issue. I noticed that the nf-azure
plugin version is 0.11.2
Could you please try again with the latest plugin?
You'd need to update your nextflow.config
with the following snippet (from docs)
plugins {
id '[email protected]'
}
If it fails again, please share the .nextflow.log
file with us.
Hi @abhi18av
Thanks very much for the fast reply and suggestion.
I've updated the nf-azure
plugin to 0.12.0
, but sadly I'm still getting the same error.
Here is my nextflow.log: nextflow.log
Thanks for the help!
Thanks @des2037 , I'll look into this soon.
The problem here is that the cleanup happens after the Azure plugin shutdown :/
Feb-28 16:04:09.986 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-azure
Feb-28 16:04:10.052 [main] DEBUG nextflow.plugin.PluginsFacade - Using Default plugins manager
Feb-28 16:04:10.054 [main] INFO org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Feb-28 16:04:10.054 [main] INFO org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Feb-28 16:04:10.055 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Feb-28 16:04:10.057 [main] WARN nextflow.file.FileHelper - Unable to start plugin 'nf-azure' required by az://hwlab-nextflow-temp/work/cf/f5868bd15108a917675cba34e5ecde
https://github.com/nextflow-io/nextflow/blob/0045834c6a36e7e5226b261c51170b3aaedb214a/modules/nextflow/src/main/groovy/nextflow/Session.groovy#L1111-L1111
Hi @pditommaso, is there any news on this issue? Was this addressed? We are seeing the same behavior with AWS S3, Nextflow version 21.10.6 build 5660
:
Jul-12 18:44:22.918 [main] WARN nextflow.file.FileHelper - Unable to start plugin 'nf-amazon' required by s3://***
java.lang.NullPointerException: null
at nextflow.plugin.PluginsFacade.isStarted(PluginsFacade.groovy:290)
at nextflow.plugin.PluginsFacade.startIfMissing(PluginsFacade.groovy:382)
at nextflow.plugin.Plugins.startIfMissing(Plugins.groovy:73)
at nextflow.file.FileHelper.autoStartMissingPlugin(FileHelper.groovy:284)
at nextflow.file.FileHelper.asPath(FileHelper.groovy:264)
at nextflow.Session$_cleanup_closure22.doCall(Session.groovy:1089)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
at groovy.lang.Closure.call(Closure.java:412)
at nextflow.CacheDB.eachRecord(CacheDB.groovy:291)
at nextflow.Session.cleanup(Session.groovy:1085)
at nextflow.script.ScriptRunner.terminate(ScriptRunner.groovy:223)
at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:122)
at nextflow.cli.CmdRun.run(CmdRun.groovy:308)
at nextflow.cli.Launcher.run(Launcher.groovy:480)
at nextflow.cli.Launcher.main(Launcher.groovy:639)
Jul-12 18:44:22.919 [main] DEBUG nextflow.plugin.PluginsFacade - Using Default plugins manager
Jul-12 18:44:22.919 [main] WARN nextflow.Session - Failed to cleanup work dir: /***
Thanks!
Upd: same issue with version 22.04.4, plugin [email protected]
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Related to #3645
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.