[Bug]: Writing files fails when using Azure Data Lake Gen 2

[phishncode]

Apache Hop version?

2.5

Java version?

19.0.2

Operating system

Windows

What happened?

New install of Hop 2.5.0 setup to write to Azure Data Lake Gen 2

name key

create a new work flow name demo

try to save the workflow to azure

org.apache.hop.core.exception.HopException: Error validating file existence for 'azure:\datalakedemo\hop\demo.hwf'

Error saving workflow to file 'azure:\datalakedemo\hop\demo.hwf'

org.apache.commons.vfs2.FileSystemException: Could not create file "azure:///datalakedemo/hop/demo.hwf". Could not create file "azure:///datalakedemo/hop/demo.hwf".

Could not create file "azure:///datalakedemo/hop/demo.hwf".

at org.apache.hop.ui.hopgui.file.workflow.HopGuiWorkflowGraph.saveAs(HopGuiWorkflowGraph.java:3257)
at org.apache.hop.ui.hopgui.delegates.HopGuiFileDelegate.fileSaveAs(HopGuiFileDelegate.java:153)
at org.apache.hop.ui.hopgui.HopGui.menuFileSaveAs(HopGui.java:751)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:578)
at org.apache.hop.ui.core.gui.BaseGuiWidgets.lambda$getListener$1(BaseGuiWidgets.java:216)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:89)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4256)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1066)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4054)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3642)
at org.apache.hop.ui.hopgui.HopGui.open(HopGui.java:474)
at org.apache.hop.ui.hopgui.HopGui.main(HopGui.java:352)

Caused by: org.apache.hop.core.exception.HopException: Error saving workflow to file 'azure:\datalakedemo\hop\demo.hwf'

org.apache.commons.vfs2.FileSystemException: Could not create file "azure:///datalakedemo/hop/demo.hwf". Could not create file "azure:///datalakedemo/hop/demo.hwf".

Could not create file "azure:///datalakedemo/hop/demo.hwf".

at org.apache.hop.ui.hopgui.file.workflow.HopGuiWorkflowGraph.save(HopGuiWorkflowGraph.java:3229)
at org.apache.hop.ui.hopgui.file.workflow.HopGuiWorkflowGraph.saveAs(HopGuiWorkflowGraph.java:3254)
... 12 more

Caused by: org.apache.hop.core.exception.HopFileException:

org.apache.commons.vfs2.FileSystemException: Could not create file "azure:///datalakedemo/hop/demo.hwf". Could not create file "azure:///datalakedemo/hop/demo.hwf".

Could not create file "azure:///datalakedemo/hop/demo.hwf".

at org.apache.hop.core.vfs.HopVfs.getOutputStream(HopVfs.java:327)
at org.apache.hop.ui.hopgui.file.workflow.HopGuiWorkflowGraph.save(HopGuiWorkflowGraph.java:3213)
... 13 more

Caused by: org.apache.commons.vfs2.FileSystemException: Could not create file "azure:///datalakedemo/hop/demo.hwf". at org.apache.commons.vfs2.provider.AbstractFileObject.createFile(AbstractFileObject.java:328) at org.apache.hop.core.vfs.HopVfs.getOutputStream(HopVfs.java:301) at org.apache.hop.core.vfs.HopVfs.getOutputStream(HopVfs.java:325) ... 14 more Caused by: org.apache.commons.vfs2.FileSystemException: Could not write to "azure:///datalakedemo/hop/demo.hwf". at org.apache.commons.vfs2.provider.AbstractFileObject.getOutputStream(AbstractFileObject.java:1277) at org.apache.commons.vfs2.provider.AbstractFileObject.getOutputStream(AbstractFileObject.java:1239) at org.apache.commons.vfs2.provider.AbstractFileObject.createFile(AbstractFileObject.java:322) ... 16 more Caused by: com.microsoft.azure.storage.StorageException: Specified feature is not yet supported for hierarchical namespace accounts. at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87) at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:196) at com.microsoft.azure.storage.blob.CloudPageBlob.create(CloudPageBlob.java:538) at com.microsoft.azure.storage.blob.CloudPageBlob.openOutputStreamInternal(CloudPageBlob.java:1055) at com.microsoft.azure.storage.blob.CloudPageBlob.openWriteNew(CloudPageBlob.java:930) at org.apache.hop.vfs.azure.AzureFileObject.doGetOutputStream(AzureFileObject.java:450) at org.apache.commons.vfs2.provider.AbstractFileObject.getOutputStream(AbstractFileObject.java:1273) ... 18 more

Issue Priority

Priority: 2

Issue Component

Component: VFS

[phishncode]

This shows that VFS is creating folders and writing small files to the directory

[hansva]

It seems not all features work on data lake gen 2, do you have similar options when using a gen 1 blob storage?

[phishncode]

Thank you for the quick reply. It looks like Azure does not allow gen 1 storage anymore.

Is this a problem with VFS or specifically with HOP. I could look into fixing the issue, I would just need some pointers.

[hansva]

It seems we are doing something that was working in Gen1 but no longer works in Gen2

this is the error thrown by Azure Caused by: com.microsoft.azure.storage.StorageException: Specified feature is not yet supported for hierarchical namespace accounts.

The VFS implementation that we have written for azure can be found here:

https://github.com/apache/hop/tree/master/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure

[hansva]

should be solved in the near future by #3676