docker-airflow
docker-airflow copied to clipboard
Detect and copy files in a shared windows folder from under the airflow docker container.
I need my DAG to check for a file in a shared folder in the Windows operating system.
A suitable solution was to use the FileSensor tool.
I use this tool as follows:
wait_file = FileSensor(
task_id="wait_file",
filepath="//192.168.0.100/shared/log.csv",
fs_conn_id='my_smb_connect',
poke_interval = 3,
timeout = 15
)
There are no problems when using folders in the local file system.
I was setting up a connection via the web interface Admin -> Connections.
I set Connection Type to File (path) and everything worked.
However, to access remote resources in Windows, you must use the SMB protocol. I have installed additional packages in the airflow image apache-airflow-providers-samba because it is not available by default.
My connection settings are following:
Connection Id: my_smb_connect
Connection Type: Samba
Host: 192.168.0.100
Schema:
Login: user
Password: ****
Port: 445
But this does not give any result. The file is still not visible. Please help me, is it the wrong connection setup?