docker-airflow icon indicating copy to clipboard operation
docker-airflow copied to clipboard

Detect and copy files in a shared windows folder from under the airflow docker container.

Open coolvs4 opened this issue 1 year ago • 0 comments

I need my DAG to check for a file in a shared folder in the Windows operating system. A suitable solution was to use the FileSensor tool. I use this tool as follows:

wait_file = FileSensor(
        task_id="wait_file",
        filepath="//192.168.0.100/shared/log.csv",
        fs_conn_id='my_smb_connect',
        poke_interval = 3,
        timeout = 15
    )

There are no problems when using folders in the local file system. I was setting up a connection via the web interface Admin -> Connections. I set Connection Type to File (path) and everything worked.

However, to access remote resources in Windows, you must use the SMB protocol. I have installed additional packages in the airflow image apache-airflow-providers-samba because it is not available by default.

My connection settings are following:

Connection Id: my_smb_connect
Connection Type: Samba
Host: 192.168.0.100
Schema: 
Login: user
Password: ****
Port: 445

But this does not give any result. The file is still not visible. Please help me, is it the wrong connection setup?

coolvs4 avatar Jan 14 '24 17:01 coolvs4