mirror using fxp = stalling and incomplete

[kornos]

hello

I have lftp version 4.8.4 /cygwin I am trying to transfer some folder from one server to another using lftp like this:

set ftp:use-fxp true set ssl:verify-certificate false set net:timeout 5 set net:max-retries 20 set net:reconnect-interval-base 5 set mirror:parallel-transfer-count 10 set ftp:ssl-allow false

open -p port_srcip_src user user_src pass_src repeat 10s mirror --exclude-glob somefile --parallel=10 localdir_src ftp://user_dst:pass_dst@ipaddr_dst:port_dst/remotedir_dst

Line connection is good , ftprush program does it without any issue. with lftp I get stalls at "Getting directory contents (0) [Receiving data]" and errors like:

mirror: thetest.r30: Fatal error: 500 'TAT': Command not understood (this error is not for all files) Some files are missing on destination dirs despite lftp not issuing any error. Folder I'm trying to mirror has around 7Gb of roughly 70 files.

Errors dont occurs exactly at the beginning of transferring process, but at around 50%.

Can u suggest any fix? Thanks

[lavv17]

Try "set ftp:use-stat off"

ср, 25 сент. 2019, 23:01 The Fscking Sultan [email protected]:

hello

I am trying to transfer some folder from one server to another using lftp like this:

set ftp:use-fxp true set ssl:verify-certificate false set net:timeout 5 set net:max-retries 20 set net:reconnect-interval-base 5 set mirror:parallel-transfer-count 10 set ftp:ssl-allow false

open -p port_srcip_src user user_src pass_src repeat 10s mirror --exclude-glob somefile --parallel=10 localdir_src ftp://user_dst:pass_dst@ipaddr_dst:port_dst/remotedir_dst

Line connection is good , ftprush program does it without any issue. with lftp I get stalls at "Receiving data" and errors like:

mirror: thetest.r30: Fatal error: 500 'TAT': Command not understood Some files are missing on destination dirs despite lftp not issuing any error. Folder I'm trying to mirror has around 7Gb of roughly 70 files.

Errors dont occurs exactly at the beginning of transferring process, but at around 50%.

Can u suggest any fix? Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/550?email_source=notifications&email_token=AAHLWXBM6LH5QETREW35LVLQLO7RXA5CNFSM4I2RNRW2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HNWM3GA, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHLWXBI6ZQNJ2E6BS4BRSLQLO7RXANCNFSM4I2RNRWQ .

[lavv17]

Also enable debug in lftp to see what is going on (command debug)

чт, 26 сент. 2019, 6:32 Alexander Lukyanov [email protected]:

Try "set ftp:use-stat off"

ср, 25 сент. 2019, 23:01 The Fscking Sultan [email protected]:

hello

I am trying to transfer some folder from one server to another using lftp like this:

set ftp:use-fxp true set ssl:verify-certificate false set net:timeout 5 set net:max-retries 20 set net:reconnect-interval-base 5 set mirror:parallel-transfer-count 10 set ftp:ssl-allow false

open -p port_srcip_src user user_src pass_src repeat 10s mirror --exclude-glob somefile --parallel=10 localdir_src ftp://user_dst:pass_dst@ipaddr_dst:port_dst/remotedir_dst

Line connection is good , ftprush program does it without any issue. with lftp I get stalls at "Receiving data" and errors like:

mirror: thetest.r30: Fatal error: 500 'TAT': Command not understood Some files are missing on destination dirs despite lftp not issuing any error. Folder I'm trying to mirror has around 7Gb of roughly 70 files.

Errors dont occurs exactly at the beginning of transferring process, but at around 50%.

Can u suggest any fix? Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/550?email_source=notifications&email_token=AAHLWXBM6LH5QETREW35LVLQLO7RXA5CNFSM4I2RNRW2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HNWM3GA, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHLWXBI6ZQNJ2E6BS4BRSLQLO7RXANCNFSM4I2RNRWQ .

[kornos]

ok, enabled debug and "set ftp:use-stat off".

1. Error below doesn't appear anymore mirror: thetest.r30: Fatal error: 500 'TAT': Command not understood (this error is not for all files)

2.Other relevant things I am seeing in debug.log:

• SITE CHMOD 644 file.rar -> you dont have access to this command(but this looks benign) ..... and we reach the point of stall:
• I see a long list of MDTM rar_vol.rar.. up to the one where stalling occured...
• at some point in the middle of MDTM list: --- 227 Entering Passive Mode (........) ---- Connecting data socket to (.........) port 31552 and an endless loop of:

---- Data connection established ---> LIST <--- 150 Opening ASCII mode data connection for directory listing. <--- 226- [Ul: 0.0MB] [Dl: 0.0MB] [Speed: 22.40K/s] [Free: ....MB] **** Timeout - reconnecting ---- Closing data socket ---- Closing control socket ---> TYPE A <--- 200 Type set to A. ---> PASV <--- 227 Entering Passive Mode (...............) ---- Connecting data socket to (.........) port 30149 ---- Data connection established ---> LIST <--- 150 Opening ASCII mode data connection for directory listing. <--- 226- [Ul: 0.0MB] [Dl: 0.0MB] [Speed: 20.99K/s] [Free: ....MB] **** Timeout - reconnecting ---- Closing data socket ---- Closing control socket ---> TYPE A <--- 200 Type set to A. ---> PASV <--- 227 Entering Passive Mode (........) ---- Connecting data socket to (.........) port 30869 ---- Data connection established

Line quality is very good,tested with other fxp clients, I think that timeout is not related to that.

[ml-]

IIRC the following were recently defaulted to yes:

ftp:ssl-protect-data
ftp:ssl-protect-fxp
ftp:ssl-protect-list

I'm not sure if your set ftp:ssl-allow false setting disables all SSL. Might want to turn the protect settings off since it looks like you don't want SSL.

Your parallel connection seem quiet high, are you sure it works fine with the same settings on FTPRush?

[kornos]

OK , I've added those setting 3 settings above, nothing changed.

"Your parallel connection seem quiet high" -> it is not high for me , because I have a 10gbit/s line which needs to be saturated. For 10gbit/s lines, u need multiple connections(10-15) in order to saturate the line , this is what I use(successfully) for FtpRush.

I made tests with 10, 5 and 1 connection. In all cases it blocks with same error I posted initially + "Getting directory contents (0) [Receiving data]".

Weird issue is that this happens around after 20-25 files, not at the beginning. Just a reminder : this transfer is done in FXP mode.

[ml-]

Can you enable the most verbose debug (debug 9) and post all the logs? Don't forget to censor private data. Would be nice to know where it stops.

Also, do you control the FTP Server that is FXPd from/to? Anything helpful in the logs?

[kornos]

The servers are not mine , but I can ask the sysops to check out the logs. "Can you enable the most verbose debug (debug 9) and post all the logs?" -> already done that , I pasted in my second post.(just debug though , not debug 9)

Would be nice to know where it stops. -> it stops randomly , never at the same file, but around half way to complete. It always stops after that long list of MDTM commands lftp is executing from time to time, with the message ""Getting directory contents (0) [Receiving data]"." and printing in a loop what I wrote in my 2nd post. I will try to see if I can get more details , but those are production servers, I am not allowed to run tcpdump on them :\

[lavv17]

The server sends 226- which means another line is coming, but is never received.

сб, 28 сент. 2019, 5:03 The Fscking Sultan [email protected]:

The servers are not mine , but I can ask the sysops to check out the logs. "Can you enable the most verbose debug (debug 9) and post all the logs?" -> already done that , I pasted in the first post.(just debug though , not debug 9)

Would be nice to know where it stops. -> it stops randomly , never at the same file, but around half way to complete. It always stops after that long list of MDTM commands lftp is executing from time to time, with the message ""Getting directory contents (0) [Receiving data]"." and printing in a loop what I posted first. I will try to see if I can get more details , but those are production servers, I am not allowed to run tcpdump on them :\

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/550?email_source=notifications&email_token=AAHLWXHNNYLFUCAJCZID4ATQL23QJA5CNFSM4I2RNRW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD72OJBA#issuecomment-536142980, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHLWXEHDYW6QFNM3KMCURDQL23QJANCNFSM4I2RNRWQ .

[lavv17]

You can try "set use-stat-for-list yes" to avoid LIST command.

сб, 28 сент. 2019, 20:07 Alexander Lukyanov [email protected]:

The server sends 226- which means another line is coming, but is never received.

сб, 28 сент. 2019, 5:03 The Fscking Sultan [email protected]:

The servers are not mine , but I can ask the sysops to check out the logs. "Can you enable the most verbose debug (debug 9) and post all the logs?" -> already done that , I pasted in the first post.(just debug though , not debug 9)

Would be nice to know where it stops. -> it stops randomly , never at the same file, but around half way to complete. It always stops after that long list of MDTM commands lftp is executing from time to time, with the message ""Getting directory contents (0) [Receiving data]"." and printing in a loop what I posted first. I will try to see if I can get more details , but those are production servers, I am not allowed to run tcpdump on them :\

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/550?email_source=notifications&email_token=AAHLWXHNNYLFUCAJCZID4ATQL23QJA5CNFSM4I2RNRW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD72OJBA#issuecomment-536142980, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHLWXEHDYW6QFNM3KMCURDQL23QJANCNFSM4I2RNRWQ .

[kornos]

"You can try "set use-stat-for-list yes" to avoid LIST command."

This will work considering you recommended me above to "set ftp:use-stat off" ?

[lavv17]

Yes, these are different settings

сб, 28 сент. 2019, 21:50 The Fscking Sultan [email protected]:

"You can try "set use-stat-for-list yes" to avoid LIST command."

This will work cause you recommended me above to "set ftp:use-stat off" ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/lavv17/lftp/issues/550?email_source=notifications&email_token=AAHLWXCYJOWWLDRKO232ISTQL6RPNA5CNFSM4I2RNRW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD73AEIQ#issuecomment-536216098, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHLWXCIUJ6TNL4JWUY745TQL6RPNANCNFSM4I2RNRWQ .