proftpd
A lot of connections in TIME_WAIT state when transfering files with ProFTPD
Hello,
I'm using proftpd with ISPmanager Business on server with more than 300 users. Also I use CSF as a firewall. But when using ftp to transfer files to/from server I see that proftpd spawns a lot of connections in TIME_WAIT state. I guess that 1 transferred file = 1 connection in TIME_WAIT state. In my /etc/csf/csf.conf I've set CT_LIMIT parameter to 200. So that will block IP that has more than 200 connections to server.
What I Did
In my /etc/proftpd.conf I've tried to set the following directives:
MaxClientsPerHost 10 "Sorry, the maximum number clients (%m) from your host are already connected."
MaxClientsPerUser 10
IdentLookups off
AllowStoreRestart on
AllowRetrieveRestart on
TimeoutNoTransfer 65534
TimeoutIdle 65534
So I wanted to limit the amount of connections from ftp users (that use filezilla with default settings to transfer files) to server with MaxClientsPerHost / MaxClientsPerUser directives. That are set in server config. But unfortunately - that didn't help. Users are always using ftp to upload sites and are blocked by CSF because there appears more than 200 connections to the server, where only serveral are in ESTABLISHED state, others are in TIME_WAIT. I can't change this amount of connections in CSF, cause it blocks most of DoS attacks on the server.
What I Expected/Wanted
I expect that ProFTPD can "reuse" connections to transfer files in passive mode. I've checked one shared server that uses cpanel with pureftpd - when transfering 4500 files - it creates only ~20 connections per IP. So I guess there must be the same option in ProFTPD, that makes it to reuse connections, and don't spawn so many TIME_WAIT connections.
There is no way to update ProFTPD to the latest version, cause I'm bount to ISPmanager updates. Also, I didn't find anything in mailing lists about this option
ProFTPD Version and Configuration
proftpd -V
Compile-time Settings:
Version: 1.3.5e (maint)
Platform: LINUX [Linux 3.10.0-962.3.2.lve1.5.24.9.el7.x86_64 x86_64]
Built: Tue Jul 23 2019 14:04:17 UTC
Built With:
configure '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--libexecdir=/usr/libexec/proftpd' '--localstatedir=/run/proftpd' '--disable-strip' '--enable-ctrls' '--enable-dso' '--enable-facl' '--enable-ipv6' '--enable-memcache' '--enable-nls' '--enable-openssl' '--enable-pcre' '--enable-shadow' '--enable-tests' '--with-libraries=/usr/lib64/mysql' '--with-includes=/usr/include/mysql' '--with-modules=mod_readme:mod_auth_pam:mod_tls' '--with-shared=mod_sql:mod_sql_passwd:mod_sql_mysql:mod_sql_postgres:mod_sql_sqlite:mod_quotatab:mod_quotatab_file:mod_quotatab_ldap:mod_quotatab_radius:mod_quotatab_sql:mod_ldap:mod_ban:mod_wrap:mod_ctrls_admin:mod_facl:mod_load:mod_vroot:mod_radius:mod_ratio:mod_rewrite:mod_site_misc:mod_exec:mod_shaper:mod_geoip:mod_wrap2:mod_wrap2_file:mod_wrap2_sql:mod_copy:mod_deflate:mod_ifversion:mod_qos:mod_sftp:mod_sftp_pam:mod_sftp_sql:mod_tls_shmcache:mod_tls_memcache:mod_ifsession' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic'
CFLAGS: -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -Wall
LDFLAGS: -L$(top_srcdir)/lib -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/usr/lib64/mysql -L/usr/lib64/mysql -L/usr/lib64
LIBS: -lacl -lpcreposix -lpcre -lssl -lcrypto -lssl -lcrypto -lcap -lmemcached -lmemcachedutil -lssl -lcrypto -lpam -lsupp -lcrypt -ldl
Files:
Configuration File:
/etc/proftpd.conf
Pid File:
/run/proftpd/proftpd.pid
Scoreboard File:
/run/proftpd/proftpd.scoreboard
Header Directory:
/usr/include/proftpd
Shared Module Directory:
/usr/libexec/proftpd
Features:
- Autoshadow support
+ Controls support
+ curses support
- Developer support
+ DSO support
+ IPv6 support
+ Largefile support
- Lastlog support
+ Memcache support
+ ncursesw support
+ NLS support
+ OpenSSL support (FIPS enabled)
+ PCRE support
+ POSIX ACL support
+ Shadow file support
+ Sendfile support
+ Trace support
Tunable Options:
PR_TUNABLE_BUFFER_SIZE = 1024
PR_TUNABLE_DEFAULT_RCVBUFSZ = 8192
PR_TUNABLE_DEFAULT_SNDBUFSZ = 8192
PR_TUNABLE_GLOBBING_MAX_MATCHES = 100000
PR_TUNABLE_GLOBBING_MAX_RECURSION = 8
PR_TUNABLE_HASH_TABLE_SIZE = 40
PR_TUNABLE_NEW_POOL_SIZE = 512
PR_TUNABLE_SCOREBOARD_BUFFER_SIZE = 80
PR_TUNABLE_SCOREBOARD_SCRUB_TIMER = 30
PR_TUNABLE_SELECT_TIMEOUT = 30
PR_TUNABLE_TIMEOUTIDENT = 10
PR_TUNABLE_TIMEOUTIDLE = 600
PR_TUNABLE_TIMEOUTLINGER = 30
PR_TUNABLE_TIMEOUTLOGIN = 300
PR_TUNABLE_TIMEOUTNOXFER = 300
PR_TUNABLE_TIMEOUTSTALLED = 3600
PR_TUNABLE_XFER_SCOREBOARD_UPDATES = 10
/etc/proftpd.conf
ServerName "ProFTPD server"
ServerIdent on "FTP Server ready."
ServerAdmin root@localhost
DefaultServer on
MaxClientsPerHost 10 "Sorry, the maximum number clients (%m) from your host are already connected."
MaxClientsPerUser 10
IdentLookups off
AllowStoreRestart on
AllowRetrieveRestart on
TimeoutNoTransfer 65534
TimeoutIdle 65534
DefaultRoot ~
AuthPAMConfig proftpd
AuthOrder mod_auth_file.c
UseReverseDNS off
User nobody
Group nobody
MaxInstances 20
UseSendfile off
LogFormat default "%h %l %u %t \"%r\" %s %b"
LogFormat auth "%v [%P] %h %t \"%r\" %s"
LoadModule mod_ctrls_admin.c
LoadModule mod_vroot.c
ModuleControlsACLs insmod,rmmod allow user root
ModuleControlsACLs lsmod allow user *
ControlsEngine on
ControlsACLs all allow user root
ControlsSocketACL allow user *
ControlsLog /var/log/proftpd/controls.log
<IfModule mod_ctrls_admin.c>
AdminControlsEngine on
AdminControlsACLs all allow user root
</IfModule>
<IfModule mod_vroot.c>
VRootEngine on
</IfModule>
<IfDefine TLS>
TLSEngine on
TLSRequired off
TLSRSACertificateFile /etc/pki/tls/certs/proftpd.pem
TLSRSACertificateKeyFile /etc/pki/tls/certs/proftpd.pem
TLSCipherSuite ALL:!ADH:!DES:!SSLv2:!SSLv3
TLSOptions NoCertRequest NoSessionReuseRequired
TLSVerifyClient off
TLSLog /var/log/proftpd/tls.log
<IfModule mod_tls_shmcache.c>
TLSSessionCache shm:/file=/var/run/proftpd/sesscache
</IfModule>
</IfDefine>
<IfDefine DYNAMIC_BAN_LISTS>
LoadModule mod_ban.c
BanEngine on
BanLog /var/log/proftpd/ban.log
BanTable /var/run/proftpd/ban.tab
BanOnEvent MaxLoginAttempts 2/00:10:00 01:00:00
BanMessage "Host %a has been banned"
BanControlsACLs all allow user ftpadm
</IfDefine>
<IfDefine QOS>
LoadModule mod_qos.c
QoSOptions dataqos throughput ctrlqos lowdelay
</IfDefine>
<Global>
Umask 022
AllowOverwrite yes
<Limit ALL SITE_CHMOD>
AllowAll
</Limit>
</Global>
<IfDefine ANONYMOUS_FTP>
</IfDefine>
AuthUserFile /etc/proftpd.passwd
RequireValidShell off
ListOptions -la
PassivePorts 49152 65534
TransferLog /usr/local/mgr5/var/xferlog
Hope to hear from you soon
Some good background reading on the TIME_WAIT state, and why it's a good thing:
- https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux
Of particular interest is the section on socket lingering.
That made me remember the TimeoutLinger directive. In newer versions of ProFTPD, that default TimeoutLinger value is 10 secs; in your older version, we can see that it's 30 secs:
PR_TUNABLE_TIMEOUTLINGER = 30
Thus you might try adding this to your proftpd.conf:
TimeoutLinger 1
I can't recall of your older ProFTPD version would allow a value of 0 secs or not.
Some good background reading on the
TIME_WAITstate, and why it's a good thing:
- https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux
Of particular interest is the section on socket lingering.
That made me remember the
TimeoutLingerdirective. In newer versions of ProFTPD, that defaultTimeoutLingervalue is 10 secs; in your older version, we can see that it's 30 secs:PR_TUNABLE_TIMEOUTLINGER = 30Thus you might try adding this to your
proftpd.conf:TimeoutLinger 1I can't recall of your older ProFTPD version would allow a value of 0 secs or not.
Thanks for the advice. I've tried to add "TimeoutLinger 1" to proftpd.conf and restarted ProFTPD, but that didn't work - proftpd -V shows the same PR_TUNABLE_TIMEOUTLINGER = 30
So 2 questions appears:
- What can be the reason for TimeoutLinger not working?
- Is there any other way to reduce the amount of TIME_WAIT state connections?
You will not see the PR_TUNABLE_TIMEOUTLINGER value change, in the proftpd -V output, even though you have:
TimeoutLinger 1
in your proftpd.conf. The proftpd -V output shows the compile-time defaults; that PR_TUNABLE_TIMEOUTLINGER = 30 is that value that is used when you do not have TimeoutLinger explicitly configured.