"Too many open files" errors cause some targets was missed

[Dliv3]

Steps to reproduce

https://github.com/byt3bl33d3r/CrackMapExec/blob/master/cme/protocols/smb.py#L411

Modify the create_smbv3_conn function to the following code to print exception message

    def create_smbv3_conn(self):
        try:
            self.conn = SMBConnection(self.host, self.host, None, self.args.port)
            self.smbv1 = False
        except socket.error as e:
            logging.debug('SMBv3 connection error on {}: {}'.format(self.host, e))
            return False
        except Exception as e:
            logging.debug('Error creating SMBv3 connection to {}: {}'.format(self.host, e))
            return False

        return True

Then, execute CME, using the default 100 threads or more

python crackmapexec.py -t 100 --verbose smb 10.0.1.1/24

CME verbose output (using the --verbose flag)

DEBUG SMBv3 connection error on 10.0.1.242: [Errno Connection error (10.0.1.242:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.113: [Errno Connection error (10.0.1.113:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.244: [Errno Connection error (10.0.1.244:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.245: [Errno Connection error (10.0.1.245:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.249: [Errno Connection error (10.0.1.249:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.248: [Errno Connection error (10.0.1.248:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.247: [Errno Connection error (10.0.1.247:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.250: [Errno Connection error (10.0.1.250:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.251: [Errno Connection error (10.0.1.251:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.252: [Errno Connection error (10.0.1.252:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.254: [Errno Connection error (10.0.1.254:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.253: [Errno Connection error (10.0.1.253:445)] [Errno 24] Too many open files
DEBUG SMBv3 connection error on 10.0.1.255: [Errno Connection error (10.0.1.255:445)] [Errno 24] Too many open files

CME Version (cme --version)

5.1.7dev

OS

macOS Big Sur 11.2.3

Detailed issue explanation

The Too many open files errors cause some targets was missed in the CME scanning process and final results

I know that I can use ulimit to increase the maximum file descriptors, but It seems better if this issue can be solved in the code level.

[mpgn]

thanks for the report, I will look into it :)

[executionByFork]

Not only a problem on macOS. This exists in Kali Linux as well (Version: 5.1.5dev)

Workarounds until a bugfix is pushed:

• Increase the upper bound on file descriptors with ulimit
• Chunk the scan into smaller batches

[mpgn]

Increase the upper bound on file descriptors with ulimit

That the solution I use. Maybe I should print the error not only in debug mode

[executionByFork]

Printing this error outside of debug mode would certainly help raise awareness of the issue and notify people that data is getting dropped during their scans, but I wouldn't call ulimit a solution. It's a workaround. The issue is in how CME handles connections.

From what I understand, this bug is either caused by too many concurrent connections, or the file descriptors for these sockets are not getting cleaned up properly.

• If the file descriptors aren't being cleaned up programmatically and it is just left up to garbage collection, they can easily pile up, causing the upper bound to be hit and this error to occur before garbage collection attempts to close them. In this case the bug can be fixed by closing file descriptors properly.
• If it is too many concurrent connections, then I imagine it should be possible to limit the number of connections CME attempts to hold open at once, waiting for sockets to close before opening new ones.

[mpgn]

Another idea https://twitter.com/mcohmi/status/1514420094079700994?s=20&t=3QSVZgKiT2NuRKx9hqMv2A

[ZerkerEOD]

Workarounds until a bugfix is pushed:

• Increase the upper bound on file descriptors with ulimit
• Chunk the scan into smaller batches

Where is the ulimit supposed to be set? I don't see it in the cme.conf file.

[executionByFork]

hrm, it does seem ulimit hasn't been explained at all in this thread. That would probably be helpful.

ulimit is short for user limit (I believe) and it is a limiter in place for how many file handles a single user on a system can have open at one time. In other words, it is not a crackmapexec implemented limit, it is a linux system enforced limit. So, you have to change this limit in the system config files, not crackmapexec files.

You can change the ulimit for a user with the command ulimit. There should be a man or help page for it, and also this stack overflow question covers a fair bit about changing the limits. https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user