HS58 Linux: sshd process crashes on "Invalid Read" when issuing a command using Python's Fabric package

[srusecki]

Sometimes when sending commands to Linux over SSH from Python scripts, sshd process on Linux crashes with "Invalid Read" message.

Minimal Python example:

from fabric import Connection

connection = Connection(
    host="10.10.10.10",
    user="root",
    port=22,
    connect_kwargs={"password": "mypassword"})

connection.open()

index = 0
while True:
    for _ in range(10):
        connection.run("uname", hide=True)
        index += 1
    print("Commands count:", index)

After a while of sending uname commands in a loop the sshd crashes. Linux shows the following message:

# potentially unexpected fatal signal 11.
Path: /usr/sbin/sshd
CPU: 2 PID: 22479 Comm: sshd Tainted: G           O      5.15.127 #2
Invalid Read @ 0x0000003c by insn @ 0x20228cee
  @off 0x164cee in [/usr/lib/libcrypto.so.3]  VMA: 0x200c4000 to 0x202f0000
ECR: 0x00050100 EFA: 0x0000003c ERET: 0x20228cee
STAT: 0x80080282 [IE U     ]   BTA: 0x2035fbd0
 SP: 0x5f85f520  FP: 0xbcb494e4 BLK: 0x20229542
r00: 0x400e9540 r01: 0x400e95a8 r02: 0xec61ff54
r03: 0x00000080 r04: 0x27f79973 r05: 0xd830f2ea
r06: 0x9db4cdba r07: 0x6d8e9534 r08: 0x0384c021
r09: 0x4a54ebd5 r10: 0x00000010 r11: 0xa48f14f4
r12: 0x33d6d26f r13: 0x00000000 r14: 0x5f85f588
r15: 0x202e6030 r16: 0x5f85f58c r17: 0x5d1b18c8
r18: 0x49bedbf8 r19: 0x5f85f584 r20: 0x0000001c
r21: 0x25869130 r22: 0x800001f6 r23: 0x81937f17
r24: 0x00000012 r25: 0x00000011

Knowledge so far:

• issue doesn't reproduce when running Linux on a single core,
• reproduces only when using Python's Fabric module or underlying Paramiko module, which is a Python SSHv2 implementation. No reproduction with standard SSH tools,
  • Paramiko by default uses aes128-ctr encryption and hmac-sha2-256 mac, the same parameters can be applied in SSH tool but the result is the same,
• reproduction is somewhat random, sometimes first sent command will break sshd, sometimes it takes couple thousand commands, but most of the time it can be reproduced in a few seconds,
• command used doesn't matter, in the example uname is sent and the reply is simply Linux,
• seen on OpenSSH_9.3p2 with OpenSSL 3.0.12. Using OpenSSH_9.6p1 fixes the issue, though the root cause is unknown.

This issue looks similar to https://github.com/foss-for-synopsys-dwc-arc-processors/linux/issues/163 but is much easier to trigger.