Functional test for Frang

[sergsever]

The issue can be reproduced on multiprocessor, on branch sl-396. After set of test requests the system crashes.

    $ tempesta_fw/t/functional/tests.py test_frang
    
    BUG: unable to handle kernel paging request at ffffffffa024d6e0
    IP: [<ffffffffa024d6e0>] 0xffffffffa024d6e0
    PGD 1a07067 PUD 1a08063 PMD 37369067 PTE 0
    Oops: 0010 [#1] SMP
    Modules linked in: tfw_frang(O) tfw_sched_rr(O) tfw_sched_http(O) tfw_sched_hash(O) tempesta_fw(O) tempesta_db(O) tempesta_tls(O) input_leds led_class pcspkr i2c_piix4 i2c_core ppdev parport_pc serio_raw acpi_cpufreq parport fuse ipv6 crc_ccitt autofs4 ata_generic pata_acpi crc32c_intel floppy ata_piix e1000 [last unloaded: tempesta_tls]
    CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O    4.8.15+ #5
    Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
    task: ffffffff81a0b540 task.stack: ffffffff81a00000
    RIP: 0010:[<ffffffffa024d6e0>]  [<ffffffffa024d6e0>] 0xffffffffa024d6e0
    RSP: 0018:ffff88007fc03b30  EFLAGS: 00010246
    RAX: 0000000000000302 RBX: ffff88005b9c3800 RCX: 0000000000000000
    RDX: 0000000000000003 RSI: 0000000000000b00 RDI: ffff88005b9c3800
    RBP: ffff88007fc03b48 R08: 0000000000000000 R09: 000000000000001f
    R10: 00000000a879eb29 R11: 3a68637469777364 R12: ffff88005b9c3e90
    R13: ffff88005b073cb0 R14: ffff88005b073c9c R15: ffff8800374f40f8
    FS:  0000000000000000(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
    CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    CR2: ffffffffa024d6e0 CR3: 00000000373ac000 CR4: 00000000000006f0
    Stack:
     ffffffff81542961 ffff88005b9c3800 ffff8800374f40f8 ffff88007fc03ba0
     ffffffff815430ee 0000000000000025 0000000000000000 ffffffff817eae42
     00008800374f40f8 ffff88005b9c3800 ffff8800374f40f8 ffff88005b073cb0
    Call Trace:
     <IRQ> 
     [<ffffffff81542961>] ? tcp_fin+0x181/0x1f0
     [<ffffffff815430ee>] tcp_data_queue+0x71e/0xcc0
     [<ffffffff81543870>] tcp_rcv_established+0x1e0/0x700
     [<ffffffff8150037d>] ? sk_filter_trim_cap+0x2d/0x150
     [<ffffffff8154df0b>] tcp_v4_do_rcv+0x13b/0x220
     [<ffffffff8154f7eb>] tcp_v4_rcv+0x8eb/0xae0
     [<ffffffff81326629>] ? __dynamic_pr_debug+0x79/0x80
     [<ffffffffa0156c01>] ? tdb_rec_get+0x21/0x60 [tempesta_db]
     [<ffffffff81529260>] ip_local_deliver_finish+0xa0/0x1e0
     [<ffffffff815295b0>] ip_local_deliver+0xc0/0xd0
     [<ffffffffa0bbf5f4>] ? tfw_ipv4_nf_hook+0x124/0x1a0 [tempesta_fw]
     [<ffffffff8151fb6d>] ? nf_iterate+0x5d/0x70
     [<ffffffff81528ef4>] ip_rcv_finish+0xe4/0x3b0
     [<ffffffff81529880>] ip_rcv+0x2c0/0x3b0
     [<ffffffff81528e10>] ? inet_del_offload+0x40/0x40
     [<ffffffff814e8778>] __netif_receive_skb_core+0x2e8/0xaa0
     [<ffffffff814e8f4b>] __netif_receive_skb+0x1b/0x70
     [<ffffffff814ea106>] process_backlog+0x96/0x120
     [<ffffffff814e975d>] net_rx_action+0x1fd/0x350
     [<ffffffff815b9910>] __do_softirq+0xa0/0x2b0
     [<ffffffff810695c8>] irq_exit+0x88/0x90
     [<ffffffff8103f423>] smp_call_function_single_interrupt+0x33/0x40
     [<ffffffff815b900f>] call_function_single_interrupt+0x7f/0x90
     <EOI> 
     [<ffffffff81027360>] ? default_idle+0x20/0xe0
     [<ffffffff81027b2f>] arch_cpu_idle+0xf/0x20
     [<ffffffff810a3035>] default_idle_call+0x35/0x40
     [<ffffffff810a333c>] cpu_startup_entry+0x2fc/0x3a0
     [<ffffffff815ac637>] rest_init+0x77/0x80
     [<ffffffff81b28ecc>] start_kernel+0x42f/0x43c
     [<ffffffff81b2842f>] x86_64_start_reservations+0x2a/0x2c
     [<ffffffff81b2851b>] x86_64_start_kernel+0xea/0xed
    Code:  Bad RIP value.
    RIP  [<ffffffffa024d6e0>] 0xffffffffa024d6e0
     RSP <ffff88007fc03b30>
    CR2: ffffffffa024d6e0
    ---[ end trace 7cdd439871321491 ]---
    Kernel panic - not syncing: Fatal exception in interrupt
    Kernel Offset: disabled
    ---[ end Kernel panic - not syncing: Fatal exception in interrupt

[keshonok]

The real cause of the crash is that the test script test_frang.py doesn't close all client connections that it opens during the test run. Tempesta is loaded and unloaded for each test run. All open connections are hooked up to internal Tempesta functions in the Linux kernel. When Tempesta is unloaded while there are open client connections, any action on those connections causes an attempt to invoke a Tempesta's internal function that is no longer there. That causes the crash.

The script must close all connections that it opens before it unloads Tempesta.

Note that currently Tempesta doesn't unload correctly precisely because of that - it doesn't close open connections, and so a crash may occur due to invocation of functions that are no longer loaded in the kernel. That is about to be fixed very soon, and so Tempesta will close all open connection itself before it's unloaded. Still, the script has full control of the client connections, and it must take care of closing them correctly itself.

[krizhanovsky]

The issue relates to #116 and #254.

[krizhanovsky]

Actually PR https://github.com/tempesta-tech/tempesta/pull/679 removed test for Frang, so it must be reimplemented, but it shall be functional test, not a unit test.

[krizhanovsky]

With issue #717 and it's PR #837, please test invalid HTTP response codes like 50 or 700.

With the recent Client resource accounting changes in #1178 please test multiple clients (you can use loopback addresses 127.0.0.0/8) - a blocked IP address (client) must not affect other clients. This must test TDB walk and insertion routines.

[krizhanovsky]

With the changes from #1200 (fix of #1028) please also add following tests (see the Wiki for the logic to be tested):

• [ ] HTTP trailer headers are processed according to HTTP body limits.
• [ ] timeouts must be reproduced and tested
• [ ] chunked messages in different places (ensure that we do call Frang state machine on every skb - keep in mind GRO)

[krizhanovsky]

Security tests for other features:

~Drop HTTP session on security events (#1045)~

• [ ] sessions rate limit (#598) ~HTTP/2 frames rate limit to prevent HTTP/2 floods (#1346).~ ~HTTP/2 streams rate limit to prevent HTTP/2 slow read (#1346).~ ~Check that all the current Frang limits work with HTTP/2~
• [ ] Cause response timeout for HTTP/1.1 ~and HTTP/2 requests~ ~Cause HPACK bomb, probably with many connections if necessary, and make sure that http_header_cnt limit prevents the attack.~ ~Check that Tempesta sends SETTINGS_MAX_HEADER_LIST_SIZE settings parameter to a client if http_header_cnt HTTP limit is enabled.~ to https://github.com/tempesta-tech/tempesta-test/issues/88 ~Check that client_header_timeout limits time from connection established and long TLS handhshake is correctlyu blocked with the limit. If so, please update description of the limit at the HTTP limits Wiki~ to https://github.com/tempesta-tech/tempesta-test/issues/88
• [x] http_host_required must not allow fake host name in the URL, e.g. GET http://www.soso.com/ with the Host: foo.com

Update the usage examples to https://github.com/tempesta-tech/tempesta/wiki/HTTP-security for the limit and add the attacks' descriptions and the mitigation techniques to https://github.com/tempesta-tech/tempesta/wiki/DDoS-mitigation#http-ddos-attacks .

Also fix the disabled tests for the issue.

[krizhanovsky]

One more requirement for test_host_required from https://github.com/tempesta-tech/tempesta-test/pull/219 : with #1629 Tempesta FW processes host argument of Forwarded header, so http_host_required limit adds

Also when Forwarded: header with host parameter is provided, it also must be the same as :authority and host:. In case of multiple Forwarded: headers were provided, only first header need to match.

See https://github.com/tempesta-tech/tempesta/wiki/HTTP-security#message-level-limits

[krizhanovsky]

Please also check that we do verify SNI against Host header: check that mismatched SNI and Host header value with switched on http_host_required true is blocked

[krizhanovsky]

Merged in https://github.com/tempesta-tech/tempesta-test/pull/265