[WIP] libntoh: switch from broken htable to uthash

[megahall]

Problem Statement

When using libntoh on almost any nontrivial PCAP it fails to work pretty much at all, with tons of NTOH_IP_ADDRESSES_MISMATCH and/or NTOH_TCP_PORTS_MISMATCH errors or other similar errors.

Root Cause

From extensive debugging over many hours, it appears that the simplistic htable implementation included in libntoh is causing the problem:

Detailed Steps

1. The unsigned int key generation code and modulo-to-hash-chain code causes many collisions.
2. The hash chain lookup code did not properly compare the contents of tuple key fields inside of each ntoh_tcp_stream_t, ntoh_ipv4_flow_t, and ntoh_ipv4_flow_t object, to be sure if the records were true duplicates of each other. Instead it just keeps returning semi-random, usually incorrect flow and stream records at the beginning of the hash chains.
3. Because the hash table was buggy, add_fragment and add_segment operations fail with the MISMATCH errors from operating on the wrong socket objects.

History

Some previous attempts to fix the issues with htable are present in the commit history, but they were inconsistently written, not fully implemented, commented out, and/or not actually used in the code performing the find operations for flows and streams and thus ineffective.

Solution

This code appears to fix most of the issues by completely replacing the htable implementation with a known good C hash table named uthash. It passes some basic and medium complex testing but more testing from the community will be required to be 100% sure everything works.

Testing

It should cause little to no change for real applications because it just fixed internal implementation bugs. However some minor unimportant features are temporarily broken, like some of the code which limits the hash table size, as it needs some modifications before it will work with uthash.

uthash information

Detailed information on uthash:

https://github.com/troydhanson/uthash https://troydhanson.github.io/uthash/ https://troydhanson.github.io/uthash/userguide.html

[megahall]

Note: Travis-CI fails presently due to new uthash dependency. It could be fixed using uthash-dev DEB package or uthash RPM package or the relevant package for the Travis-CI environment.

[leonn]

all count_flows functions are not returning the count from uthash @megahall. and there are some unused variables left from the previous implementation