Fix for the Cloudflare 524 Timeout Error after 99s

[ghost]

The crawl.php code as it is, at line 160 (line 153 for crawl-manual) could use these bits: $spacer_size = 8; // increment me until it works echo str_pad('', (1024 * $spacer_size), "\n"); if(ob_get_level()) ob_end_clean();

It will prevent a inevitable 524 Timeout if you are using Cloudflare services (ran into this issue myself)

Source of fix: https://github.com/marcialpaulg/Fixing-Cloudflare-Error-524

Example in action: https://github.com/safesploit/doogle/commit/dd72700c3919c6d3b1150f06ef726ff081fe479c