Performance and permutation upgrade.

[stephanpieterse]

Turned all options OFF by default, command line parameters turn them on. Added some new common words (most common passwords + general words in passwords) Output is now immediately sent to STDOUT, instead of being stored in an array. This significantly reduces memory used, and allows for easy piping to other scripts for example ssh-keygen to test for passwords. Please note that redirecting to file (> output.txt) is going to munch all your disk space pretty quick on larger sets. Added a new --prep-words flag, which allows words to be modified before testing for permutations. See note [1] *All this might probably break any scripts using previous versions.

Performance notes: Options --upper --lower --capital --prep-words generate a 986409 wordlist in roughly 1 seconds using 3 starting words. However, adding more words or options will EXPONENTIALLY increase the wordlist as well as the time taken to generate it.

Other notes: [1] For wordlist: foo, bar, baz

With --upper, without --prep-words:

foo FOO bar BAR baz BAZ foobar FOOBAR foobaz FOOBAZ barfoo BARFOO barbaz BARBAZ bazfoo BAZFOO bazbar BAZBAR foobarbaz FOOBARBAZ foobazbar FOOBAZBAR barfoobaz BARFOOBAZ barbazfoo BARBAZFOO bazfoobar BAZFOOBAR bazbarfoo BAZBARFOO ......30 Combinations

With --upper, with --prep-words: foo FOO bar BAR baz BAZ fooFOO foobar fooBAR foobaz fooBAZ FOOfoo FOObar FOOBAR FOObaz FOOBAZ barfoo barFOO barBAR barbaz barBAZ BARfoo BARFOO BARbar BARbaz BARBAZ bazfoo bazFOO bazbar bazBAR bazBAZ BAZfoo BAZFOO BAZbar BAZBAR BAZbaz fooFOObar fooFOOBAR ... ... ... ...... 1956 Total combinations

Signed-off-by: Stephan Pieterse [email protected]

[ethicalhack3r]

Thanks for the pull request!

I like the idea of straight to STDOUT instead of an array, although this does then not allow to uniq the final result within the script itself. I'm wondering if choosing a default and allowing both options would be a good idea.

Same with turning all options on/off. I think it would be best to choose a default, but allow both options.

What are your thoughts @digininja?

[digininja]

I'll have a look later or tomorrow.

[digininja]

Its been a long day, can you explain what --prep-words is doing, I can't see from the example.

I sort of like the idea of sending directly to STDOUT but it does create the problem of non-unique values which isn't fixed by using uniq. uniq only looks for recurring non-unique words not those with gaps between them, for example:

$ cat a
a
a
a
b
b
b
a
b
x
p
a

$ cat a | uniq
a
b
a
b
x
p
a

You still get the non-unique values. The only way I know to prevent this is to pipe through sort -u which then holds all the words in memory and so would be just as much a memory hog as doing it internally and would prevent words from going directly to STDOUT. An option to dump straight out would probably be the best idea as it would allow those who don't care about uniques to run in your way and those who do to stick with as it is now.

I think having things on by default was a bad idea but also don't like breaking backward compatibility so I wouldn't be happy changing that.

[stephanpieterse]

The idea of --prep-words is to delay the execution of the permutations section, and to use the variations of the initial wordlist (upper, lower, common, etc) as part of the wordlist. So instead of entire uppercase strings, we can have certain words in uppercase and some in lower, or capitalized.

I've run the script (the version for the pull request) now to test, with prep-word, uppercase, lowercase, and capitilisation on a wordlist with 3 words, and "cat outputlist | sort -u | wc -l" , as well as "cat outputlist | wc -l" , are returning the same value. So the output is fine with prep-words enabled, and seems to be unique without it as well. I think as long as the original wordlist used doesn't contain any duplicates or variations of those words as generated by the script, no additional checking for duplicates needs to be done. If that's still the case with some more testing, it would seem the problem of the massive amounts of memory or disk space needed to check through all the possible combinations is solved as long as the original wordlist is checked.

I don't like breaking backward compatibility either... having all the options on is only a REALLY bad idea in the case of --prep-words, as the number of combinations becomes really really big very quickly. Perhaps have all the options on by default as it was, and flipping the behaviour when --prep-words is passed? Or a warning, perhaps with a quick calculation just to show how many words are being generated?