SecLists
SecLists copied to clipboard
danielmiessler
Create list of files in that could go in /etc for fuzzing
This is a list of all of the files that are installed into etc by all deb packages from the debian and ubuntu apt repositories.
Nice! But how did you make this list? I have a feeling it'll have to be constantly maintained
Thanks! While working on a different project, I ended up creating awk and bash scripts to download over 2 million deb files and unpack at least the metadata section of the deb files, so I had access to plenty of data. I also opted to sort it case insensitively so it should be pretty clear to see any new additions in the git diff. This list was created with scripts, so keeping it up to date is definitely something that could be scripted or run semi-annually to pick up any new packages.
Very well. If you can share the scripts you used to generate this wordlist, I may be able to automate it with Github Actions.
But if you're familiar with Github Actions or you'd just like to learn, you can do it yourself by modifying this file https://github.com/ItsIgnacioPortal/SecLists/blob/baaec330cf33ac21061ac9f613a07a5d5908c79e/.github/workflows/wordlist-updater_awesome-list-of-secrets-in-environment-variables.yml
I had an idea... So, the main issue now with the scripts is that they took days to run and downloaded 2 million files. I've been trying to figure out how to turn that into a smaller script that could run in a reasonable amount of time periodically. I think, perhaps, I could keep track of the URLs it has already scanned in a gzip file so they don't need to be repeated and use an awk script to parse the new releases file from each apt repository and check if there's any new deb files to download.
A list of all of the URLs already scanned, compressed in gzip format, is about 12 megs. Since it is possible to append gzip records to a gzip file, we could add to the list of completed URLs after scanning new packages. However, if we append URLs to this after scanning them, and then commit it to git, it will include a separate blob in git each time this file is updated and introduce a lot of bloat. I would like to propose including this 12 megabyte list and starting a new gzipped list of URLs that have been scanned. Both of these lists can be read together to avoid downloading deb files repeatedly. Once the new list grows beyond a certain size, a new file can be created.
the script is working. it takes about 40 minutes for me to run it on an AWS server. interestingly, it does actually find at least one new /etc/ file right now. i am putting the new URLs that get scanned in a file named after the current year, so it will automatically start a new gzip file every year to keep the files small and avoid appending to large files in git. i am going to try out github actions and see if i can hook it up to a cron type of action that runs every month.
this github action works as a manual workflow_dispatch event and as a cron.
A list of all of the URLs already scanned, compressed in gzip format, is about 12 megs. Since it is possible to append gzip records to a gzip file, we could add to the list of completed URLs after scanning new packages.
How big is the file without being compressed?
Thank you (and also for the GitHub actions too!)
If the actions side of things doesn't work out, this can be manually ran just before ever tag release of SecList =)