michenriksen
Updated Project Status and Roadmap
I have reached out via Twitter and email several times over the last 6 months to @michenriksen with no response. I am looking to gauge the interest of those that use this with helping to maintain it. I have been using this for about 2 years now and forked it long ago for a separate project adding support for Github Enterprise, Gitlab CE, and Enterprise as well. I have some very rough code for using this with Perforce also.
My goal would be to bring some additional modularity to this. I would fork and then merge the existing PR's that make sense and then start to work through any issues remaining in this repo and look to transfer the rest to the fork. I welcome anyone who wants to help with the triage process.
Once that is done and stability and usability have been ensured I would like to explore using plugins for various types of source control systems such as perforce, svn, mercurial, as well as sites like Pastebin. Again, any help is welcome.
Another improvement I started to make was the move away from regex's or at least make them a secondary discovery method in favor of language-specific parsers. Again my thought would be to create a pluggable architecture that would allow others to contribute without having to learn the entire codebase. The first step in this though would be to break out the regex's into a separate file or repo and write tests for them and get some baseline metrics on their performance.
This is a lot of work but I have a vested interest in expanding and updating the codebase and would love to preserve the community and experience others have built up over the years and continue with this project using as much of the great code and effort already put into this from so many people.
I am using this fork and have merged in the following pull requests:
- [X] #140 @josephmilla
- [x] #141 @josephmilla
- [X] #149 @betobrandao
- [X] #153 @micksmix
- [X] #155 @dcepler
- [X] #156 @dcepler
- [X] #158 @dcepler
- [X] #159 @muffix
- [x] #164 @wdahlenburg
- [X] #177 @plasticuproject
- [X] #188 @da-edra
- [ ] #194 @codeEmitter
- [x] #199 @HollywoodMarks
I am happy to take additional pull requests in that fork to continue moving this project forward.
I am currently taking an inventory of the issues here and looking at a plan forward for any remaining ones.
I am starting to triage issues now and will track the list of current issues here. If possible please open all new issues in this fork. Thanks!
| Issue | Reporter | Type | Status |
|---|---|---|---|
| #197 | @spekulatius | Help Requested | Closed |
| #196 | @harsh-kk | Help Requested | Closed - Dupe |
| #195 | @shadihh9 | Help Requested | In Progress |
| #193 | @codeEmitter | Feature Request | In Progress |
| #192 | @intruderketo | Help Requested | In Progress |
| #191 | @kirtiso | Bug | Closed - Moved to fork |
| #190 | @dzxindex | Help Requested | Closed |
| #189 | @melardev | Feature Request | Closed - Moved to fork |
| #187 | @da-edra | Feature Request | Closed - Merged |
| #186 | @ranjithprethan | Help requested | Closed |
| #185 | @dzhenway | Help Requested | Closed |
| #184 | @prosecurity | Help Requested | Closed - Dupe |
| #183 | @SaurabhDev | Help Requested | Closed |
| #182 | @SaurabhDev | Feature Request | Closed - Dupe |
| #181 | @newpIDC | Help requested | Closed - Needs to be moved to fork |
| #180 | @jarvis-parker | Feature Request | Closed - Moved to fork |
| #179 | @soupnatzi | Help requested | Closed |
| #178 | @iamwilhelm | Help requested | Closed |
| #176 | @VikzSharma | Bug | Closed - Dupe |
| #175 | @leroywomack | Help Requested | Closed - Dupe |
| #172 | @mixua | Feature Request | Closed - Moved to fork |
| #171 | @igorljubuncic | Feature Request | Closed - Moved to fork |
| #170 | @SeanAmmirati | Feature Request | Closed - Merged |
| #169 | @farmankhan977 | Feature Request | Closed - Moved to fork |
| #167 | @TheHmadQureshi | Help Requested | Closed |
| #166 | @derricksong | Help requested | Closed - Dupe |
| #165 | @wdahlenburg | Feature Request | Closed - Moved to fork |
| #163 | @wdahlenburg | Feature Request | Closed - Merged |
| #161 | @payloadartist | Bug | Closed - Moved to fork |
| #157 | @dcepler | Feature Request | Closed - Merged |
| #154 | @dcepler | Feature Request | Closed - Merged |
| #152 | @micksmix | Bug | Closed - Merged |
| #151 | @satssin | Bug | Closed - Dupe |
| #150 | @ajholland | Bug | Closed - Moved to fork |
| #148 | @nuritizra | Feature Request | Closed - Merged |
| #147 | @sohailnajar | Feature Request | Closed - Moved to fork |
| #146 | @hanjian007 | Bug | Closed - Dupe |
| #145 | @0xtavian | Feature Request | Closed - Merged |
| #144 | @josephmilla | Bug | Closed - Moved to fork |
| #143 | @menzow | Feature Request | Closed - Moved to fork |
| #142 | @josephmilla | Feature Request | Closed - Merged |
| #139 | @josephmilla | Feature Request | Closed - Moved to fork |
| #138 | @josephmilla | Feature Request | Closed - Moved to fork |
| #133 | @bugbaba | Feature Request | Closed - Moved to fork |
| #132 | @skim-milk | Feature Request | Closed - Moved to fork |
| #131 | @jaikishantulswani | Help Requested | Closed |
| #130 | @ScatteredThoughts | Feature request | Closed - Moved to fork |
I am now closing out this issue and tracking everything in the new fork. All pull requests with the exception of one have been merged. All open issues have been triaged and either moved or closed. One door shuts, and a new one opens.
Hey thanks for reaching out. I’ve recently been in contact with the original maintainer through a friend. Would you be willing to give him a little more time to look things over please?
Given the nature of the refactorings I needed to do to add the features I did, my PR will most likely not merge cleanly on your fork. It was the perfect stepping off point for what we needed to accomplish for one of operations. Red Team at GitLab is still actively working on enhancements and fixes and there will be lots more work coming. I’m more than willing to share these contributions with the community and would be happy to work with you and the original maintainer if interested.
For context, in order for our team to track efforts more easily given the recently stale status here, we’ve created a pull mirror of this repo to track issues temporarily: https://gitlab.com/gitlab-com/gl-security/gl-redteam/gitrob/-/issues
Could we sync up via zoom next week?
@codeEmitter I am happy to sync with you and @michenriksen as well as figure out a path forward for all of this, ideally, it would be via a single repo and ideally, if all parties were ok with this I would prefer to use the new repo and archive this one with links back and forth and then close the pull requests and issues. Myself making a pull request into this repo is possible, it will just take a little time as the code has diverged and the pull requests stack on top of each other, and getting them working together took a fair amount of time. I would also be happy as originally planned to assume the maintainership of the repo while leaving @michenriksen the owner. I could care less about glory, specifics, and drama, I just want to write code and move this project forward to were it could be a serious open-sourced recon tool for source code. @codeEmitter If your code has diverged I am happy to roll up my sleeves and help to get it cleanly merged as GitLab support was something I didn't even fully implement.
My reasoning for hard forking this was along the same lines as yours. I built a tool based on this that I am not able to share in its current form. It was a more modular version with the following functionality
- Github support
- Github Enterprise support
- more extensible and modular (using cobra)
- 10x speed improvement (think masscan) through continued use of threads
- full support for telemetry and metrics to aide in tuning the performance
- full logging (using logrus)
- unit/integration tests
- ability to scan a single repo
- ability to scan offline git repos
- ability to be run as a service in a CI/CD pipeline
- ability to run in AWS Lamdba at-will when necessary
- all sig's are broken out into a separate repo and independently versioned and releasable
- the ability to display the secret itself
- support for sqlite giving the ability to just do a delta scan or pause/resume
- ability to set the commit depth of a scan
- ability to read configs from a yaml file (using viper)
My version was designed for extensibility and brute, raw performance. I pulled the plug on my dev work about a year ago but the final thing I was working on was a grpc/rpc implementation to allow the addition of new target capabilities or detection methods without having to know the entire codebase. Go plugin was not yet available and it does not work on Windows. When it comes to performance, scanning 100k offline repos with a combined size of multiple TB to full depth in 8-9 hours was entirely possible. I have the release to bring these features into the open but have to implement them cleanly.
I would also throw a github pages site up in hugo for docs, communications, and demos, use cases. This would include registering the domain name of gitrob which I am happy to pay for. Again my intent is not to "steal" the project or splinter it, but to move it forward at a faster more supported pace. Using my original version of this is a weekly task yet I cannot use it on public projects and I miss my capabilities so I want to replicate them.
There is a fair amount of work here and the tool would look differently by the time I was done so before I started I was going to create a requirements doc like any other project and solicit feedback from the community to see how useful these features would be to the greater world and not just my specific use cases.
@codeEmitter @michenriksen Now that I had some non-keyboard time to think about this, here are my current thoughts on the matter. Full disclosure there several people on my team that also use our fork of gitrob and miss the features we implemented and want to recreate them. We tried to get our employer to maintain my fork officially but they gave it a pass so I, being the lead dev, got the ok to have at it. A few people on the team may contribute as time goes on though.
@michenriksen If you want to resume day to day managing of the project again that is fine by me and I will return to the backseat and try to pull in my fork of all the pull requests and bug fixes. I understand how much work can be involved as this is not my first or only rodeo and nowhere near the biggest thing I maintain or advise on. With that said if you have other commitments and simply don't have the time or want to invest the time but don't want to see the project languish or you don't want to give it up I can understand that as well. In that case, if you would like, we can continue with the hard fork, which means I have not forked the git commit history, I pulled the code and removed .git/ and did a clean git init You can close out these issues and pull requests, archive the repo, and my fork can become the new codebase. You still retain everything as I have no plans to ever remove the effort you and the community put into this or your name/copyright from the code. That is not how I work. I grew up on BBS's and USENET when open source was still just chatter at late night MIT hackfests. I would be thrilled to give you full admin rights to my repo and we can collaborate together on the various lines of thinking for moving this forward.
As I said above, documentation is a huge frustration factor and there were several issues that either requested demos/docs or could have been solved with more detailed docs. At some point sooner rather than later I was going to grab the DNS for gitrob.io and standup a quick Hugo site with Github pages and throw all the docs, release info, a demo or two and maybe some related posts on secrets management, and the failing of it as it relates to AWS S3, GDrive, Github/GitLab, Perforce, etc.
With all of that said I still feel this tool is the ideal mix of blue team and red team methodologies. I have used it for both, in a CIO/CD pipeline for catching things before they get built and also for sifting through a company's public repos when I have the need to. Please feel to continue this conversation in this thread, or if you need to reach me privately, you are welcome to DM me on Twitter. I am going to continue on with my fork and cleanup though as that is the best path forward at the moment.
@codeEmitter I took a look at the repo you linked me and your team has some damn good code and ideas in there. A lot of it is not where I see my needs focused but the features and direction are great! I honestly don't think it would be hard to do a transplant if we sat down over time and looked at it from an engineering standpoint and carved out the code.
One of the big things I am looking at is breaking this out into a few packages, and if/when that happens it would be a great time to then take a frozen branch of your code and start to slowly implement it package by package, such as front end, GitLab targets, new regex's and discovery methods, etc. At least from the mile-high view that is what I am looking at. Once that is done we can look at pulling in smaller chunks and then if you wish maintaining parity where you could simply pull from upstream when you wanted or open a new request.
As I have done in the past with major contributors or larger entities, I would be happy to at some point give you commit bit to the repo and the ability to review code to make it easier to function as a cohesive unit. @michenriksen if we go the route of keeping my fork I would be more than happy to do the same and give you admin rights as this was your baby for a long time and you deserve to bask in the glory. My reason for doing what I am doing is that I believe in community and shared responsibility first so I tend to fight to not splinter a repo or community unless absolutely necessary.
I kinda think going with my fork is a little easier as I have open source project management experience as well as developer advocate experience. The repo I have is already plugged in with issue tracking and release management via Zenhub, automated tests via CircleCI, code coverage, vulnerability scanning with synk, and code quality via code climate. These tools make it much easier to maintain and manage a codebase from a birds-eye view, especially when they are all integrated together.
It is still very much a beta project and there are some know bugs with the web interface that will be fixed shortly but wraith is functional and stable at last. The gitlab redteam had a fork of gitrob that they added features to and fixed bugs on. I also had a fork of this from an internal project at my work. I, in conjunction with @codeEmitter combined both projects into a new project, wraith, and have been adding features and fixing bugs there.
There are gaps in the documentation and a few things were broken in the brain surgery, we also uncovered several nasty bugs affecting the functionality of gitrob that we are still working to fix. It is fairly stable though and will work, just not to its full potential yet.
Pull requests and bug/feature requests are welcome!