Custom Projects List Feature

[mrthankyou]

trafficstars

Issues Resolved

#11 #15

What Does This PR Do

This PR adds an optional choice for users to add projects they follow to custom lists. The way this works is as follows:

# This command finds repositories based on a search term. The projects are then followed. Please note that the custom list name will be the name of the custom list you want these projects to eventually go to.
python3 follow_repos_by_search_term.py javascript <keyword_search> <custom_list_name>

# This command actually moves the projects to your custom list and then unfollows the projects.
python3 move_repos_to_lgtm_lists.py

One neat feature is that if a project is still being built and we can't move it to the custom list, the move_repos_to_lgtm_lists.py script will stop and report that the project is still being built. This informs the user that they must wait at a later time to re-run the script

Besides this major new feature, there are numerous under-the-hood/misc changes:

• The SimpleProject class was updated to handle processing LGTM projects for our new feature. It could use some refactoring down the road but for now it works.
• Code in lgtm.py was refactored to allow this new feature to take advantage of some of the baked-in logic in dealing with proto vs. real projects.
• A new utils/cacher.py was added to handle caching projects in a txt file. This txt file can then later be used in the move_repos_to_lgtm_lists.py script.
• A unfollow_all_followed_projects.py script was added. I mostly used this for testing purposes but since I wrote it felt like I should keep it in the PR.
• Added Python documentation to several methods.
• All cached files are now placed in the cache folder.

Anything Else We Should Know

This took way longer than expected. As I wrap up this PR I want to note that so far I've experienced no errors in testing this new functionality. If possible, I encourage others to test this as well. Although I think all the kinks are out I've grown weary of LGTM and their internal workings.

I also have to admit that this code can be better. But I've spent more than two weeks on this and am a bit exhausted and would like to go back to writing CodeQL queries. If you decide to shelve this for some time as you want to let this feature "bake" so to speak, that's fine with me. I'll be using this script on a daily basis so if there are more bugs I'll find them.

[mrthankyou]

@JLLeitschuh,

If you want to, you can start taking a look at the work I've done. It would be appreciated. That way, I can catch any errors early on before I invest more time in polishing/testing code.

[lgtm-com[bot]]

This pull request introduces 4 alerts when merging e69003a9f091831ed9e837300934aa62ba1402f8 into 00273ac69927084605a37d937fddd12b9caa9534 - view on LGTM.com

new alerts:

• 2 for Unused import
• 1 for Unused local variable
• 1 for Wrong number of arguments in a call

[JLLeitschuh]

I'm a fan of this! Nice work!

[lgtm-com[bot]]

This pull request introduces 2 alerts when merging 69543fb298a1d812fd9b5b93111988bcb8a076c6 into 00273ac69927084605a37d937fddd12b9caa9534 - view on LGTM.com

new alerts:

• 2 for Unused import

[lgtm-com[bot]]

This pull request introduces 2 alerts when merging a853b9821be9980b580559594e5db947f414047a into 00273ac69927084605a37d937fddd12b9caa9534 - view on LGTM.com

new alerts:

• 1 for Unused local variable
• 1 for Unused import

[mrthankyou]

Short update:

I'm still working on sorting out bugs for this code. I keep running into really really nasty issues that are both unclear and frustrating to deal with during testing. I might have this PR officially ready sometime next week.

[lgtm-com[bot]]

This pull request introduces 2 alerts when merging f44b9593baf287fd573d37dbc2f7d95ed6ce0fe5 into 4cbdb21de819645ca1ffdb95047d7028e6999de9 - view on LGTM.com

new alerts:

• 2 for Unused import

[mrthankyou]

@JLLeitschuh This is now ready for review. I've updated the PR description with details on this PR request. Thank you.

[mrthankyou]

I'm holding off on fixing these changes. I've been exploring other code options and would like to field-test this code a bit more before I commit a thumps-up from me for this PR. I hope that's ok with you.

[mrthankyou]

Short update: I'm actively using these scripts and have squashed most potential bugs that come from it. I think I want to spend a little more time working out these scripts. Interestingly enough I'm using a script that utilizes ghtopdep to collect repos based on a libraries dependencies and have found great results from that. Almost every query I'm running produces hits. Obviously this only works if you have a dependency library you're query identifies vulnerabilities in.