Making the Whole Project/Repo Menu-Driven

[avinashkranjan]

[UPDATE]: Check this comment for progress on this issue: https://github.com/avinashkranjan/Amazing-Python-Scripts/issues/831#issuecomment-817241049

Is your feature request related to a problem? Please describe. Changing directories looking for scripts to use one by one will be quite a headache.

Describe the solution you'd like Making it a menu-driven script, where a user can select the option and the specific script will run (can take arguments too if needed at runtime) in sort like Lazy Script if you are aware of.

Let me brief you about what you have to do on this issue:

1. Make a menu-driven script which will perform the task as per selected option.
2. There are 100+ scripts currently so we need to provide all in the menu. Which can be done in the following ways
   • Scripts can be further categorized into sub categories
     • Automation
     • AI/ML
     • Calculators
     • Games
     • Fun
     • Scrappers
     • Bots and More
   • It'll help in searching through the numerous script easy
3. Give all the folder 777 permission for the execution of python files
4. Do provide comments while doing scripting
5. Be aware of file path as per linux system ie. /root/your_filename while working on github
6. Follow all the guidelines provided

Marking Guidelines

• Since This is a big Issues, Multiple contributors are welcome to contribute to the script, Depending on the Contribution PR level would be marked..the Issue would be a Level3, Each Linked PR would be of Minimum Level2.

@kaustubhgupta @antrikshmisri @santushtisharma10 Kindly Helpout the Contributors working on this particular Issue..As this one is a complex issue..If Any Issues, DM me on Discord..

Happy Coding..👨🏻‍💻

[AshuKV]

Can I work on this issue under GSSOC21? Kindly assign it to me.

[XZANATOL]

I've a suggestion. We can start by categorizing each script into a dictionary.. something like: {"Scrappers": ["Project1", "Project2"], "Automation":["Project3", "Project4"]} Each list can contain records of scripts that are in another category. This will assure that effiency of sorting and tag searching, and put each script into its suitable place making it organized. This dictionary can be saved in a CSV file for future adds/edits.

We can also make a function to act as an API which adds a new script with the suitable tags, and help contributors to automate the process of adding their scripts to this script instead of manually adding them. (Let's leave that for after we settle on how we will save the data of the existing scripts.)

[Ajay-Singh-Rana]

Hey..! What if we create a script that takes folder names as arguments and lets the user decide which folder thier script is to be moved in to and also an option to add a newer category. script.py folder_name category or script.py -n new_category folder_name [folder_name [folder_name]

[kaustubhgupta]

Hi, @Ajay-Singh-Rana @XZANATOL @AshuKV,

As I have just returned home, I need some time to plan out the overall structure of the script, how it will handle new scripts and how we can have a GUI for that.

I will update you on this day after tomorrow

[Ajay-Singh-Rana]

Any updates..?

[kaustubhgupta]

Any updates..?

Will update by EOD

[avinashkranjan]

Hey @kaustubhgupta If Need any help or any doubt kindly text me on discord..Participants are asking about this Issue..

[kaustubhgupta]

Hi, @AshuKV @XZANATOL @avinashkranjan @Ajay-Singh-Rana (Apologies for updating late, was busy with important work),

I liked the @XZANATOL implementation to create the database of scripts based on categories. I think a big JSON would do the work. Why JSON? Because it can be loaded into the dictionary in Python. Creating this JSON would be a manual task initially as we can't trace back to every merged pull request and check if it was a scrapper or ML script as neither the participants have filled the PR template properly nor there was an option to add a category. @Ajay-Singh-Rana Your idea has a problem that a new user would not be aware of the folder where the script lies.

Also, one major problem apart from these ideas is that all participants have different entry points for their script. Some of them have main.py, scrapper.py, or script-specific names.

Here is my workaround for this issue in 4 tasks (Please read till the end before assuming why I am asking these):

• Creating the manual JSON: This JSON file, as suggested by @XZANATOL will have the root element as Category and its value as another JSON with the following information:

  • Script Name
  • Script folder relative path
  • Entry point of the script (Which file runs the program)
  • arguments it takes (if any). Now here we need to define that if the arguments will be passed using argument parser or direct CLI.
  • requirements_path: None if no requirements.txt file present else the relative path of this file
  • Contributor Name
  • Small Description

• Creating the demo Script for testing: After this database is prepared, we can start working on the implementation part. This demo script will present the user with categories of the script. Then as per user input, a list of available scripts will be presented. and then the user input script will be run. The script can be run by the subprocess module. Why not OS? Because in the subprocess module we can define the script entry point and arguments more easily. Refer this article for implementation. Now as soon as the user gives the script choice, our script should use the database to see if it has requirements.txt. If yes then first run: pip install -r $requirements_path/requirements.txt and then find the entry point of the script to call: python entrypoint.py. This command can be called via subprocess too with arguments.

• Porting this demo to GUI: If this demo script works, then we can port this to a suitable GUI with a menu to select the category, sub-script, and provide arguments if needed. We can also have a checkbox for installing requirements.txt! The GUI can be made in any framework depending upon which contributor takes this sub-issue.

• Automating JSON updation: The manual information adding is a one-time task. To add new script information to this JSON, we will use the PR template and GitHub Actions to add update this information. The PR template needs to be updated with the new parameters mentioned in the manual JSON work and every participant will have to compulsorily fill in this information. The GitHub action, on every PR merged, will push this information to this JSON and hence an application will be converted out of this repo! When this task will be performed. all other PRs will be stopped for a while and they will need to update the PR template accordingly.

• BONUS task, website updation: The website currently has hardcoded information about the scripts. The JSON generated can be used by web designers to display script information with contributor name, small description, and link to the script. That's why I asked in the PR template this information in the first place! This will help both projects!

As these tasks are themselves way too big, every task will be rewarded level3. Also, no separate issue will be created for this. All 4-5 PRs will be merged tagging this issue only. The subtasks will be assigned and ticked off in the order mentioned below in this comment only:

• [x] Creating Manual JSON - @XZANATOL (#963)
• [ ] Creating demo Script - @devRawnie
• [x] Automating JSON updation - @XZANATOL (#1025)
• [x] BONUS task, Website updation - @Ayushjain2205 (#1030)
• [ ] Porting demo to GUI

Let me know what should be modified, any new ideas except this, or anything else. Also, which task you are interested in.

VERY IMPORTANT INFO: THIS WHOLE ISSUE IS NOT FIRST COME FIRST SERVE. IF YOU CAN GENUINELY HELP AND HAVE THE SKILLS TO CODE THIS THEN ONLY COMMENT AND DISCUSS.

I have created a separate discussion thread for this. Join here: https://github.com/avinashkranjan/Amazing-Python-Scripts/discussions/882

[Ajay-Singh-Rana]

Hey...Why not make things easy by moving scripts to specific task categories in the repo itself..?This would also make the repository much more accessible based on specificity.

[XZANATOL]

@Ajay-Singh-Rana many scripts can belong to many different categories, not only one.. so doing this will make it even harder to keep track of the scripts when we try to display them according to specificity 🤔

[kaustubhgupta]

Hey...Why not make things easy by moving scripts to specific task categories in the repo itself..?This would also make the repository much more accessible based on specificity.

We are not aiming to restructure the repo but create a master script to initiate any script of this repo

[XZANATOL]

Finally got free from my commitments.. xd @kaustubhgupta I would like to start with the base of the project of creating the JSON file. Gonna read the each project of the repo and collect the necessary information. I can also add the code snippet of the function which allows the developer to add his/her script to the repo as I will automate this part on my local machine. xd

[kaustubhgupta]

@XZANATOL Cool, you can start working on this sub-task.

[Ajay-Singh-Rana]

@kaustubhgupta @XZANATOL do you guys mean that the repository would remain as it is and the JSON has to be manually created at first and lateron we can give an option to add the script under a particular section by passing the arguments..? Is it..?

[XZANATOL]

@kaustubhgupta Hi there, I've finsihed most of the JSON file, here is a small part of how it is structured.

The JSON file follows this structure: {category: {name: [path, entry, arguments, requirments_path, contributor, description]}} Tell me what you think about it, and whether should I change anything about it.

[kaustubhgupta]

@kaustubhgupta Hi there, I've finsihed most of the JSON file, here is a small part of how it is structured.

The JSON file follows this structure: {category: {name: [path, entry, arguments, requirments_path, contributor, description]}} Tell me what you think about it, and whether should I change anything about it.

Perfect, 💯

I am working on the Pull request automation locally and maybe I will create the pull request for that. Let's see how this works and then we can move towards making the demo script!

[kaustubhgupta]

@kaustubhgupta @XZANATOL do you guys mean that the repository would remain as it is and the JSON has to be manually created at first and lateron we can give an option to add the script under a particular section by passing the arguments..? Is it..?

Yes @Ajay-Singh-Rana. If you want, you can work on the pull request automation.

[Ayushjain2205]

@kaustubhgupta I would like to work the 2nd task - Creating the demo Script for testing Just have a small question regarding how to deal with scripts that require arguments, should the user first be prompted what argument is required and then take the argument from the user as a command line argument?

[kaustubhgupta]

Hi @Ayushjain2205, Let @XZANATOL update the database and then I can explain you better about actual implementation

[XZANATOL]

@kaustubhgupta I've a doubt, There are some scripts that needs a requirments.txt file but it does not exist, however the pip commands present in the Readme.md file. Shall I make a seperate requirments.txt file and add the needed for each script (if necessary) or shall I give the path to the Readme.md file instead? tho doing the second option will prevent adding a step to prepare the script libraries using our menu-driven script. :thinking:

[devRawnie]

Hi. I created a menu-driven script for another Repo containing scripts related to penetration and Hacking. Would also like to work on this one.

[devRawnie]

I'll set-up a draft PR in some time, and will continue pushing changes into it

[kaustubhgupta]

@devRawnie don't make the draft PR now. Read this comment https://github.com/avinashkranjan/Amazing-Python-Scripts/issues/831#issuecomment-817241049 and let me know in which part you're interested.

[kaustubhgupta]

@kaustubhgupta I've a doubt, There are some scripts that needs a requirments.txt file but it does not exist, however the pip commands present in the Readme.md file. Shall I make a seperate requirments.txt file and add the needed for each script (if necessary) or shall I give the path to the Readme.md file instead? tho doing the second option will prevent adding a step to prepare the script libraries using our menu-driven script. 🤔

No, don't make requirements for those scripts. I create another issue for fixing these. Just include those requirements which are already pushed by contributors

[devRawnie]

Hi @kaustubhgupta. I would like to work on the second task. The code that I wrote earlier for the menu-driven program loosely followed the methodology as you mentioned here. So I would like to work on the demo script part where we are taking input from the user based on the arguments required. After that, I would also like to learn about github actions and automate JSON updation

[kaustubhgupta]

Hi @kaustubhgupta. I would like to work on the second task. The code that I wrote earlier for the menu-driven program loosely followed the methodology as you mentioned here. So I would like to work on the demo script part where we are taking input from the user based on the arguments required. After that, I would also like to learn about github actions and automate JSON updation

Okay, for now wait for Json file to be updated.

[kaustubhgupta]

@XZANATOL updates on this?

[XZANATOL]

Already finished 3 categories, about 4 left. By the way where should I place the JSON file? In the root directory of the repo or shall I make a lazy script folder?

[kaustubhgupta]

Already finished 3 categories, about 4 left. By the way where should I place the JSON file? In the root directory of the repo or shall I make a lazy script folder?

Umm, yes create a folder for this, "Master Script"

[XZANATOL]

@kaustubhgupta Hi there, sorry for that delay, There is a hidden mess behind each folder in the repo it caused me a headache trying to filter each. xd

The Databse is ready and I'm ready to make a PR (made a git pull from a couple of minutes to make sure that everything is up-to-date), but one last doubt before submitting everything, there are 6 projects which idk how to implement. (gonna mention them using their folder names)

1. AWS Management Scripts This one has multiple scripts each with it's own job. Any ideas how it should be implemented? I can fill up the needed and at the python-path I will just put a "multiple" value as it indicates that there are multiple scripts there.
2. Spelling-Checker, Data-Visualization Same thing here.
3. Restoring-Divider IDK what this one is here for.. I even read the code but couldn't really understand its purpose. xd There is no Readme.md file.
4. Convert2jpg, Internet-Speed-Test Finally, these 2.. they are just a 2 Readme.md files which tells how to install an external library and use them. (No scripts exist there)

Note for future contributions

There are .ipybn beside .py files in the DB, so consider checking for the extension first to decide whether the Master-Script will run the project using Python shell or Notebook.

What's now Ready

1. JSON DB file. Fun fact: the file is 50K characters long. xD
2. A python script that has the code to add & read Projects in DB, & give a report for any unadded projects. I can execlude that from submission, but it will help alot with the upcoming contributions.