Add Getting Started guide to README, with a walkthrough of running it on a Django repo

[lfatty]

I discovered many SyntaxError during setup.py installation.

build/bdist.macosx-10.12-x86_64/egg/pyt/__main__.py", line 323
    if args.print:
                ^
SyntaxError: invalid syntax

File "build/bdist.macosx-10.12-x86_64/egg/pyt/analysis_base.py", line 9
    class AnalysisBase(metaclass=ABCMeta):
                                ^
SyntaxError: invalid syntax

File "build/bdist.macosx-10.12-x86_64/egg/pyt/fixed_point.py", line 32
    def analyse(cfg_list, *, analysis_type):
                           ^
SyntaxError: invalid syntax

File "build/bdist.macosx-10.12-x86_64/egg/pyt/github_search.py", line 124
    class Search(metaclass=ABCMeta):
                          ^
SyntaxError: invalid syntax

yield from get_vulnerability_chains(
             ^
SyntaxError: invalid syntax

[KevinHock]

Ahh so this is due to you running in under Python 2, under Python 3, (especially 3.6), everything will run well :)

[cclauss]

https://travis-ci.org/python-security/pyt/builds/366758979#L509

[lfatty]

Thanks @KevinHock. What is the best way to test a whole repo or a directory that holds all the python files.

[cclauss]

python3 -m tests

[KevinHock]

Thanks for the PR @cclauss, I will check it out tomorrow night :)

@lfatty which web framework is the repo? You can use find to list all the files and then run PyT on them with -f but the framework matters a little bit.

[lfatty]

Lets say it is django

[lfatty]

I also when running a test against django.nV, I got the following:

python -m pyt -f examples/django.nV/taskManager/views.py 
/bla/bla/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py:125: RuntimeWarning: 'pyt.__main__' found in sys.modules after import of package 'pyt', but prior to execution of 'pyt.__main__'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
No vulnerabilities found.

Is this expected or normal?

[KevinHock]

Try -a E or -a D, it's the framework adaptor option.

[lfatty]

Ok, i am not being successful. Can you please write down the commands for that? thx

[lfatty]

Thanks @KevinHock, I think this tool is very promising especially if we can make it very easy to use. Also we need think about a new option such as -r, --recursive find and process files in subdirectories. Just a suggestion.

[KevinHock]

Thank for the kind words @lfatty, I'll definitely work on that too, I've been overly focused on fixing the false positive and negative problems from our last evaluation, I'll also work on de duplicating any duplicate vulnerabilities as a post-processing step until I fix the root cause.

[KevinHock]

You can try this python -m pyt -a E -f examples/django.nV/taskManager/views.py --interactive before I implement the -r and de-duping options.

[KevinHock]

python -m pyt -a E -f examples/django.nV/taskManager/upload_controller.py --interactive has slightly less worse findings, you can edit the sink information by editing pyt/vulnerability_definitions/all_trigger_words.pyt

[lfatty]

@KevinHock, I ran python -m pyt -a E -f examples/django.nV/taskManager/upload_controller.py --interactive and all looks fine. But, when I ran the command against my code, i got the following output:

Traceback (most recent call last):
  File "/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "projects/pythonsca/pyt/pyt/__main__.py", line 371, in <module>
    main()
  File "projects/pythonsca/pyt/pyt/__main__.py", line 295, in main
    FrameworkAdaptor(cfg_list, project_modules, local_modules, framework_route_criteria)
  File "/projects/pythonsca/pyt/pyt/framework_adaptor.py", line 21, in __init__
    self.run()
  File "/projects/pythonsca/pyt/pyt/framework_adaptor.py", line 70, in run
    function_cfgs.extend(self.find_route_functions_taint_args())
  File "/projects/pythonsca/pyt/pyt/framework_adaptor.py", line 64, in find_route_functions_taint_args
    yield self.get_func_cfg_with_tainted_args(definition)
  File "/projects/pythonsca/pyt/pyt/framework_adaptor.py", line 30, in get_func_cfg_with_tainted_args
    definition.module_definitions
  File "/projects/pythonsca/pyt/pyt/expr_visitor.py", line 580, in make_cfg
    module_definitions
  File "/projects/pythonsca/pyt/pyt/expr_visitor.py", line 59, in __init__
    self.init_function_cfg(node, module_definitions)
  File "/projects/pythonsca/pyt/pyt/expr_visitor.py", line 96, in init_function_cfg
    module_statements = self.stmt_star_handler(node.body)
  File "/projects/pythonsca/pyt/pyt/stmt_visitor.py", line 75, in stmt_star_handler
    node = self.visit(stmt)
  File "/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ast.py", line 253, in visit
    return visitor(node)
  File "/projects/pythonsca/pyt/pyt/stmt_visitor.py", line 303, in visit_Try
    handler_body = self.stmt_star_handler(handler.body)
  File "/projects/pythonsca/pyt/pyt/stmt_visitor.py", line 75, in stmt_star_handler
    node = self.visit(stmt)
  File "/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ast.py", line 253, in visit
    return visitor(node)
  File "/projects/pythonsca/pyt/pyt/stmt_visitor.py", line 233, in visit_Raise
    path=self.filenames[-1]
TypeError: __init__() missing 1 required keyword-only argument: 'line_number'

[KevinHock]

Thanks for reporting that, I missed it in a recent refactor, I'm sorry about that, stmt_visitor has 90% coverage but visit_Raise is somehow missing, I'll add test coverage to that as well.

[lfatty]

np. i should be noted that something like def my_data(self) by way of example. I think we need to revisit selfas user input because it may generate many false positives.

[KevinHock]

That is a very simple fix, I'll do that after work today, I thought we already had that. Also FYI -a D is the adaptor for Django, it'll only mark functions with the first arg as request tainted https://github.com/python-security/pyt/blob/master/pyt/framework_helper.py#L24 I know that isn't fool-proof though.

[lfatty]

Great!

[KevinHock]

Sorry I haven't made/merged the 'don't mark self tainted' PR (finished handling IfExp's and partially done with BoolOp last night on a non-pushed branch) if you're curious it should be just checking the first arg here https://github.com/python-security/pyt/blob/master/pyt/framework_adaptor.py#L44

[KevinHock]

Fixed it ;) https://github.com/python-security/pyt/pull/119 Thanks 👍

[lfatty]

Thanks @KevinHock. I will test it and will keep you posted.