cli
cli copied to clipboard
Published
20 hours ago •
snyk
snyk
[🐛] Can't check python project requirements.txt - unknown error
$ snyk -v 1.715.0 (standalone) $ npm -v 6.14.15 $ node -v v12.22.6
Both Win10 and Ubuntu18.04 Both Python 3.6 and 3.8
Expected behaviour
The python project is validated successfully.
Actual behaviour
snyk test --file=./requirements.txt -d
The requirements.txt is empty.
Ubuntu output:
snyk test <ref *1> {
_: [ [Circular *1] ],
file: './requirements.txt',
debug: true
} +0ms
snyk:run-test Error running test {
error: 'Traceback (most recent call last):\n' +
' File "/venv3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2698, in _dep_map\n' +
' return self.__dep_map\n' +
' File "/venv3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2813, in __getattr__\n' +
' raise AttributeError(attr)\n' +
'AttributeError: _Distribution__dep_map\n' +
'\n' +
'During handling of the above exception, another exception occurred:\n' +
'\n' +
'Traceback (most recent call last):\n' +
' File "/tmp/tmp-25066-BujQPn9a0VBT/pip_resolve.py", line 329, in <module>\n' +
' sys.exit(main())\n' +
' File "/tmp/tmp-25066-BujQPn9a0VBT/pip_resolve.py", line 321, in main\n' +
' create_dependencies_tree_by_req_file_path(\n' +
' File "/tmp/tmp-25066-BujQPn9a0VBT/pip_resolve.py", line 264, in create_dependencies_tree_by_req_file_path\n' +
' dist_tree = utils.construct_tree(dist_index)\n' +
' File "/tmp/tmp-25066-BujQPn9a0VBT/utils.py", line 30, in construct_tree\n' +
' return dict((p, [ReqPackage(r, index.get(r.key))\n' +
' File "/tmp/tmp-25066-BujQPn9a0VBT/utils.py", line 31, in <genexpr>\n' +
' for r in p.requires()])\n' +
' File "/venv3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2734, in requires\n' +
' dm = self._dep_map\n' +
' File "/venv3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2700, in _dep_map\n' +
' self.__dep_map = self._filter_extras(self._build_dep_map())\n' +
' File "/venv3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2715, in _filter_extras\n' +
' invalid_marker(marker)\n' +
' File "/venv3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1347, in invalid_marker\n' +
' evaluate_marker(text)\n' +
' File "/venv3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1365, in evaluate_marker\n' +
' return marker.evaluate()\n' +
' File "/venv3.8/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/markers.py", line 328, in evaluate\n' +
' return _evaluate_markers(self._markers, current_environment)\n' +
' File "/venv3.8/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/markers.py", line 244, in _evaluate_markers\n' +
' lhs_value = _get_env(environment, lhs.value)\n' +
' File "/venv3.8/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/markers.py", line 224, in _get_env\n' +
' raise UndefinedEnvironmentName(\n' +
"pkg_resources.extern.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.\n"
} +0ms
snyk-test Failed to test 1 projects, errors: +0ms
snyk-test error: FailedToRunTestError: Failed to test pip project
at runTest (/snapshot/snyk/dist/cli/webpack:/snyk/src/lib/snyk-test/run-test.ts:368:11)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at test (/snapshot/snyk/dist/cli/webpack:/snyk/src/cli/commands/test/index.ts:93:15)
at runCommand (/snapshot/snyk/dist/cli/webpack:/snyk/src/cli/index.ts:57:25)
at main (/snapshot/snyk/dist/cli/webpack:/snyk/src/cli/index.ts:317:11) +451ms
Error:
Testing ...
Failed to test pip project
at test (/snapshot/snyk/dist/cli/webpack:/snyk/src/cli/commands/test/index.ts:241:19)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at runCommand (/snapshot/snyk/dist/cli/webpack:/snyk/src/cli/index.ts:57:25)
at main (/snapshot/snyk/dist/cli/webpack:/snyk/src/cli/index.ts:317:11)
snyk Exit code: 2 +0ms
snyk analytics {
"args": [
{
"file": "./requirements.txt",
"debug": true
}
],
"command": "bad-command",
"metadata": {
"local": true,
"error-message": "\nTesting ...\n\nFailed to test pip project",
"error": "Error: \nTesting ...\n\nFailed to test pip project\n at test (/snapshot/snyk/dist/cli/webpack:/snyk/src/cli/commands/test/index.ts:241:19)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at runCommand (/snapshot/snyk/dist/cli/webpack:/snyk/src/cli/index.ts:57:25)\n at main (/snapshot/snyk/dist/cli/webpack:/snyk/src/cli/index.ts:317:11)",
"error-code": 500,
"command": "test"
},
"os": "Linux 4.15",
"version": "1.715.0",
"nodeVersion": "v14.17.6",
"standalone": true,
"integrationName": "",
"integrationVersion": "",
"integrationEnvironment": "",
"integrationEnvironmentVersion": "",
"id": "a8e361558d56a039ae574088b4ad37a3d4c5e6ff",
"ci": false,
"environment": {},
"durationMs": 1443,
"metrics": {
"network_time": {
"type": "timer",
"values": [],
"total": 0
},
"cpu_time": {
"type": "synthetic",
"values": [
1443
],
"total": 1443
}
}
} +0ms
Windows output:
snyk test <ref *1> { _: [ [Circular *1] ], file: 'requirements.txt', debug: true } +0ms
snyk:run-test Error running test {
error: "\\Python\\Python38\\lib\\site-packages\\setuptools\\distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.\r\n" +
' warnings.warn(\r\n' +
'Traceback (most recent call last):\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\_vendor\\packaging\\requirements.py", line 93, in __init__\r\n' +
' req = REQUIREMENT.parseString(requirement_string)\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\_vendor\\pyparsing.py", line 1654, in parseString\r\n' +
' raise exc\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\_vendor\\pyparsing.py", line 1644, in parseString\r\n' +
' loc, tokens = self._parse( instring, 0 )\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\_vendor\\pyparsing.py", line 1402, in _parseNoCache\r\n' +
' loc,tokens = self.parseImpl( instring, preloc, doActions )\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\_vendor\\pyparsing.py", line 3417, in parseImpl\r\n' +
' loc, exprtokens = e._parse( instring, loc, doActions )\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\_vendor\\pyparsing.py", line 1402, in _parseNoCache\r\n' +
' loc,tokens = self.parseImpl( instring, preloc, doActions )\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\_vendor\\pyparsing.py", line 3739, in parseImpl\r\n' +
' return self.expr._parse( instring, loc, doActions, callPreParse=False )\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\_vendor\\pyparsing.py", line 1402, in _parseNoCache\r\n' +
' loc,tokens = self.parseImpl( instring, preloc, doActions )\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\_vendor\\pyparsing.py", line 3400, in parseImpl\r\n' +
' loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\_vendor\\pyparsing.py", line 1406, in _parseNoCache\r\n' +
' loc,tokens = self.parseImpl( instring, preloc, doActions )\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\_vendor\\pyparsing.py", line 2711, in parseImpl\r\n' +
' raise ParseException(instring, loc, self.errmsg, self)\r\n' +
'pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)\r\n' +
'\r\n' +
'During handling of the above exception, another exception occurred:\r\n' +
'\r\n' +
'Traceback (most recent call last):\r\n' +
' File "\\Temp\\tmp-35948-1SzXc6soDWtI\\pip_resolve.py", line 329, in <module>\r\n' +
' sys.exit(main())\r\n' +
' File "\\Temp\\tmp-35948-1SzXc6soDWtI\\pip_resolve.py", line 321, in main\r\n' +
' create_dependencies_tree_by_req_file_path(\r\n' +
' File "\\Temp\\tmp-35948-1SzXc6soDWtI\\pip_resolve.py", line 267, in create_dependencies_tree_by_req_file_path\r\n' +
' required = get_requirements_list(requirements_file_path, dev_deps=dev_deps)\r\n' +
' File "\\Temp\\tmp-35948-1SzXc6soDWtI\\pip_resolve.py", line 235, in get_requirements_list\r\n' +
' req_list = list(requirements.parse(f))\r\n' +
' File "\\Temp\\tmp-35948-1SzXc6soDWtI\\requirements\\parser.py", line 78, in parse\r\n' +
' req = Requirement.parse(line)\r\n' +
' File "\\Temp\\tmp-35948-1SzXc6soDWtI\\requirements\\requirement.py", line 240, in parse\r\n' +
' return cls.parse_line(line)\r\n' +
' File "\\Temp\\tmp-35948-1SzXc6soDWtI\\requirements\\requirement.py", line 218, in parse_line\r\n' +
' pkg_req = Req.parse(line)\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\__init__.py", line 3153, in parse\r\n' +
' req, = parse_requirements(s)\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\__init__.py", line 3098, in parse_requirements\r\n' +
' yield Requirement(line)\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\__init__.py", line 3108, in __init__\r\n' +
' super(Requirement, self).__init__(requirement_string)\r\n' +
' File "\\Python\\Python38\\lib\\site-packages\\pkg_resources\\_vendor\\packaging\\requirements.py", line 95, in __init__\r\n' +
' raise InvalidRequirement(\r\n' +
`pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at "'��'": Expected W:(abcd...)\r\n`
} +0ms
snyk-test Failed to test 1 projects, errors: +0ms
snyk-test error: FailedToRunTestError: Failed to test pip project
at runTest (C:\snapshot\snyk\dist\cli\webpack:\snyk\src\lib\snyk-test\run-test.ts:368:11)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at test (C:\snapshot\snyk\dist\cli\webpack:\snyk\src\cli\commands\test\index.ts:93:15)
at runCommand (C:\snapshot\snyk\dist\cli\webpack:\snyk\src\cli\index.ts:57:25)
at main (C:\snapshot\snyk\dist\cli\webpack:\snyk\src\cli\index.ts:317:11) +408ms
Error:
Testing C:\work\datumaro\test_snyk...
Failed to test pip project
at test (C:\snapshot\snyk\dist\cli\webpack:\snyk\src\cli\commands\test\index.ts:241:19)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at runCommand (C:\snapshot\snyk\dist\cli\webpack:\snyk\src\cli\index.ts:57:25)
at main (C:\snapshot\snyk\dist\cli\webpack:\snyk\src\cli\index.ts:317:11)
snyk Exit code: 2 +0ms
snyk analytics {
"args": [
{
"file": "requirements.txt",
"debug": true
}
],
"command": "bad-command",
"metadata": {
"local": true,
"error-message": "\nTesting test_snyk...\n\nFailed to test pip project",
"error": "Error: \nTesting test_snyk...\n\nFailed to test pip project\n at test (C:\\snapshot\\snyk\\dist\\cli\\webpack:\\snyk\\src\\cli\\commands\\test\\index.ts:241:19)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at runCommand (C:\\snapshot\\snyk\\dist\\cli\\webpack:\\snyk\\src\\cli\\index.ts:57:25)\n at main (C:\\snapshot\\snyk\\dist\\cli\\webpack:\\snyk\\src\\cli\\index.ts:317:11)",
"error-code": 500,
"command": "test"
},
"os": "Windows 10",
"version": "1.715.0",
"nodeVersion": "v14.17.6",
"standalone": true,
"integrationName": "",
"integrationVersion": "",
"integrationEnvironment": "",
"integrationEnvironmentVersion": "",
"id": "1bfbcdbeb9d9610589af67c4e03aaaea605e2cdf",
"ci": false,
"environment": {},
"durationMs": 1613,
"metrics": {
"network_time": {
"type": "timer",
"values": [],
"total": 0
},
"cpu_time": {
"type": "synthetic",
"values": [
1613
],
"total": 1613
}
}
} +0ms
Steps to reproduce
mkdir -p test_snyk
cd test_snyk/
echo "" > requirements.txt
snyk test --file=./requirements.txt -d
Seeing there is an 'extra' does not exist in evaluation environment in the stack trace.
Might be related to this issue: https://github.com/getsentry/sentry/issues/3143 and a setuptools version?
Updating the system version of setuptools and recreating the virtual env seem to have helped on linux. Updating just the venv one didn't.
Moving further, is it seems that comments on lines with links to other requirements files is not supported? I'm getting
snyk:run-test Error running test {
error: 'Traceback (most recent call last):\n' +
' File "/tmp/tmp-23464-kb8e9Zy4R2yU/pip_resolve.py", line 329, in <module>\n' +
' sys.exit(main())\n' +
' File "/tmp/tmp-23464-kb8e9Zy4R2yU/pip_resolve.py", line 325, in main\n' +
' only_provenance=args.only_provenance,\n' +
' File "/tmp/tmp-23464-kb8e9Zy4R2yU/pip_resolve.py", line 267, in create_dependencies_tree_by_req_file_path\n' +
' required = get_requirements_list(requirements_file_path, dev_deps=dev_deps)\n' +
' File "/tmp/tmp-23464-kb8e9Zy4R2yU/pip_resolve.py", line 235, in get_requirements_list\n' +
' req_list = list(requirements.parse(f))\n' +
' File "/tmp/tmp-23464-kb8e9Zy4R2yU/requirements/parser.py", line 58, in parse\n' +
' _, new_filename = line.split()\n' +
'ValueError: too many values to unpack (expected 2)\n'
When the requirements.txt has a line like "-r requirements-xxx.txt # comment". With no such lines the checks pass.
Moving further, is it seems that comments on lines with links to other requirements files is not supported? I'm getting
snyk:run-test Error running test { error: 'Traceback (most recent call last):\n' + ' File "/tmp/tmp-23464-kb8e9Zy4R2yU/pip_resolve.py", line 329, in <module>\n' + ' sys.exit(main())\n' + ' File "/tmp/tmp-23464-kb8e9Zy4R2yU/pip_resolve.py", line 325, in main\n' + ' only_provenance=args.only_provenance,\n' + ' File "/tmp/tmp-23464-kb8e9Zy4R2yU/pip_resolve.py", line 267, in create_dependencies_tree_by_req_file_path\n' + ' required = get_requirements_list(requirements_file_path, dev_deps=dev_deps)\n' + ' File "/tmp/tmp-23464-kb8e9Zy4R2yU/pip_resolve.py", line 235, in get_requirements_list\n' + ' req_list = list(requirements.parse(f))\n' + ' File "/tmp/tmp-23464-kb8e9Zy4R2yU/requirements/parser.py", line 58, in parse\n' + ' _, new_filename = line.split()\n' + 'ValueError: too many values to unpack (expected 2)\n'When the requirements.txt has a line like "-r requirements-xxx.txt # comment". With no such lines the checks pass.
Hey @zhiltsov-max , were you able to solve the issue?
