aws-toolkit-jetbrains Some pip packages fail to access their own dist-info in Debug but not Run

Describe the bug Some packages use pkg_resource.get_distribution() at runtime (e.g., to find their own version) and for some packages this seems to fail in Debug mode, while it succeeds in Run mode (e.g., jsonschema==3.0.1).

To reproduce

Create new AWS Serverless Application project from hello-world template
Add jsonschema==3.0.1 to requirements.txt
In launch configuration settings, switch SAM CLI to "Build function inside container"
import jsonschema in app.py
Click Run and observe that it succeeds
Click Debug and observe that it fails with the error:

"C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd" local invoke --template C:\Users\chris\AppData\Local\Temp\lambdaBuild2884\template.yaml --event "C:\Users\chris\AppData\Local\Temp\[Local] HelloWorldFunction-event3586.json" HelloWorldFunction --debug-port 56537 --debugger-path "C:\Program Files\JetBrains\PyCharm 2019.1.1\helpers\pydev" --debug-args "-u /tmp/lambci_debug_files/pydevd.py --multiprocess --port 56537 --file"
2019-06-26 16:10:17 Invoking app.lambda_handler (python3.7)
2019-06-26 16:10:17 Found credentials in environment variables.

Fetching lambci/lambda:python3.7 Docker container image......
2019-06-26 16:10:19 Mounting C:\Users\chris\AppData\Local\Temp\lambdaBuild2884\HelloWorldFunction as /var/task:ro,delegated inside runtime container
Connected to pydev debugger (build 191.7479.30)
Â Â Â Â raise DistributionNotFound(req, requirers)
START RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72 Version: $LATEST
END RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72
REPORT RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72	Init Duration: 1309.12 ms	Duration: 0.00 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 34 MB	
{
  "errorType": "DistributionNotFound",
  "errorMessage": "The 'jsonschema' distribution was not found and is required by the application",
  "stackTrace": [
    "  File \"/var/lang/lib/python3.7/imp.py\", line 234, in load_module\n    return load_source(name, filename, file)\n",
    "  File \"/var/lang/lib/python3.7/imp.py\", line 171, in load_source\n    module = _load(spec)\n",
    "  File \"\u003cfrozen importlib._bootstrap\u003e\", line 696, in _load\n",
    "  File \"\u003cfrozen importlib._bootstrap\u003e\", line 677, in _load_unlocked\n",
    "  File \"\u003cfrozen importlib._bootstrap_external\u003e\", line 728, in exec_module\n",
    "  File \"\u003cfrozen importlib._bootstrap\u003e\", line 219, in _call_with_frames_removed\n",
    "  File \"/var/task/app.py\", line 3, in \u003cmodule\u003e\n    import jsonschema\n",
    "  File \"/var/task/jsonschema/__init__.py\", line 33, in \u003cmodule\u003e\n    __version__ = get_distribution(__name__).version\n",
    "  File \"/var/lang/lib/python3.7/site-packages/pkg_resources/__init__.py\", line 481, in get_distribution\n    dist = get_provider(dist)\n",
    "  File \"/var/lang/lib/python3.7/site-packages/pkg_resources/__init__.py\", line 357, in get_provider\n    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]\n",
    "  File \"/var/lang/lib/python3.7/site-packages/pkg_resources/__init__.py\", line 900, in require\n    needed = self.resolve(parse_requirements(requirements))\n",
    "  File \"/var/lang/lib/python3.7/site-packages/pkg_resources/__init__.py\", line 786, in resolve\n    raise DistributionNotFound(req, requirers)\n"
  ]
}

Expected behavior Importing the package succeeds like it does in Run mode.

Your Environment

OS: Windows 10 1803 x64
JetBrains' Product: PyCharm Professional Edition
JetBrains' Product Version: 2019.1.3 (Build #PY-191.7479.30, built on May 29, 2019)
Toolkit Version: v1.4
SAM CLI Version: 0.17.0
JVM/Python Version: Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] on win32
Docker Desktop Version: Community Edition, Version 2.0.0.3 (31259)

Additional context Not using the SAM CLI option "Build function inside container" causes the build to fail with a pip PythonPipBuilder:ResolveDependencies error, which may or may not be related:

"C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd" build --template C:\DEVEL\SAM\test4\template.yaml --build-dir C:\Users\chris\AppData\Local\Temp\lambdaBuild49
2019-06-26 16:20:32 Building resource 'HelloWorldFunction'
2019-06-26 16:20:32 Running PythonPipBuilder:ResolveDependencies
Error: PythonPipBuilder:ResolveDependencies - list index out of range

Jun 26 '19 23:06 chkothe

Was able to replicate

Jul 03 '19 17:07 abrooksv

Outside of SAM command:

/Users/alfredbr/IdeaProjects/SamMaven/venv/bin/python /Users/alfredbr/aws-intellij-toolkit/jetbrains-core/build/idea-sandbox/plugins/python-ce/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 50466 --file /Users/alfredbr/IdeaProjects/PythonSam/hello_world/app.py

Inside of SAM command:

/var/lang/bin/python3.7 -u /tmp/lambci_debug_files/pydevd.py --DEBUG --multiprocess --port. 51971 --file /var/runtime/bootstrap

Jul 03 '19 17:07 abrooksv

Using the lambda image as a remote python interpreter results in:

python -u /opt/.pycharm_helpers/pydev/pydevd.py --multiprocess --qt-support=auto --port 53737 --file /opt/project/hello_world/app.py

Which maps with what we do in SAM and it does work.

Jul 03 '19 19:07 abrooksv

So I have a work around, but I do not know why pydevd is requiring it.

Add

      Environment:
        Variables:
          PYTHONPATH: '/var/task'

to your template.yaml.

I don't see a way I can fix this in the toolkit, since the toolkit can only add environment variables if already defined in your template.

Jul 03 '19 19:07 abrooksv

Ok good to know!

Jul 03 '19 20:07 chkothe

Okay, I see. Pydev adds the directory of argument to --file into sys path using sys.path(0, ...), but that is /var/runtime, not /var/task. (https://github.com/fabioz/PyDev.Debugger/blob/fff160b1327e248eb847f5adc70113f1fb60b8e5/pydevd.py#L1770)

Jul 03 '19 20:07 abrooksv

I tried to put pydevd into module module, but that just led to failing to start.

('Bound to port ', '61540')
('Connection accepted',)
debugger: received >>501	1	0.1	UNIX
<<
Received command: CMD_VERSION 501	1	0.1	UNIX

sending cmd -->          CMD_VERSION 501	1	@@BUILD_NUMBER@@ 
Connected to pydev debugger (build 191.6183.87)
debugger: received >>501	3	0.1	UNIX
<<
Received command: CMD_VERSION 501	3	0.1	UNIX

sending cmd -->          CMD_VERSION 501	3	@@BUILD_NUMBER@@ 
debugger: received >>111	5	python-line	/var/task/app.py	38	lambda_handler	THREAD	None	None
<<
Received command: CMD_SET_BREAK 111	5	python-line	/var/task/app.py	38	lambda_handler	THREAD	None	None

Added breakpoint:/var/task/app.py - line:38 - func_name:b'lambda_handler'

debugger: received >>122	7	python-BaseException	None	None	0	1	0
146	9	SHOW_RETURN_VALUES	0
101	11	
<<
Received command: CMD_ADD_EXCEPTION_BREAK 122	7	python-BaseException	None	None	0	1	0

Exceptions to hook on terminate: {'BaseException': <_pydevd_bundle.pydevd_breakpoints.ExceptionBreakpoint object at 0x7f4001e166d8>}

Received command: CMD_SHOW_RETURN_VALUES 146	9	SHOW_RETURN_VALUES	0

Show return values: False

Received command: CMD_RUN 101	11	

No module named /var/runtime/bootstrap

Jul 03 '19 20:07 abrooksv