docs: Debugging Targets PDB Documentation

[visch]

Feature scope

Targets (data type handling, batching, SQL object generation, etc.)

Description

Not sure if we want to make changes to the SDK but I wanted to document this somewhere publicly.

How to debug a Target written with the SDK.

• Why

Debugging is a bit tougher with targets for two reasons

1. Targets use STDIN and PDB does the same
2. Targets use joblib.Parallel to Parallelize each sink see https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html

• How

1. Add the remote-pdb dependency, poetry add remote-pdb
2. Override max_parallelism in your target.py to only use 1 parallel job (ie no parallelization is used at all). ie

@property
def max_parallelism(self) -> int:
    return 1                  

3. export PYTHONBREAKPOINT=remote_pdb.set_trace , export REMOTE_PDB_QUIET=1 so no output from remote-pdb gets sent to the target export REMOTE_PDB_HOST=127.0.0.1 , export REMOTE_PDB_PORT=4444 ,
4. Run the target like normal, and in a seperate terminal run telnet 127.0.0.1 4444
5. Note that you can use breakpoint() to set a breakpoint
6. Remember that if you're using Meltano you'll have to reinstall (even if you're using -e) as this is a new dependency

These commands are tweakable read through the documentation here https://pypi.org/project/remote-pdb/

[stale[bot]]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[stale[bot]]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.