bug: pass through install logs from venv_backend

[visch]

Meltano Version

Latest

Python Version

3.13

Bug scope

CLI (options, error messages, logging, etc.)

Operating System

No response

Description

At some point we swapped from passing through venv logs to meltano.

The outcome I'd like:

1. Pass through stdout/stderr from the backend_venv process to meltano

Even with debugging turned on we get the below:

Code

2024-11-26T14:34:19.469162Z [debug    ] Installing with args "'tap-universal-file[s3]@git+https://github.com/MeltanoLabs/[email protected]'" into virtual environment for 'extractors/tap-universal-file'
2024-11-26T14:34:19.523007Z [info     ] Installing extractor 'tap-postgres'
2024-11-26T14:34:19.524406Z [debug    ] Using uv executable at /home/visch/.cargo/bin/uv
2024-11-26T14:34:19.525001Z [debug    ] Installing with args 'meltanolabs-tap-postgres==0.0.15' into virtual environment for 'extractors/tap-postgres'
2024-11-26T14:34:19.537153Z [info     ] Installing extractor 'tap-hubspot'
2024-11-26T14:34:19.538413Z [debug    ] Using uv executable at /home/visch/.cargo/bin/uv
2024-11-26T14:34:19.538705Z [debug    ] Installing with args 'git+ssh://[email protected]/aitbi/tap-hubspot.git' into virtual environment for 'extractors/tap-hubspot'
2024-11-26T14:34:19.546167Z [info     ] Installing loader 'target-hubspot'
2024-11-26T14:34:19.547058Z [debug    ] Using uv executable at /home/visch/.cargo/bin/uv
2024-11-26T14:34:19.547355Z [debug    ] Installing with args 'git+ssh://[email protected]/aitbi/target-hubspot.git' into virtual environment for 'loaders/target-hubspot'
2024-11-26T14:34:19.555176Z [debug    ] Variable '$MELTANO_EXTRACT__LOAD_SCHEMA' is not set in the provided env dictionary.
2024-11-26T14:34:19.560537Z [debug    ] Variable '$MELTANO_EXTRACT__LOAD_SCHEMA' is not set in the provided env dictionary.
2024-11-26T14:34:19.566472Z [debug    ] Variable '$MELTANO_EXTRACT__LOAD_SCHEMA' is not set in the provided env dictionary.
2024-11-26T14:34:19.572453Z [debug    ] Variable '$TARGET_POSTGRES_SCHEMA' is not set in the provided env dictionary.
2024-11-26T14:34:19.572896Z [info     ] Installing loader 'target-postgres'
2024-11-26T14:34:19.573695Z [debug    ] Using uv executable at /home/visch/.cargo/bin/uv
2024-11-26T14:34:19.573999Z [debug    ] Installing with args 'meltanolabs-target-postgres==0.0.16' into virtual environment for 'loaders/target-postgres'
2024-11-26T14:34:19.584872Z [info     ] Installing loader 'target-apprise'
2024-11-26T14:34:19.586234Z [debug    ] Using uv executable at /home/visch/.cargo/bin/uv
2024-11-26T14:34:19.586567Z [debug    ] Installing with args 'target-apprise' into virtual environment for 'loaders/target-apprise'
2024-11-26T14:34:19.592565Z [info     ] Installing loader 'target-jsonl'
2024-11-26T14:34:19.593433Z [debug    ] Using uv executable at /home/visch/.cargo/bin/uv
2024-11-26T14:34:19.593725Z [debug    ] Installing with args 'target-jsonl' into virtual environment for 'loaders/target-jsonl'
2024-11-26T14:34:19.602482Z [info     ] Installing transformer 'dbt-postgres'
2024-11-26T14:34:19.603584Z [debug    ] Using uv executable at /home/visch/.cargo/bin/uv
2024-11-26T14:34:19.604027Z [debug    ] Installing with args "'dbt-core~=1.8.0' 'dbt-postgres~=1.8.0'" into virtual environment for 'transformers/dbt-postgres'
2024-11-26T14:34:19.605702Z [info     ] Installed extractor 'tap-postgres'
2024-11-26T14:34:19.606119Z [info     ] Installed loader 'target-postgres'
2024-11-26T14:34:19.606428Z [info     ] Installed loader 'target-apprise'
2024-11-26T14:34:19.606690Z [info     ] Installed loader 'target-jsonl'
2024-11-26T14:34:19.612508Z [info     ] Installed transformer 'dbt-postgres'
2024-11-26T14:34:20.372468Z [info     ] Installed extractor 'tap-hubspot'
2024-11-26T14:34:20.379812Z [info     ] Installed loader 'target-hubspot'
2024-11-26T14:34:21.967895Z [info     ] Installed extractor 'tap-universal-file'
2024-11-26T14:34:21.968126Z [info     ] Installed 8/11 plugins
2024-11-26T14:34:21.968197Z [info     ] Skipped installing 3/11 plugins

Link to Slack/Linen

No response

[visch]

Use case: This has been hung for 25 minutes and I don't know why. I'll figure it out but if I had the logs life would be easier

[visch]

Found the root cause here of my issue

Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps or to
join our friendly Slack community.

Failed to parse JSON array from string: '[BadJsonArray?name="Data"]'

This happened after every plugin was installed, but it'd hang the job, and the stderr output listed above never got sent instead we just hung, not exactly sure why in regards to stderr or some kind of python stdserr buffering. Might be something github actions-related.

[edgarrmondragon]

https://github.com/meltano/meltano/pull/9565 is an improvement here