feature: Catalog extra still puts a properties.json file in .meltano

[visch]

Feature scope

CLI (options, error messages, logging, etc.)

Description

Ultimately, my goal is to be able to run multiple processes with the same tap in parallel. Right now, we hit Cannot start plugin tap-mssql: [WinError 32] The process cannot access the file because it is being used by another process: '.meltano\\run\\tap-mssql\\tap.properties.json' when we provide a Catalog extra to a tap which is odd as I'd expect Meltano to point to my provided catalog and not remove/add files.

Logs of current behavior

2024-09-12T12:47:15.563708Z [debug    ] Skipped installing extractor 'tap-smoke-test'
2024-09-12T12:47:15.563933Z [debug    ] Skipped installing 1/1 plugins
2024-09-12T12:47:15.713970Z [debug    ] Created configuration at /home/visch/git/meltano-projects/getting_started/.meltano/run/tap-smoke-test/tap.ceb61f67-c362-4a72-86b7-787eef00aa2a.config.json
2024-09-12T12:47:15.714332Z [debug    ] Could not find tap.properties.json in /home/visch/git/meltano-projects/getting_started/.meltano/extractors/tap-smoke-test/tap.properties.json, skipping.
2024-09-12T12:47:15.714573Z [debug    ] Could not find tap.properties.cache_key in /home/visch/git/meltano-projects/getting_started/.meltano/extractors/tap-smoke-test/tap.properties.cache_key, skipping.
2024-09-12T12:47:15.714807Z [debug    ] Could not find state.json in /home/visch/git/meltano-projects/getting_started/.meltano/extractors/tap-smoke-test/state.json, skipping.
2024-09-12T12:47:15.715151Z [debug    ] Cached catalog is outdated, running discovery...
2024-09-12T12:47:15.715464Z [info     ] Found catalog in /home/visch/git/meltano-projects/getting_started/catalog.json
2024-09-12T12:47:15.717292Z [debug    ] Invoking: ['/home/visch/git/meltano-projects/getting_started/.meltano/extractors/tap-smoke-test/venv/bin/tap-smoke-test', '--config', '/home/visch/git/meltano-projects/getting_started/.meltano/run/tap-smoke-test/tap.ceb61f67-c362-4a72-86b7-787eef00aa2a.config.json', '--catalog', '/home/visch/git/meltano-projects/getting_started/.meltano/run/tap-smoke-test/tap.properties.json']

It would be great if --catalog pointed to /home/visch/git/meltano-projects/getting_started/catalog.json instead with the meltano.yml (just the extractor portion below)

  - name: tap-smoke-test
    variant: meltano
    pip_url: git+https://github.com/meltano/tap-smoke-test.git
    catalog: catalog.json
    config:
      streams:
      - stream_name: User-pzjs3twtm2y4ip6uvfvy6ukhpa-prod
        input_filename:
          https://gitlab.com/meltano/tap-smoke-test/-/raw/main/demo-data/animals-data.jsonl
      - stream_name: Question-pzjs3twtm2y4ip6uvfvy6ukhpa-prod
        input_filename:
          https://gitlab.com/meltano/tap-smoke-test/-/raw/main/demo-data/pageviews-data.jsonl

Slack thread: https://meltano.slack.com/archives/C06A1MD6A6L/p1726095527072649?thread_ts=1726077531.449699&cid=C06A1MD6A6L

[edgarrmondragon]

Similar issue with state files: https://meltano.slack.com/archives/C069CQNHDNF/p1737575451962729?thread_ts=1737134613.343909&cid=C069CQNHDNF