sdk icon indicating copy to clipboard operation
sdk copied to clipboard
verified publisher icon meltano

feat: Schema Only Flag

Open joshua-janicas opened this issue 1 year ago • 4 comments

Feature scope

Taps (catalog, state, tests, etc.)

Description

Feature Idea

  • A configurable option or command of a tap that would perform a schema/header extract only with no data attached. This would not update any state files.
  • Should be flexible with most taps (including those that are not database focused) where we just pull the "fields/column names" from our data sources and ignore the data itself.
  • Example: pg_dump with --schema-only in PostGres, BACPAC vs DACPAC in MSSQL.
  • --schema-only or some more appropriate generic naming

Use Case

  • Populating a target with empty fields
  • Creating a "EMPTY_DB" in Snowflake with only empty tables, no data
it's a weird intersection of our business intelligence software and how our upstream data is handled. we need to template reports with no data to give to our clients to publish. the rub is that each client is on a different version of our database which we import data from using tap-mssql and those versions may have schema differences. we can pin a client to a specific version of our elt pipeline but that results in potentially different tables (a column added, renamed...) (edited)

joshua-janicas avatar Dec 12 '24 14:12 joshua-janicas

I wonder if a mapper would do this today with filters. Try setting the filter to False for every record https://sdk.meltano.com/en/v0.43.1/stream_maps.html#filtering-out-records-from-a-stream-using-filter-operation

Might work 🤷

visch avatar Dec 12 '24 15:12 visch

If I really needed this, I'd mess with the CLI test options and manually do it so something like

  1. meltano invoke tap-name --test schema > out
  2. cat out | meltano invoke target-name

visch avatar Dec 12 '24 15:12 visch

So I've finally gotten around to trying this out. I was trying to use a tap tap-mssql-static that inherits from tap-mssql.

plugins:
  extractors:
  - name: tap-mssql
    variant: buzzcutnorman
    pip_url: git+https://github.com/BuzzCutNorman/[email protected]
    config:
      dialect: mssql
      driver_type: pyodbc
      sqlalchemy_eng_params:
        fast_executemany: 'True'
      sqlalchemy_url_query:
        driver: ODBC Driver 18 for SQL Server

plugins:
  extractors:
  - name: tap-mssql-static
    inherit_from: tap-mssql
    select:
    - Static-Country.*
    - Static-Region.*

A few things:

  1. Running meltano invoke tap-name --test schema > out ignores the specific tap I'm trying to use and selects all the tables from the database that exist.
  2. Then running cat out | meltano invoke target-snowflake breaks on tables in Snowflake that does not exist to begin with (particularly because of (1) listing all the tables instead of the one I want).

Stack trace is below, note that the first Admin-AdminCart already exists as a table (from another tap i have configured), but because Admin-AdminCartEtag does not exist it blows up. In reality I was hoping neither would show up, the tables that it should be trying to do is Static-Country and Static-Region.

Even if it did that though, if those two tables didn't exist it'd be crashing out anyways. Thoughts?

Stack Trace 
Environment 'dev' is active   
Initializing 'target-snowflake' target sink...
Initializing target sink for stream 'Admin-AdminCart'...
Private key is in PEM format
Initializing 'target-snowflake' target sink...
Initializing target sink for stream 'Admin-AdminCartEtag'...
Error creating "TEST"."RAW"."ADMINCARTETAG" {'properties': {}, 'type': 'object', 'required': ['AdminCartId']}
Traceback (most recent call last):
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\engine\base.py", line 1964, in _exec_single_context
    self.dialect.do_execute(
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\engine\default.py", line 945, in do_execute
    cursor.execute(statement, parameters)
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\snowflake\connector\cursor.py", line 1134, in execute
    Error.errorhandler_wrapper(self.connection, self, error_class, errvalue)
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\snowflake\connector\errors.py", line 279, in errorhandler_wrapper
    handed_over = Error.hand_to_other_handler(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\snowflake\connector\errors.py", line 334, in hand_to_other_handler
    cursor.errorhandler(connection, cursor, error_class, error_value)
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\snowflake\connector\errors.py", line 210, in default_errorhandler
    raise error_class(
snowflake.connector.errors.ProgrammingError: 001003 (42000): SQL compilation error:
syntax error line 2 at position 0 unexpected ')'.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\target_snowflake\sinks.py", line 81, in setup
    self.connector.prepare_table(
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\singer_sdk\connectors\sql.py", line 1388, in prepare_table
    self.create_empty_table(
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\singer_sdk\connectors\sql.py", line 1331, in create_empty_table
    meta.create_all(self._engine)
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\sql\schema.py", line 5925, in create_all
    bind._run_ddl_visitor(
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\engine\base.py", line 3249, in _run_ddl_visitor
    conn._run_ddl_visitor(visitorcallable, element, **kwargs)
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\engine\base.py", line 2456, in _run_ddl_visitor
    visitorcallable(self.dialect, self, **kwargs).traverse_single(element)
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\sql\visitors.py", line 664, in traverse_single
    return meth(obj, **kw)
           ^^^^^^^^^^^^^^^
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\sql\ddl.py", line 981, in visit_metadata
    self.traverse_single(
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\sql\visitors.py", line 664, in traverse_single
    return meth(obj, **kw)
           ^^^^^^^^^^^^^^^
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\sql\ddl.py", line 1019, in visit_table
    )._invoke_with(self.connection)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\sql\ddl.py", line 321, in _invoke_with
    return bind.execute(self)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\engine\base.py", line 1416, in execute
    return meth(
           ^^^^^
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\sql\ddl.py", line 187, in _execute_on_connection
    return connection._execute_ddl(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\engine\base.py", line 1527, in _execute_ddl
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\engine\base.py", line 1843, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\engine\base.py", line 1983, in _exec_single_context
    self._handle_dbapi_exception(
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\engine\base.py", line 2352, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\engine\base.py", line 1964, in _exec_single_context
    self.dialect.do_execute(
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\sqlalchemy\engine\default.py", line 945, in do_execute
    cursor.execute(statement, parameters)
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\snowflake\connector\cursor.py", line 1134, in execute
    Error.errorhandler_wrapper(self.connection, self, error_class, errvalue)
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\snowflake\connector\errors.py", line 279, in errorhandler_wrapper
    handed_over = Error.hand_to_other_handler(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\snowflake\connector\errors.py", line 334, in hand_to_other_handler
    cursor.errorhandler(connection, cursor, error_class, error_value)
  File "C:\Git\elt-pipeline\.meltano\loaders\target-snowflake\venv\Lib\site-packages\snowflake\connector\errors.py", line 210, in default_errorhandler
    raise error_class(
sqlalchemy.exc.ProgrammingError: (snowflake.connector.errors.ProgrammingError) 001003 (42000): SQL compilation error:
syntax error line 2 at position 0 unexpected ')'.
[SQL:
CREATE TABLE "RAW"."ADMINCARTETAG" (
)

]

joshua-janicas avatar May 08 '25 15:05 joshua-janicas

I am trying visch's first idea with the mappers and that is more what I am looking for I think.

plugins:
  loaders:
  - name: target-snowflake-overwrite
    inherit_from: target-snowflake
    config:
      load_method: overwrite

plugins:
  extractors:
  - name: tap-mssql-static
    inherit_from: tap-mssql
    select:
    - Static-Country.*
    - Static-Region.*
  - name: tap-mssql-static-empty
    inherit_from: tap-mssql-static
    config:
      stream_maps:
        Static-Country:
          __filter__:  'False'
        Static-Region:
          __filter__: 'False'

Running meltano run tap-mssql-static-empty target-snowflake-overwrite behaves as I expect it to - so I'll go down that path.

Separate issue is I was trying to use the "*" https://sdk.meltano.com/en/v0.43.1/stream_maps.html#applying-a-mapping-across-two-or-more-streams but it dies because it's trying to apply the filter to all the tables it finds instead of the one's I'm trying to filter on via the inherited select.

      stream_maps:
        "*":
          __filter__: 'False'
Stack Trace 
elt-pipeline: [CAM-23234 ≡ +2 ~4 -0 !]> meltano run tap-mssql-static-empty target-snowflake-overwrite --full-refresh
2025-05-08T16:06:47.293207Z [info     ] Environment 'dev' is active   
2025-05-08T16:06:49.628945Z [info     ] Performing full refresh, ignoring state left behind by any previous runs.
2025-05-08T16:07:36.730188Z [info     ] Found 'Admin-AdminCart' filter rule: False cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.731690Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.732695Z [info     ]   File "<frozen runpy>", line 198, in _run_module_as_main cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.733690Z [info     ]   File "<frozen runpy>", line 88, in _run_code cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.734691Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Scripts\tap-mssql.exe\__main__.py", line 7, in <module> cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.734691Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Lib\site-packages\click\core.py", line 1161, in __call__ cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.735702Z [info     ]     return self.main(*args, **kwargs) cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.736712Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.737221Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Lib\site-packages\click\core.py", line 1082, in main cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.737720Z [info     ]     rv = self.invoke(ctx)      cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.738719Z [info     ]          ^^^^^^^^^^^^^^^^      cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.739219Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Lib\site-packages\singer_sdk\plugin_base.py", line 80, in invoke cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.739719Z [info     ]     return super().invoke(ctx) cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.740720Z [info     ]            ^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.741221Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Lib\site-packages\click\core.py", line 1443, in invoke cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.741722Z [info     ]     return ctx.invoke(self.callback, **ctx.params) cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.742719Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.743217Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Lib\site-packages\click\core.py", line 788, in invoke cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.744218Z [info     ]     return __callback(*args, **kwargs) cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.744218Z [info     ]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.745812Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Lib\site-packages\singer_sdk\tap_base.py", line 500, in invoke cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.746312Z [info     ]     tap = cls(                 cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.746812Z [info     ]           ^^^^                 cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.747825Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Lib\site-packages\singer_sdk\tap_base.py", line 649, in __init__ cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.748321Z [info     ]     super().__init__(*args, **kwargs) cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.749322Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Lib\site-packages\singer_sdk\tap_base.py", line 109, in __init__ cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.750323Z [info     ]     self.setup_mapper()        cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.751321Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Lib\site-packages\singer_sdk\tap_base.py", line 179, in setup_mapper cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.752319Z [info     ]     self.mapper.register_raw_streams_from_catalog(self.catalog) cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.752819Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Lib\site-packages\singer_sdk\mapper.py", line 688, in register_raw_streams_from_catalog cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.753321Z [info     ]     self.register_raw_stream_schema( cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.754323Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Lib\site-packages\singer_sdk\mapper.py", line 774, in register_raw_stream_schema cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.755332Z [info     ]     mapper = CustomStreamMap(  cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.756842Z [info     ]              ^^^^^^^^^^^^^^^^  cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.757343Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Lib\site-packages\singer_sdk\mapper.py", line 272, in __init__ cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.758352Z [info     ]     ) = self._init_functions_and_schema(stream_map=map_transform) cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.758852Z [info     ]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.759852Z [info     ]   File "C:\Git\elt-pipeline\.meltano\extractors\tap-mssql\venv\Lib\site-packages\singer_sdk\mapper.py", line 524, in _init_functions_and_schema cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.760352Z [info     ]     raise StreamMapConfigError(msg) cmd_type=elb consumer=False job_name=dev:tap-mssql-static-empty-to-target-snowflake-overwrite name=tap-mssql-static-empty producer=True run_id=8868580e-6fa9-438f-adcc-6e1bed5785ce stdio=stderr string_id=tap-mssql-static-empty
2025-05-08T16:07:36.761353Z [info     ] singer_sdk.exceptions.StreamMapConfigError: Invalid key properties for 'Admin-AdminCart': [AdminCartId]. Property 'AdminCartId' was not detected in schema. cmd_ty

joshua-janicas avatar May 08 '25 16:05 joshua-janicas