Fault on Load Collector with MSSQL

[paulorobertokm]

The error occur when I configure collector for get information from microsoft sql server.

odd-collector_1 | import pyodbc odd-collector_1 | ImportError: libodbc.so.2: cannot open shared object file: No such file or directory odd-collector_1 | [2022-07-27 20:58:12,894] ERROR in odd-collector: libodbc.so.2: cannot open shared object file: No such file or directory odd-collector_1 | Traceback (most recent call last): odd-collector_1 | File "/app/odd_collector/main.py", line 24, in odd-collector_1 | collector = Collector( odd-collector_1 | File "/usr/local/lib/python3.9/site-packages/odd_collector_sdk/collector.py", line 32, in init odd-collector_1 | self.adapters_with_plugins = adapter_initializer.init_adapters() odd-collector_1 | File "/usr/local/lib/python3.9/site-packages/odd_collector_sdk/domain/adapters_initializer.py", line 60, in init_adapters odd-collector_1 | for package, plugin in self._load_packages() odd-collector_1 | File "/usr/local/lib/python3.9/site-packages/odd_collector_sdk/domain/adapters_initializer.py", line 44, in _load_packages odd-collector_1 | self._import_odd_package_modules(imported_package) odd-collector_1 | File "/usr/local/lib/python3.9/site-packages/odd_collector_sdk/domain/adapters_initializer.py", line 33, in _import_odd_package_modules odd-collector_1 | return [import_module(module_path) for module_path in package_modules] odd-collector_1 | File "/usr/local/lib/python3.9/site-packages/odd_collector_sdk/domain/adapters_initializer.py", line 33, in odd-collector_1 | return [import_module(module_path) for module_path in package_modules] odd-collector_1 | File "/usr/local/lib/python3.9/importlib/init.py", line 127, in import_module odd-collector_1 | return _bootstrap._gcd_import(name[level:], package, level) odd-collector_1 | File "", line 1030, in _gcd_import odd-collector_1 | File "", line 1007, in _find_and_load odd-collector_1 | File "", line 986, in _find_and_load_unlocked odd-collector_1 | File "", line 680, in _load_unlocked odd-collector_1 | File "", line 850, in exec_module odd-collector_1 | File "", line 228, in _call_with_frames_removed odd-collector_1 | File "/app/odd_collector/adapters/mssql/adapter.py", line 2, in

My YAML file (Collector)

platform_host_url: http://192.168.0.153:8080 default_pulling_interval: 10 token: "LXld28sSxxxxxxxxxxxxxxxxxrg3tsohRfrBmkqXhxY" plugins:

• type: mssql name: AdventureWorsSQLServer description: "" # Optional string database: AdventureWorks2017 host: IP_HOST port: 1433 user: USERNAME password: xxxxx driver: 'ODBC Driver 17 for SQL Server'

The problem occur after start the collector container.

[DementevNikita]

@PauloROliveiraKM thank you for letting us know!

We are looking at this issue as we speak, meanwhile could you please elaborate on your OS and OS's version? Are you running collector on a local machine or docker?

From my point of view you lack some system dependencies for pyodbc. If you're running host linux try installing unixodbc-dev package using your package manager such as, for example, apt

[paulorobertokm]

Hi, I'm running collector in Docker, using your image and yaml file. I cannot run any command in container, because i'm not have administrative privileges for running apt command. The docker is running in Ubuntu 22.04

[paulorobertokm]

I did it insert mssql collector but the error change. In some cases this error showing for me and the lineage function is not working with MSSQL Databases.

odd-platform_1 | 2022-07-28 21:09:56.665 INFO 1 --- [tor-tcp-epoll-1] o.o.o.s.DataSourceIngestionServiceImpl : Going to update datasources with oddrns odd-platform_1 | 2022-07-28 21:09:56.666 INFO 1 --- [tor-tcp-epoll-1] o.o.o.s.DataSourceIngestionServiceImpl : Going to create datasources with oddrns //mssql/host/192.168.0.106/databases/Credit odd-collector_1 | [2022-07-28 21:09:56,762] INFO in apscheduler.scheduler: Adding job tentatively -- it will be properly scheduled when the scheduler starts odd-collector_1 | [2022-07-28 21:09:56,763] INFO in apscheduler.scheduler: Added job "Collector.__ingest_data" to job store "default" odd-collector_1 | [2022-07-28 21:09:56,763] INFO in apscheduler.scheduler: Scheduler started odd-collector_1 | [2022-07-28 21:09:56,764] INFO in apscheduler.executors.default: Running job "Collector.__ingest_data (trigger: interval[0:10:00], next run at: 2022-07-28 21:19:56 UTC)" (scheduled at 2022-07-28 21:09:56.692708+00:00) odd-collector_1 | [2022-07-28 21:09:57,072] ERROR in root: Failed to load metadata for tables odd-collector_1 | [2022-07-28 21:09:57,072] ERROR in root: 'NoneType' object has no attribute 'match' odd-collector_1 | Traceback (most recent call last): odd-collector_1 | File "/app/odd_collector/adapters/mssql/adapter.py", line 45, in get_data_entities odd-collector_1 | return map_table(self.__oddrn_generator, tables, columns) odd-collector_1 | File "/app/odd_collector/adapters/mssql/mappers/tables.py", line 52, in map_table odd-collector_1 | data_entity.data_transformer = extract_transformer_data( odd-collector_1 | File "/app/odd_collector/adapters/mssql/mappers/views.py", line 13, in extract_transformer_data odd-collector_1 | inputs, outputs = sql_parser.get_response() odd-collector_1 | File "/usr/local/lib/python3.9/site-packages/odd_models/utils.py", line 70, in get_response odd-collector_1 | self.get_tokens() odd-collector_1 | File "/usr/local/lib/python3.9/site-packages/odd_models/utils.py", line 79, in get_tokens odd-collector_1 | cte_name, cte_intables = self.parse_cte(tokens) odd-collector_1 | File "/usr/local/lib/python3.9/site-packages/odd_models/utils.py", line 110, in parse_cte odd-collector_1 | if not as_keyword.match(T.Keyword, values=['AS']): odd-collector_1 | AttributeError: 'NoneType' object has no attribute 'match' odd-collector_1 | [2022-07-28 21:09:57,162] INFO in apscheduler.executors.default: Job "Collector.__ingest_data (trigger: interval[0:10:00], next run at: 2022-07-28 21:19:56 UTC)" executed successful

[Vixtir]

@PauloROliveiraKM thank you for report. I created issue in odd-models-package repo.

I have added more friendly error handling. Now you can pull newest odd-collector image and run collector again, you should see some metadata about selected database on UI. Also you can add LOGLEVEL='DEBUG' environment variable to odd-collector container to see failed view.

[paulorobertokm]

I set de LOGLEVEL =DEBUG in my docker compose file.

version: "3.8" services: odd-platform: image: ghcr.io/opendatadiscovery/odd-platform:latest restart: always environment: - SPRING_DATASOURCE_URL=jdbc:postgresql://192.168.0.115:5432/${POSTGRES_DATABASE} - SPRING_DATASOURCE_USERNAME=${POSTGRES_USER} - SPRING_DATASOURCE_PASSWORD=${POSTGRES_PASSWORD} ports: - 8080:8080

odd-platform-enricher: image: python:3.9.12-alpine3.15 volumes: - ./injector:/injector command: - sh - ./injector/start.sh environment: - PLATFORM_HOST_URL=${PLATFORM_HOST_URL} - APP_PATH=./injector - PYTHONUNBUFFERED=1 depends_on: - odd-platform

odd-collector: image: ghcr.io/opendatadiscovery/odd-collector:latest restart: always volumes: - ./config/collector_config.yaml:/app/collector_config.yaml environment: - PLATFORM_HOST_URL=${PLATFORM_HOST_URL} - LOGLEVEL=DEBUG - But, the lineage function, still not working, not is showing the interaction between tables, it's show only select table.

In MySQL and PostGres, it's works.

[DementevNikita]

@PauloROliveiraKM

Thank you for letting us know. We'll keep you in touch

[paulorobertokm]

I don't know if you can help me, but, how I use the functionality, dictionary. Exist any documentation about the tools functionality?

---

De: Nikita Dementev @.> Enviado: segunda-feira, 1 de agosto de 2022 02:01 Para: opendatadiscovery/odd-platform @.> Cc: Paulo Roberto de Oliveira @.>; Mention @.> Assunto: Re: [opendatadiscovery/odd-platform] Fault on Load Collector with MSSQL (Issue #874)

@PauloROliveiraKMhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FPauloROliveiraKM&data=05%7C01%7C%7C56f4b491501948e1901908da73728ae9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637949233069629068%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rv%2B4WRCX6LFsrglFsSp1AlK5HvcktNRWTk6O%2BcIitho%3D&reserved=0

Thank you for letting us know. We'll keep you in touch

— Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopendatadiscovery%2Fodd-platform%2Fissues%2F874%23issuecomment-1200667640&data=05%7C01%7C%7C56f4b491501948e1901908da73728ae9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637949233069629068%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=D2cp4UNRgS%2BTSc5zJXC41i9hqry0th9GqBK5ikD1qc0%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACAXLNNL3TSWI7II2YIA3U3VW5D2LANCNFSM543BDTMQ&data=05%7C01%7C%7C56f4b491501948e1901908da73728ae9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637949233069629068%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=B55LvP0cNxQ4D9IgXP8jYoxC4QZskT%2BSeTZRliu3C7Q%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

[DementevNikita]

@PauloROliveiraKM hey there! sorry for the late response

dictionary is basically a business glossary. It contains concepts and definitions of business terms frequently used in day-to-day activities within an organization or business unit

[DementevNikita]

@paulorobertokm please take a look at the new version of odd-collector