meltano
Add Plugin: tap-planetscale
What is the plugin type?
extractor
Repository URL
https://github.com/planetscale/singer-tap/
Extension Repository URL
No response
Description
PlanetScale is the world’s most advanced MySQL platform
Domain URL
https://planetscale.com/docs
Logo Image
Any other notes?
This tap is written in Go.
Note that with some settings set automatically MeltanoLabs tap-mysql tap can support this as well (After https://github.com/MeltanoLabs/tap-mysql/pull/45 is merged)
I hit an issue with their tap (I was trying to write Meltano instructions to place in the planetscale readme) see https://github.com/planetscale/singer-tap/issues/51
The problem with having this in the Hub would be that meltano add wouldn't be sufficient to set up the tap: users would still need to download or build the right binary, and then point executable to it. I don't know if we have another non-Python tap in the Hub.
@edgarrmondragon I have a memory of another non-python tap, but I don't think we added it to the Hub. Maybe @pnadolny13 remembers? It may have been this mysql one to be honest.
There's https://github.com/MeltanoLabs/target-jsonl-blob which is also Golang-based, and we never added it to the Hub.
@edgarrmondragon @tayloramurphy yeah I dont know of any that exist. I wonder if its helpful to still list it but exclude it from the API so its not accessible by meltano automatically. The hub page can explain how to install it manually if needed.
I'd have to look into this but we could also list it as a docker airbyte variant https://hub.docker.com/r/planetscale/airbyte-source/tags.
We could probably do something better than this but here's a "working" example. Note that their tap doesn't follow the singer spec so I don't think we can add this anyway :/ See https://github.com/planetscale/singer-tap/issues/51 maybe they'll adjust it though!
extractors:
- name: tap-planetscale
namespace: tap_planetscale
pip_url: cowsay # I don't think we need to do this, could just have no pip url as you don't have to run install?
executable: /home/visch/git/tap-mysql/ps-singer-tap
capabilities:
- state
- catalog
- discover
- about
- stream-maps
settings:
- name: host
- name: username
- name: database
- name: password
sensitive: true
config:
host: aws.connect.psdb.cloud
username: 0fiqne6txvcqtjbdywan
database: tap-mysql
select:
- "*.*"
I played around a bit with the latest version of the planet scale tap (they fixed the catalog output)
I can't get meltano to properly select the fields I have, or add the proper metadata. If I manually add a select, and update the replication-method to FULL_TABLE the tap works as advertised.
Not sure exactly what's going on here, my best guess is the stream_id having a : in the name (right now it's tap-mysql:testtable ) might be throwing off something maybe a regex strips them internally.
visch@DESKTOP-9BDPA9T:~/git/tap-mysql$ meltano --log-level=debug invoke --dump=catalog tap-planetscale
2024-02-22T20:03:56.095140Z [debug ] meltano 3.3.0, Linux
2024-02-22T20:03:56.098361Z [debug ] /etc/timezone found, contents:
America/New_York
2024-02-22T20:03:56.098817Z [debug ] /etc/localtime found
2024-02-22T20:03:56.099580Z [debug ] 2 found:
{'/etc/timezone': 'America/New_York', '/etc/localtime is a symlink to': 'America/New_York'}
2024-02-22T20:03:56.101959Z [info ] Environment 'test' is active
2024-02-22T20:03:56.119210Z [debug ] Creating DB engine for project at '/home/visch/git/tap-mysql' with DB URI 'sqlite://home/visch/git/tap-mysql/.meltano/meltano.db'
2024-02-22T20:03:56.243347Z [debug ] Created configuration at /home/visch/git/tap-mysql/.meltano/run/tap-planetscale/tap.b204835c-78f5-4ac4-a5af-d6f5a295f714.config.json
2024-02-22T20:03:56.243643Z [debug ] Could not find tap.properties.json in /home/visch/git/tap-mysql/.meltano/extractors/tap-planetscale/tap.properties.json, skipping.
2024-02-22T20:03:56.243844Z [debug ] Could not find tap.properties.cache_key in /home/visch/git/tap-mysql/.meltano/extractors/tap-planetscale/tap.properties.cache_key, skipping.
2024-02-22T20:03:56.244042Z [debug ] Could not find state.json in /home/visch/git/tap-mysql/.meltano/extractors/tap-planetscale/state.json, skipping.
2024-02-22T20:03:56.245698Z [debug ] Invoking: ['/home/visch/git/tap-mysql/ps-singer-tap', '--config', '/home/visch/git/tap-mysql/.meltano/run/tap-planetscale/tap.b204835c-78f5-4ac4-a5af-d6f5a295f714.config.json', '--discover']
PlanetScale Tap : INFO : PlanetScale Singer Tap : version ["0.27.0"], commit ["29d23da"], published on ["2024-02-21T19:10:50Z"]
PlanetScale Tap : INFO : running in discovery mode
PlanetScale Tap : INFO : Discovering Schema for PlanetScale database : tap-mysql
2024-02-22T20:03:56.924387Z [debug ] Visiting CatalogNode.STREAM at '.streams[0]'.
2024-02-22T20:03:56.924620Z [debug ] Unknown node type 'CatalogNode.STREAM'.
2024-02-22T20:03:56.924774Z [debug ] Skipping node at '.streams[0].type'
2024-02-22T20:03:56.924897Z [debug ] Skipping node at '.streams[0].stream'
2024-02-22T20:03:56.925024Z [debug ] Skipping node at '.streams[0].tap_stream_id'
2024-02-22T20:03:56.925159Z [debug ] Skipping node at '.streams[0].schema.type[0]'
2024-02-22T20:03:56.925283Z [debug ] Skipping node at '.streams[0].schema.type[1]'
2024-02-22T20:03:56.925441Z [debug ] Skipping node at '.streams[0].schema.additionalProperties'
2024-02-22T20:03:56.925622Z [debug ] Visiting CatalogNode.PROPERTY at '.streams[0].schema.properties.test'.
2024-02-22T20:03:56.925752Z [debug ] Unknown node type 'CatalogNode.PROPERTY'.
2024-02-22T20:03:56.925878Z [debug ] Skipping node at '.streams[0].table-name'
2024-02-22T20:03:56.926016Z [debug ] Skipping node at '.streams[0].metadata[0].metadata.selected'
2024-02-22T20:03:56.926139Z [debug ] Skipping node at '.streams[0].metadata[0].metadata.replication-method'
2024-02-22T20:03:56.926261Z [debug ] Skipping node at '.streams[0].metadata[0].metadata.inclusion'
2024-02-22T20:03:56.926402Z [debug ] Skipping node at '.streams[0].metadata[1].metadata.selected'
2024-02-22T20:03:56.926537Z [debug ] Skipping node at '.streams[0].metadata[1].metadata.inclusion'
2024-02-22T20:03:56.926662Z [debug ] Skipping node at '.streams[0].metadata[1].metadata.breadcrumb[0]'
2024-02-22T20:03:56.926786Z [debug ] Skipping node at '.streams[0].metadata[1].metadata.breadcrumb[1]'
2024-02-22T20:03:56.926935Z [debug ] Skipping node at '.streams[0].key_properties'
2024-02-22T20:03:56.927061Z [debug ] Skipping node at '.streams[0].bookmark_properties'
2024-02-22T20:03:56.929199Z [debug ] Invoking: ['/home/visch/git/tap-mysql/ps-singer-tap', '--config', '/home/visch/git/tap-mysql/.meltano/run/tap-planetscale/tap.b204835c-78f5-4ac4-a5af-d6f5a295f714.config.json', '--catalog', '/home/visch/git/tap-mysql/.meltano/run/tap-planetscale/tap.properties.json']
{
"streams": [
{
"type": "",
"stream": "testtable",
"tap_stream_id": "tap-mysql:testtable",
"schema": {
"type": [
"null",
"object"
],
"additionalProperties": false,
"properties": {
"test": {
"type": [
"null",
"string"
]
}
}
},
"table-name": "testtable",
"metadata": [
{
"metadata": {
"selected": false,
"replication-method": "INCREMENTAL",
"inclusion": "available",
"breadcrumb": []
}
},
{
"metadata": {
"selected": false,
"inclusion": "available",
"breadcrumb": [
"properties",
"test"
]
}
}
],
"key_properties": null,
"bookmark_properties": null
}
]
}
Meltano.yml
extractors:
- name: tap-planetscale
namespace: tap_planetscale
pip_url: cowsay
executable: /home/visch/git/tap-mysql/ps-singer-tap
capabilities:
- state
- catalog
- discover
settings:
- name: host
- name: username
- name: database
- name: password
sensitive: true
config:
host: aws.connect.psdb.cloud
username: 0fiqne6txvcqtjbdywan
database: tap-mysql
select:
- '*.*'
metadata:
'*':
replication-method: FULL_TABLE