Are there detailed tutorials or demos

[userxxj]

[jlloyd-widen]

If there are any users of this tap that do have publicly available examples, please post them here.

Here is an "obfuscated" example of a meltano.yml file. I'm using it to hit an API internal to my company and another that is an API to datadog for which I couldn't find an acceptable tap and didn't want to develop one of my own (not an efficient solution). FYI, This doesn't even get close to exposing the full functionality of the tap; these examples are very simple.

  - name: tap-rest-api
    namespace: tap_rest_api_msdk
    pip_url: tap-rest-api-msdk
    executable: tap-rest-api-msdk
    capabilities:
    - state
    - catalog
    - discover
    settings:
    - name: api_url
    - name: next_page_token_path
    - name: pagination_request_style
    - name: pagination_response_style
    - name: pagination_page_size
    - name: streams
    - name: path
    - name: params
    - name: headers
    - name: records_path
    - name: primary_keys
    - name: replication_key
    - name: except_keys
    - name: num_inference_records
  - name: tap-***
    inherit_from: tap-rest-api
    config:
      api_url: https://***
      streams:
      - name: ***
        path: /
        primary_keys:
        - id
        replication_key: last_data_refresh
        except_keys:
        - cnames
        - cdn_embeds
        - cdn_hosts
        - collective_message_keys
        - collective_properties
        - brandings
        - nginx
        - properties
        - skins
        records_path: $.accounts[*]
        num_inference_records: 200
    select:
    - '*.*'
  - name: tap-datadog-wdt
    inherit_from: tap-rest-api
    config:
      api_url: https://api.datadoghq.com/api/v1
      streams:
      - name: ApiSummary
        path: /query
        headers:
          Content-Type: application/json
          DD-API-KEY: TAP_DD_API_KEY
          DD-APPLICATION-KEY: TAP_DD_APPLICATION_KEY
        params:
          query: sum:widen.api.count{env:prod}by{source,account_name,user_oauth_app,trace_span_operation,http_request_method,http_response_status}.rollup(86400)
          from: TAP_DD_FROM
          to: TAP_DD_TO
        primary_keys:
        - scope
        - start
        - end
        records_path: $.series[*]
    select:
    - '*.*'

[EJOOSTEROP]

[edit]Never mind this - I now noticed you have the same/similar meltano.yml in your project already...

For the earthquakes example.

With:

• meltano add extractor tap-rest-api-msdk
• meltano add loader target-jsonl

meltano.yml as follows:

version: 1
default_environment: dev
project_id: 1f67e0b0-285b-4961-b71d-b825bce3dcc7
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-rest-api-msdk
    variant: widen
    pip_url: tap-rest-api-msdk
    capabilities:
    - state
    - catalog
    - discover
    settings:
    - name: api_url
    - name: next_page_token_path
    - name: pagination_request_style
    - name: pagination_response_style
    - name: pagination_page_size
    - name: streams
    - name: path
    - name: params
    - name: headers
    - name: records_path
    - name: primary_keys
    - name: replication_key
    - name: except_keys
    - name: num_inference_records
    config:
      api_url: https://earthquake.usgs.gov/fdsnws
      streams:
      - name: us_earthquakes
        params:
          format: geojson
          starttime: '2022-11-01'
          endtime: '2014-01-02'
          minmagnitude: 1
        path: /event/1/query
        primary_keys:
        - id
        records_path: $.features[*]
        num_inference_records: 200
    select:
    - '*.*'
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
schedules:
- name: rest-api-msdk-to-jsonl
  interval: '@hourly'
  extractor: tap-rest-api-msdk
  loader: target-jsonl
  transform: skip
  start_date: 2022-11-05 02:41:58.166575

[s7clarke10]

Look at the documentation regarding this PR: https://github.com/Widen/tap-rest-api-msdk/pull/36. It may resolve you problem.

[vincentclaes]

version: 1
default_environment: dev
project_id: d63825fc-4b14-4051-a9f4-d96776f9fd39
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-rest-api-msdk
    variant: widen
    pip_url: tap-rest-api-msdk
    config:
      api_url: https://foo.elabjournal.com/api/v1
      streams:
      - name: foo
        path: /experiments
        headers:
          Authorization: <some api key>
        primary_keys:
        - experimentID
        records_path: $.data[*]

I want all the records inside the key "data"

{
  "recordCount": 1000,
  "currentPage": 0,
  "maxRecords": 1000,
  "totalRecords": 2036,
  "data": [
    {
      "usesSignatureWorkflow": false,
      "studyID": 45453,
      "projectID": 343434,,
      "experimentStatus": {
        "experimentStatusID": 23232,
        "groupID": 6445,
        "color": "#AA00AA",
        "status": "Completed",
        "experimentStatusType": "COMPLETED"
      },
    ...
]
}