[8.14 QA] Validate connector client

[khushbu-elastic]

Non-regression QA

• [ ] Postgres
• [ ] ABS

Note: always test with the latest Build Candidate on Elastic Cloud, using the full Elastic stack

• [ ] Start the whole stack from scratch and navigate to Enterprise Search
• [ ] Check that no indices are shown in the top level Indices list
• [ ] Click on "Create an Elasticsearch index" - a new page is open where you can select an ingestion method
• [ ] Choose Connector -> Use a connector
• [ ] Choose the connector you want to test and Continue
• [ ] Create an index with a valid name and Universal language

---

• [ ] Connector name and description are editable on the Configurations page

• [ ] Connector can be deleted from the Indices page

• [ ] Connector can be deleted from the Indices page and it can be recreated with the same name after

• [ ] Pull connectors repository, run make install but do not run connector yet

• [ ] Verify that you are redirected to "configuration" page where you can create an api key and can copy connector id / whole section of config into the connector

• [ ] Update connector configuration with the api_key and connector_id, plus choose a service_type to test and set it in config

• [ ] Start the connector by running make run - verify that it starts and does not actually do anything yet

• [ ] Wait for the Kibana page with the connector configuration to update and verify that it's possible to edit connector configuration now

• [ ] Edit and save connector configuration, then reload the page and verify that configuration is properly saved

• [ ] Click on "Set schedule and sync" and verify that you're redirected to the scheduling tab

• [ ] Enable scheduling for frequency = every minute and save schedule; refresh the page and verify that the changes were stored

• [ ] Switch to the connector and wait for a minute or two, verify that connector starts to ingest data

• [ ] Verify that the data from the connector appears in the expected index

• [ ] Verify that on the index list page index information is updated properly, showing expected number of documents and new index size

• [ ] Verify that on the connector overview page "Document Count" is updated to reflect the number of documents in the index

• [ ] Verify that you can see ingested documents in documents tab

• [ ] Verify that index mappings are correct on the index mappings tab

Record a short demo showing the connectors' configuration and that there were documents ingested

[DianaJourdan]

@khushbu-elastic instead of the steps described above, please test these 2 connectors following the public documentation forConnectors API and Connectors CLI

[khushbu-elastic]

@DianaJourdan

We have referred to the documentation for Connectors API & Connectors CLI and below are our findings:

Connectors API:

1. The API for creating & mapping an Index to connector seems missing. If we can add that too in the documentation, it would be easier for the user to do that via API instead of creating & mapping it through UI in the connector.

Connectors CLI:

1. With connectors job start, if the connector is configured, it does not gets connected with connector & shows Waiting for Sync status on UI & therefore we need to execute make run.
2. With Connectors CLI we have found that the user has to execute the connector with make run command which is not mentioned in the documentation.

We think that the documentation needs to be updated accordingly for better understanding of the users. Please let us know your opinion.

[DianaJourdan]

@khushbu-elastic

The API for creating & mapping an Index to connector seems missing. If we can add that too in the documentation, it would be easier for the user to do that via API instead of creating & mapping it through UI in the connector.

https://www.elastic.co/guide/en/elasticsearch/reference/current/create-connector-api.html -> index name is optional, we will update the documentation in the next release https://www.elastic.co/guide/en/elasticsearch/reference/current/update-connector-index-name-api.html

Is there another place where you were looking into the documentation?

[DianaJourdan]

@khushbu-elastic for the Connectors CLI test case, have you followed the steps here? Which connector have you tried to create? Just to confirm it was not a native connector converted to connector client? //cc @vidok

[khushbu-elastic]

@DianaJourdan

https://www.elastic.co/guide/en/elasticsearch/reference/current/create-connector-api.html -> index name is optional, we will update the documentation in the next release https://www.elastic.co/guide/en/elasticsearch/reference/current/update-connector-index-name-api.html

Is there another place where you were looking into the documentation?

We are checking for Connector API here only. The issue with these steps are, Index is not getting created and we have create it from the UI in order to map it with Connector via Connector API. Index mapping is mandatory (from Elastic v-8.13). So not sure how the execution will happen if Index Name is made optional.

@khushbu-elastic for the Connectors CLI test case, have you followed the steps here? Which connector have you tried to create? Just to confirm it was not a native connector converted to connector client?

We are following the same steps Connector CLI & we tried with both the connectors - i.e. PostgresSQL & ABS and both were not Native. Both were executed as Connector Client but still we need to execute them with make run command. The commands mentioned in documentation are not enough to execute the connector as Connector Client.

Also while executing Connector API, we have to generate the API Key from UI, which also needs to be added in the steps of Connector API.

CC: @vidok

[jedrazb]

We are checking for Connector API here only. The issue with these steps are, Index is not getting created and we have create it from the UI in order to map it with Connector via Connector API. Index mapping is mandatory (from Elastic v-8.13). So not sure how the execution will happen if Index Name is made optional.

Hey @khushbu-elastic, in 8.13 the the index_name become 'untied' from a connector, we no longer have strict enforcement of index being always attached to a connector, see update documentation on the latest docs branch (note master in the url path): https://www.elastic.co/guide/en/elasticsearch/reference/master/create-connector-api.html here we have:

index_name (Optional, string) The target index to sync data. If the index doesn’t exist, it will be created upon the first sync.

The framework will take care of creating the index with appropriate mappings upon the first sync. If you are confused about certain steps please provide us with feedback, and feel free to reach out to me for help with the APIs.

Also, in a matter of days we should have a tutorial in our docs that should make using a APIs a bit more clear ;)

[khushbu-elastic]

@jedrazb Thanks for sharing the updated API documentation for Create Connector API.

We had referred to the documentation and tried to Create Connector via API in 8.13 but it is showing 500 Internal Server error for POST _connector error shown as { "error": { "root_cause": [ { "type": "not_x_content_exception", "reason": "Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes" } ], "type": "not_x_content_exception", "reason": "Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes" }, "status": 500 }

and with

PUT _connector/<connector_id> error shown as { "error": { "root_cause": [ { "type": "null_pointer_exception", "reason": """Cannot invoke "String.length()" because "fileName" is null""" } ], "type": "null_pointer_exception", "reason": """Cannot invoke "String.length()" because "fileName" is null""" }, "status": 500 }

so we are not sure how will it work with request body being Optional (as mentioned in the documentation).

It would be great if you can share a demo for Connector API steps for better understanding.

[vidok]

@DianaJourdan

https://www.elastic.co/guide/en/elasticsearch/reference/current/create-connector-api.html -> index name is optional, we will update the documentation in the next release https://www.elastic.co/guide/en/elasticsearch/reference/current/update-connector-index-name-api.html Is there another place where you were looking into the documentation?

We are checking for Connector API here only. The issue with these steps are, Index is not getting created and we have create it from the UI in order to map it with Connector via Connector API. Index mapping is mandatory (from Elastic v-8.13). So not sure how the execution will happen if Index Name is made optional.

@khushbu-elastic for the Connectors CLI test case, have you followed the steps here? Which connector have you tried to create? Just to confirm it was not a native connector converted to connector client?

We are following the same steps Connector CLI & we tried with both the connectors - i.e. PostgresSQL & ABS and both were not Native. Both were executed as Connector Client but still we need to execute them with make run command. The commands mentioned in documentation are not enough to execute the connector as Connector Client.

Also while executing Connector API, we have to generate the API Key from UI, which also needs to be added in the steps of Connector API.

CC: @vidok

Thank you for your feedback! I addressed it in this PR.

[jedrazb]

Hey @khushbu-elastic , those errors are an obfuscated way that your request body is empty. We should add better error messages when the request body for connector creation is empty - I will create a ticket to handle this.

You should pass the request body to the create connector endpoints as in documentation, have you tried examples from the documentation? For now you can refer to this tutorial preview that will be released soon https://www.elastic.co/guide/en/enterprise-search/master/connectors-tutorial-api.html

[jedrazb]

Also, you can refer to the postman collection with all API endpoints here: https://github.com/elastic/connectors/tree/main/resources/connectors_api

[khushbu-elastic]

Hey @khushbu-elastic , those errors are an obfuscated way that your request body is empty. We should add better error messages when the request body for connector creation is empty - I will create a ticket to handle this.

You should pass the request body to the create connector endpoints as in documentation, have you tried examples from the documentation? For now you can refer to this tutorial preview that will be released soon https://enterprise-search-pubs_bk_4374.docs-preview.app.elstc.co/guide/en/enterprise-search/master/connectors-tutorial-api.html

We tried executing the Connector API from the updated documentation but still facing the same issue of Index not getting created/attached and we have create/attach it from the UI in order to map it with Connector via Connector API. Index mapping is still mandatory (in Elastic v-8.14-latest). PFA

@jedrazb Could you please share the demo for Connector API with steps for better understanding.

[jedrazb]

Hey @khushbu-elastic we have update index name endpoint that attached the index name to the connector: https://www.elastic.co/guide/en/elasticsearch/reference/master/update-connector-index-name-api.html

All the available endpoints are listed in: https://www.elastic.co/guide/en/elasticsearch/reference/master/connector-apis.html

[khushbu-elastic]

@jedrazb Okay, we can refer that when we will check for Index creation and attachment.

But if we are treating the index-name as an optional value, we should be able to move further without creating/attaching index to connector, according to my understanding.

Please let me know if we are not on same page for considering the index-name as an optional value.

[jedrazb]

@khushbu-elastic there might be a gap in Kibana that causes this issue when index doesn't exist yet. You should be able to get around it when:

• configuring your connector via API call: https://elastic.co/guide/en/enterprise-search/master/connectors-tutorial-api.html#connectors-tutorial-api-update-connector-configuration
• triggering a sync, this will create the actual index

[khushbu-elastic]

@DianaJourdan

@khushbu-elastic instead of the steps described above, please test these 2 connectors following the public documentation forConnectors API and Connectors CLI

We have verified the Postgresql & ABS as Connectors API and Connectors CLI. Below points are our observations:

1. We have successfully recorded the demo for Connectors API and you can refer to them with the below link:

• Demo for Postgresql and ABS.

2. For Connector CLI, we are facing the below issues (same as mentioned previously):

• With connectors job start, if the connector is configured, it does not gets connected with connector & shows Waiting for Sync status on UI & therefore we need to execute make run.
• With Connectors CLI we have found that the user has to execute the connector with make run command which is not mentioned in the documentation.
• Also we need to generate API key from Kibana and add it into config file manually.

CC: @jedrazb @vidok

[khushbu-elastic]

@DianaJourdan Please let us know if there is any update on this, considering the above comment. Looking forward for the reply.