elasticsearch-evolution
elasticsearch-evolution copied to clipboard
senacor
Error Creating Index Template in AWS OpenSearch Test Environment
Hello,
During the deployment of our application in the AWS OpenSearch test environment, I encountered an issue with creating an index template using our migration script. Local testing using Docker containers (image opensearchproject/opensearch:2.7.0) was successful. However, when attempting the same in the AWS environment, I receive a 400 error when trying to create the template.
Here are the details of the error from the logs:
Caused by: com.senacor.elasticsearch.evolution.core.api.MigrationException: execution of script 'FileNameInfoImpl{version=1, description='create order achiral index template', scriptName='V0001__create_order_achiral_index_template.http'}' failed
at com.senacor.elasticsearch.evolution.core.internal.migration.execution.MigrationServiceImpl.executeScript(MigrationServiceImpl.java:158) ~[elasticsearch-evolution-core-0.5.1.jar!/:na]
at com.senacor.elasticsearch.evolution.core.internal.migration.execution.MigrationServiceImpl.executePendingScriptsWithLock(MigrationServiceImpl.java:96) ~[elasticsearch-evolution-core-0.5.1.jar!/:na]
at com.senacor.elasticsearch.evolution.core.internal.migration.execution.MigrationServiceImpl.executePendingScripts(MigrationServiceImpl.java:73) ~[elasticsearch-evolution-core-0.5.1.jar!/:na]
at com.senacor.elasticsearch.evolution.core.ElasticsearchEvolution.migrate(ElasticsearchEvolution.java:112) ~[elasticsearch-evolution-core-0.5.1.jar!/:na]
at com.roche.max.global.opensearch.OpenSearchMigrationConfig.lambda$flywayMigrationStrategy$0(OpenSearchMigrationConfig.java:58) ~[!/:4.0.2]
at org.springframework.boot.autoconfigure.flyway.FlywayMigrationInitializer.afterPropertiesSet(FlywayMigrationInitializer.java:62) ~[spring-boot-autoconfigure-3.2.0.jar!/:3.2.0]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1822) ~[spring-beans-6.1.1.jar!/:6.1.1]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1771) ~[spring-beans-6.1.1.jar!/:6.1.1]
... 25 common frames omitted
Caused by: org.elasticsearch.client.ResponseException: method [PUT], host [https://<URL_TO_OPEN_SEARCH>.eu-central-1.es.amazonaws.com], URI [_template/order-achiral-separation], status line [HTTP/1.1 400 Bad Request]
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-8.10.4.jar!/:8.10.4]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-8.10.4.jar!/:8.10.4]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-8.10.4.jar!/:8.10.4]
at com.senacor.elasticsearch.evolution.core.internal.migration.execution.MigrationServiceImpl.executeScript(MigrationServiceImpl.java:145) ~[elasticsearch-evolution-core-0.5.1.jar!/:na]
... 32 common frames omitted
I can create the same template manually using Postman, which indicates that the issue may not be related to the configuration of our OpenSearch client, as we were able to create the migration index successfully. Could you assist in diagnosing why the migration scripts work locally but not in the AWS environment?
Thank you for your help.
I have no AWS OpenSearch Instance to play around. But you could log the exact HTTP request ElasticsearchEvolution is executing and the one you do by postman. I'm quite sure there is a mismatch with the HTTP Headers or something like that.
But you should add more information:
- how do you authenticate? IAM oder BasicAuth?
- Maybe https://github.com/senacor/elasticsearch-evolution/issues/236 could be helpfull if you use IAM? But looks like they are running ElasticsearchEvolution on AWS, so maybe @RiVogel can help you?
- AWS OpneSerachService is also on version
2.7.0?
Hey, @xtermi2, authentication with BasicAuth.
Yes OpenSearch on AWS has version 2.7.0. During running tests with connection to the AWS OpenSearch I'm getting the same response as in the TEST env. 400 Bad Request.
This is really strange, cuz I can execute those calls with the postman, as I described above. Any other ideas?
Hey @kadol92: Does each *.http file contain exactly one request or do you have multiple requests per file?
... and you are sure that your integration of Elasticsearch Evolution into Flyway does not modify or replace the request files at all? Therefor I agree @xtermi2, that you log the effective Elasticsearch queries which are going to the cluster.
Hey @RiVogel, @xtermi2,
*.http file contains one request per file
PUT _template/order-achiral-separation
Content-Type: application/json
{
"index_patterns": [
"order-achiral-separation-v1*"
],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1
},
"mappings": {
"properties": {
"_entity_type": {
"type": "keyword",
"index": false,
"doc_values": true
},
...
I've logged the details of the request, and they appear identical. I can't spot any differences. I conducted a quick test in the MigrationServiceImpl, where I switched the client to a restTemplate client and was able to successfully execute the call, just like in Postman.
Here is the change that I made:
var restTemplate = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
headers.setBasicAuth("user", "password");
headers.setContentType(MediaType.APPLICATION_JSON);
var exchange = restTemplate.exchange("https://<URL_TO_AWS_OPEN_SEARCH>/" + request.getEndpoint(), HttpMethod.PUT, new HttpEntity<>(scriptToExecute.getMigrationScriptRequest().getBody(), headers), String.class);
// var response = restClient.performRequest(request);
int statusCode = exchange.getStatusCode().value();
// int statusCode = response.getStatusLine().getStatusCode();
if (statusCode >= 200 && statusCode < 300) {
success = true;
} else {
error = Optional.of(new MigrationException(String.format(
"execution of script '%s' failed with HTTP status %s: %s",
scriptToExecute.getFileNameInfo(),
statusCode,
null)));
}
Btw, here is how I setup and run this migration:
@Slf4j
@Configuration
public class OpenSearchMigrationConfig {
@Value("${open-search.host}")
private String host;
@Value("${open-search.scheme}")
private String scheme;
@Value("${open-search.username}")
private String username;
@Value("${open-search.password}")
private String password;
@Value("${open-search.migration.index}")
private String migrationIndex;
@Value("${open-search.migration.location}")
private String location;
@Bean
public ElasticsearchEvolution elasticsearchEvolution() throws NoSuchAlgorithmException, KeyStoreException, KeyManagementException {
var basicCredentialsProvider = new BasicCredentialsProvider();
basicCredentialsProvider.setCredentials(new AuthScope(host, AuthScope.ANY_PORT), new UsernamePasswordCredentials(username, password));
var sslContext = SSLContextBuilder
.create()
.loadTrustMaterial(null, (chain, authType) -> true)
.build();
return ElasticsearchEvolution.configure()
.setHistoryIndex(migrationIndex)
.setLocations(List.of(location))
.load(RestClient.builder(createHttpHost(host, scheme))
.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(basicCredentialsProvider)
.setSSLContext(sslContext)
)
.build()
);
}
@Bean
public FlywayMigrationStrategy flywayMigrationStrategy(ElasticsearchEvolution elasticsearchEvolution) {
return flyway -> {
flyway.migrate();
log.info("Starting OpenSearch migration");
var startTimeStamp = System.currentTimeMillis();
var successfullyExecutedScripts = elasticsearchEvolution.migrate();
log.info("OpenSearch migration executed successfully {} migration scripts in {} ms", successfullyExecutedScripts, System.currentTimeMillis() - startTimeStamp);
};
}
private HttpHost createHttpHost(String host, String scheme) {
var validHost = host;
var schemeIdx = host.indexOf("://");
if (schemeIdx > 0) {
validHost = host.substring(schemeIdx + 3);
}
var port = -1;
var portIdx = validHost.lastIndexOf(":");
if (portIdx > 0) {
try {
port = Integer.parseInt(validHost.substring(portIdx + 1));
} catch (NumberFormatException var7) {
throw new IllegalArgumentException("Invalid HTTP host: " + validHost);
}
validHost = validHost.substring(0, portIdx);
} else if ("https".equals(scheme)) {
port = 443;
}
return new HttpHost(validHost, port, scheme);
}
}
This is a little modified from the initial version that I started. I added custom HttpHost creation and SSLContext, cuz I wasn't sure why it's failing, so I played around a little.
@RiVogel @xtermi2 any ideas why this solution works but not the one from the library? What am I missing? Or maybe there is some bug?
@kadol92: Doesn't your response contain any response body? For me, the http files look like this:
PUT /stuff-123/_settings
Content-Type: application/json
{
"analysis": {
"analyzer": {
...
And in Opensearch Dashboards I need to remove the Content-Type section. But it tells me that I did something wrong in the response body.
@RiVogel the only response body that I get is:
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>
Nothing more
When you have a look into your history index - is that one created? Or are there also problems?
I ask because I needed to sign every request which is going to AWS OpenSearch, but I do not use basic authentication but other authentication mechanisms.
Hey @RiVogel, yes, the history index has been created properly; the problem occurs when it tries to create the template for the first index.
Currently, I've used the RestTemplate client along with Apache's client5, and this works. The downside of this solution is that I had to modify the library because it doesn't support RestTemplate.
Ideally, it would be best to use this library without any modifications, but for now, this is a workaround.
@Bean
@Qualifier("openSearchRestTemplate")
public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder, CustomResponseErrorHandler errorHandler) {
var connectionManager = new PoolingHttpClientConnectionManager();
var httpClient = HttpClients.custom()
.setConnectionManager(connectionManager)
.build();
var requestFactory = new HttpComponentsClientHttpRequestFactory(httpClient);
requestFactory.setConnectionRequestTimeout(Duration.ofSeconds(10));
requestFactory.setConnectTimeout(Duration.ofSeconds(10));
return restTemplateBuilder
.requestFactory(() -> requestFactory)
.rootUri(scheme + "://" + host)
.basicAuthentication(username, password)
.errorHandler(errorHandler)
.build();
}