elasticsearch-jdbc
elasticsearch-jdbc copied to clipboard
mysql bigint field inserto to es , find error data
example:
CREATE TABLE product_info (
item_id bigint(20) NOT NULL DEFAULT '0' ,
);
whene item_id too long , use jdbc insert es from mysql will error;
also, we can test on sense ` PUT lock/test/5 { "name":"del_5", "age":20, "num":"-8629845578222121431", "bignum":-8629845578222121431 }
GET lock/test/_search
{
"query": {
"term": {
"name": {
"value": "del_5"
}
}
}
}
result:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.15342641,
"hits": [
{
"_index": "lock",
"_type": "test",
"_id": "5",
"_score": 0.15342641,
"_source": {
"name": "del_5",
"age": 20,
"num": "-8629845578222121431",
"bignum": -8629845578222121000
}
}
]
}
}
`
we can see bignum change to 8629845578222121000 , rather than 8629845578222121431
it's precision question .
how too fix this question when use elasticsearch-jdbc tool index data to elasticsearch from mysql. don't change mysql table structure !
find a way , can change SQL to below when use elasticsearch-jdbc tool index data to elasticsearch from mysql , don't change mysql table structure !
elasticsearch-jdbc-2.3.3.1/bin/mysql-xxx.sh:
"statement": "select convert(bignum,CHAR) as bignum from xxx where xxxxx";
change to string type index to es
The fact is that database and JDBC know about big integer, Java knows big integer, but ES does not (and Javascript/JSON is silent about this). JDBC importer tries to convert to JSON numbers by using Java, and ES tries again to index a numeric value from the JSON number. JDBC importer tries by default to downgrade to numeric values to retain numbers. JDBC importer uses scale and precision defaults, they can be changed, but result must fit into Java double. If you index strings, all numeric properties are lost.
@jprante thx, if i index long type, -8629845578222121431 will change to -8629845578222121000 , but correct data is -8629845578222121431,the -8629845578222121000 is an error data. so too long type number can only change index Strings. you have good idea ?