elasticsearch-jdbc icon indicating copy to clipboard operation
elasticsearch-jdbc copied to clipboard

mysql bigint field inserto to es , find error data

Open LockGit opened this issue 8 years ago • 3 comments

example: CREATE TABLE product_info ( item_id bigint(20) NOT NULL DEFAULT '0' , ); whene item_id too long , use jdbc insert es from mysql will error;

also, we can test on sense ` PUT lock/test/5 { "name":"del_5", "age":20, "num":"-8629845578222121431", "bignum":-8629845578222121431 }

GET lock/test/_search { "query": { "term": { "name": { "value": "del_5" } } } }

result: { "took": 1, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 1, "max_score": 0.15342641, "hits": [ { "_index": "lock", "_type": "test", "_id": "5", "_score": 0.15342641, "_source": { "name": "del_5", "age": 20, "num": "-8629845578222121431", "bignum": -8629845578222121000 } } ] } } ` we can see bignum change to 8629845578222121000 , rather than 8629845578222121431

it's precision question .

how too fix this question when use elasticsearch-jdbc tool index data to elasticsearch from mysql. don't change mysql table structure !

LockGit avatar Aug 11 '17 03:08 LockGit

find a way , can change SQL to below when use elasticsearch-jdbc tool index data to elasticsearch from mysql , don't change mysql table structure !

elasticsearch-jdbc-2.3.3.1/bin/mysql-xxx.sh:

"statement": "select convert(bignum,CHAR) as bignum from xxx where xxxxx";

change to string type index to es

LockGit avatar Aug 11 '17 09:08 LockGit

The fact is that database and JDBC know about big integer, Java knows big integer, but ES does not (and Javascript/JSON is silent about this). JDBC importer tries to convert to JSON numbers by using Java, and ES tries again to index a numeric value from the JSON number. JDBC importer tries by default to downgrade to numeric values to retain numbers. JDBC importer uses scale and precision defaults, they can be changed, but result must fit into Java double. If you index strings, all numeric properties are lost.

jprante avatar Aug 11 '17 10:08 jprante

@jprante thx, if i index long type, -8629845578222121431 will change to -8629845578222121000 , but correct data is -8629845578222121431,the -8629845578222121000 is an error data. so too long type number can only change index Strings. you have good idea ?

LockGit avatar Aug 11 '17 13:08 LockGit