crecto
crecto copied to clipboard
Crecto
'Repo.insert' throw 'IndexError' sometimes
Sometimes simple insert query result is empty array [],
though all data were inserted to table correctly.
Index out of bounds (IndexError)
0x10a325535: *CallStack::new:CallStack at ??
0x10a32848e: *raise<IndexError>:NoReturn at ??
https://github.com/Crecto/crecto/blob/master/src/crecto/repo.cr#L184
The error is coming from first.
query = config.adapter.run_on_instance(tx || config.get_connection, :insert, changeset)
...
new_instance = changeset.instance.class.from_rs(query.as(DB::ResultSet)).first
I couldn't find any pattern, it looks like randomly occur.
mysql-5.6
@fridgerator I'm still experience IndexError on master. And "Unexpected byte 0xff in UTF-8 byte sequence" as well. I don't know how to debug this, I'm new in Crystal.
You get the unexpected byte sequence only on insert statements with mysql?
@fridgerator I'm experience this with postgresql. I'm parsing json and then insert changeset. Only 3 records will be saved out of 6, it will stuck after Unexpected byte error. And DB log will be only the first record.
1.77ms INSERT INTO tickers (market, currency, ticker, price, bid, ask, low_24h, high_24h, avg_24h, volume_24h, volume_30d) VALUES ('coinone', 'bch', 'KRW', '360500.0', '', '', '350100.0', '369500.0', '', '34075.8113', '') RETURNING *
Very confusing. I'm already successfully record other records.
With insert
#<Crecto::Changeset::Changeset(Ticker):0x10c569000 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "bch", :ticker => "KRW", :price => 360500.0, :bid => nil, :ask => nil, :low_24h => 350100.0, :high_24h => 369500.0, :avg_24h => nil, :volume_24h => 34075.1407, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "bch"}, {:ticker => "KRW"}, {:price => 360500.0}, {:low_24h => 350100.0}, {:high_24h => 369500.0}, {:volume_24h => 34075.1407}], @valid=true, @instance=#<Ticker:0x10ee11f20 @initial_values=nil, @market="coinone", @currency="bch", @ticker="KRW", @price=360500.0, @bid=nil, @ask=nil, @low_24h=350100.0, @high_24h=369500.0, @avg_24h=nil, @volume_24h=34075.1407, @volume_30d=nil>, @initial_values={}>
insert error - Index out of bounds
#<Crecto::Changeset::Changeset(Ticker):0x10ee23900 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "qtum", :ticker => "KRW", :price => 12490.0, :bid => nil, :ask => nil, :low_24h => 11990.0, :high_24h => 13440.0, :avg_24h => nil, :volume_24h => 1920171.575, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "qtum"}, {:ticker => "KRW"}, {:price => 12490.0}, {:low_24h => 11990.0}, {:high_24h => 13440.0}, {:volume_24h => 1920171.575}], @valid=true, @instance=#<Ticker:0x10ee11dc0 @initial_values=nil, @market="coinone", @currency="qtum", @ticker="KRW", @price=12490.0, @bid=nil, @ask=nil, @low_24h=11990.0, @high_24h=13440.0, @avg_24h=nil, @volume_24h=1920171.575, @volume_30d=nil>, @initial_values={}>
insert error - Unexpected byte 0xff in UTF-8 byte sequence
#<Crecto::Changeset::Changeset(Ticker):0x10ee294e0 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "etc", :ticker => "KRW", :price => 13910.0, :bid => nil, :ask => nil, :low_24h => 13260.0, :high_24h => 14170.0, :avg_24h => nil, :volume_24h => 1039445.4814, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "etc"}, {:ticker => "KRW"}, {:price => 13910.0}, {:low_24h => 13260.0}, {:high_24h => 14170.0}, {:volume_24h => 1039445.4814}], @valid=true, @instance=#<Ticker:0x10ee11d10 @initial_values=nil, @market="coinone", @currency="etc", @ticker="KRW", @price=13910.0, @bid=nil, @ask=nil, @low_24h=13260.0, @high_24h=14170.0, @avg_24h=nil, @volume_24h=1039445.4814, @volume_30d=nil>, @initial_values={}>
Without insert
#<Crecto::Changeset::Changeset(Ticker):0x10e76b000 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "bch", :ticker => "KRW", :price => 360500.0, :bid => nil, :ask => nil, :low_24h => 350100.0, :high_24h => 369500.0, :avg_24h => nil, :volume_24h => 34075.1407, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "bch"}, {:ticker => "KRW"}, {:price => 360500.0}, {:low_24h => 350100.0}, {:high_24h => 369500.0}, {:volume_24h => 34075.1407}], @valid=true, @instance=#<Ticker:0x111013f20 @initial_values=nil, @market="coinone", @currency="bch", @ticker="KRW", @price=360500.0, @bid=nil, @ask=nil, @low_24h=350100.0, @high_24h=369500.0, @avg_24h=nil, @volume_24h=34075.1407, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x111014900 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "qtum", :ticker => "KRW", :price => 12490.0, :bid => nil, :ask => nil, :low_24h => 11990.0, :high_24h => 13440.0, :avg_24h => nil, :volume_24h => 1920056.2903, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "qtum"}, {:ticker => "KRW"}, {:price => 12490.0}, {:low_24h => 11990.0}, {:high_24h => 13440.0}, {:volume_24h => 1920056.2903}], @valid=true, @instance=#<Ticker:0x111013e70 @initial_values=nil, @market="coinone", @currency="qtum", @ticker="KRW", @price=12490.0, @bid=nil, @ask=nil, @low_24h=11990.0, @high_24h=13440.0, @avg_24h=nil, @volume_24h=1920056.2903, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x1110142a0 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "etc", :ticker => "KRW", :price => 13910.0, :bid => nil, :ask => nil, :low_24h => 13260.0, :high_24h => 14170.0, :avg_24h => nil, :volume_24h => 1039445.4814, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "etc"}, {:ticker => "KRW"}, {:price => 13910.0}, {:low_24h => 13260.0}, {:high_24h => 14170.0}, {:volume_24h => 1039445.4814}], @valid=true, @instance=#<Ticker:0x111013dc0 @initial_values=nil, @market="coinone", @currency="etc", @ticker="KRW", @price=13910.0, @bid=nil, @ask=nil, @low_24h=13260.0, @high_24h=14170.0, @avg_24h=nil, @volume_24h=1039445.4814, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x111018c00 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "btc", :ticker => "KRW", :price => 6445000.0, :bid => nil, :ask => nil, :low_24h => 6080500.0, :high_24h => 6989500.0, :avg_24h => nil, :volume_24h => 18770.8274, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "btc"}, {:ticker => "KRW"}, {:price => 6445000.0}, {:low_24h => 6080500.0}, {:high_24h => 6989500.0}, {:volume_24h => 18770.8274}], @valid=true, @instance=#<Ticker:0x111013d10 @initial_values=nil, @market="coinone", @currency="btc", @ticker="KRW", @price=6445000.0, @bid=nil, @ask=nil, @low_24h=6080500.0, @high_24h=6989500.0, @avg_24h=nil, @volume_24h=18770.8274, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x1110185a0 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "eth", :ticker => "KRW", :price => 386950.0, :bid => nil, :ask => nil, :low_24h => 349250.0, :high_24h => 395900.0, :avg_24h => nil, :volume_24h => 282213.7917, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "eth"}, {:ticker => "KRW"}, {:price => 386950.0}, {:low_24h => 349250.0}, {:high_24h => 395900.0}, {:volume_24h => 282213.7917}], @valid=true, @instance=#<Ticker:0x111013c60 @initial_values=nil, @market="coinone", @currency="eth", @ticker="KRW", @price=386950.0, @bid=nil, @ask=nil, @low_24h=349250.0, @high_24h=395900.0, @avg_24h=nil, @volume_24h=282213.7917, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x11101cf00 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "xrp", :ticker => "KRW", :price => 294.0, :bid => nil, :ask => nil, :low_24h => 271.0, :high_24h => 298.0, :avg_24h => nil, :volume_24h => 170112858.829, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "xrp"}, {:ticker => "KRW"}, {:price => 294.0}, {:low_24h => 271.0}, {:high_24h => 298.0}, {:volume_24h => 170112858.829}], @valid=true, @instance=#<Ticker:0x111013bb0 @initial_values=nil, @market="coinone", @currency="xrp", @ticker="KRW", @price=294.0, @bid=nil, @ask=nil, @low_24h=271.0, @high_24h=298.0, @avg_24h=nil, @volume_24h=170112858.829, @volume_30d=nil>, @initial_values={}>
I found that if I replace nil values with values then all 6 will be recorded. But rescue will print out
PG::ResultSet#read returned a PG::Numeric. A (Float64 | Nil) was expected.
begin
Repo.insert(changeset)
rescue ex
puts "insert error - #{ex}"
end
@onemanstartup are you able to share your schema and json that you are parsing? I tried replicating locally and couldn't
@fridgerator This is the code. I don't understand why records persists but throw errors.. strange..
class Ticker < Crecto::Model
schema "tickers", primary_key: false do
field :market, String
field :currency, String
field :ticker, String
field :price, Float64
field :bid, Float64
field :ask, Float64
field :low_24h, Float64
field :high_24h, Float64
field :avg_24h, Float64
field :volume_24h, Float64
field :volume_30d, Float64
set_created_at_field nil
set_updated_at_field nil
end
validate_required [:market]
validate_format :market, /^[a-zA-Z]*$/
end
CREATE TABLE tickers (
time TIMESTAMPTZ NOT NULL DEFAULT now(),
market TEXT NOT NULL,
currency TEXT NOT NULL,
ticker TEXT NOT NULL,
price numeric NOT NULL,
bid numeric NULL,
ask numeric NULL,
low_24h numeric NULL,
high_24h numeric NULL,
avg_24h numeric NULL,
volume_24h numeric NULL,
volume_30d numeric NULL
);
{"bch":{"volume":"214659.4839","last":"418000","yesterday_last":"352400","yesterday_low":"348000","high":"451500","currency":"bch","low":"349700","yesterday_first":"353500","yesterday_volume":"12689.9568","yesterday_high":"360300","first":"351400"},"qtum":{"volume":"2417040.9742","last":"12830","yesterday_last":"11960","yesterday_low":"11790","high":"13200","currency":"qtum","low":"11950","yesterday_first":"12060","yesterday_volume":"657970.4149","yesterday_high":"12360","first":"11980"},"errorCode":"0","etc":{"volume":"983336.4399","last":"13700","yesterday_last":"13430","yesterday_low":"13230","high":"14050","currency":"etc","low":"13260","yesterday_first":"13470","yesterday_volume":"477616.7548","yesterday_high":"13690","first":"13440"},"result":"success","btc":{"volume":"4240.6907","last":"6372000","yesterday_last":"6347000","yesterday_low":"6200000","high":"6486500","currency":"btc","low":"6300000","yesterday_first":"6332500","yesterday_volume":"5541.9419","yesterday_high":"6499500","first":"6375000"},"timestamp":"1508249874","eth":{"volume":"83368.7347","last":"367100","yesterday_last":"380400","yesterday_low":"368500","high":"383000","currency":"eth","low":"365000","yesterday_first":"375000","yesterday_volume":"165312.9340","yesterday_high":"394900","first":"381400"},"xrp":{"volume":"306820375.5870","last":"288","yesterday_last":"319","yesterday_low":"293","high":"323","currency":"xrp","low":"278","yesterday_first":"296","yesterday_volume":"334760612.5759","yesterday_high":"327","first":"323"}}
JSON.parse(response.body).each do |k, v|
next if k == "errorCode" || k == "result" || k == "timestamp"
ticker = Mapping::Ticker.from_json(v.to_json)
new_ticker = Ticker.new
new_ticker.market = @market
new_ticker.currency = k.to_s
new_ticker.ticker = "KRW"
new_ticker.price = ticker.last.to_f
new_ticker.low_24h = ticker.low.to_f
new_ticker.high_24h = ticker.high.to_f
new_ticker.volume_24h = ticker.volume.to_f
new_ticker.avg_24h = 0.to_f
new_ticker.volume_30d = 0.to_f
new_ticker.bid = 0.to_f
new_ticker.ask = 0.to_f
changeset = Ticker.changeset(new_ticker)
if changeset.valid?
begin
Repo.insert(changeset)
rescue ex
puts "insert error - #{ex}"
end
else
puts changeset.errors
end
end
@fridgerator I tried set types to PG::Numeric in Crecto::Model, but it's doesn't work. It seems either I don't understand how casting works in pg shard or crystal lacks decimal field leads to this. Anyway thanks for helping out, will save in floats for now, until figure out how this works.
yeah you'll have to use float right now, PG::Numeric wont work in the model right now. I'll add an issue to support this type. Thanks!
I'm still getting this error with Crecto 0.8.4. Here's my Postgres schema:
CREATE TABLE foods(
id SERIAL PRIMARY KEY,
upc VARCHAR,
usda_db_id VARCHAR,
name VARCHAR NOT NULL,
description VARCHAR,
protein FLOAT,
fat FLOAT,
carbohydrates FLOAT,
serving_unit VARCHAR,
serving_label VARCHAR,
serving_value INTEGER,
created_at TIMESTAMP,
updated_at TIMESTAMP
);
Looks like it was bad permissions settings on my part. Solved by making my dev postgres user a superuser with:
ALTER USER postgres WITH SUPERUSER;
Nevermind, still get the IndexError when using Repo.get_by/2 or Repo.insert/1
@timkendall Does this happen all the time or randomly? It would be great to get a log of the sql generated
All of the time - I'll add that when I get back to my personal computer.
@timkendall Creating a table with withe sql you posted, the following is working for me (crystal 0.24.2, crecto master branch)
require "pg"
require "./src/crecto"
Crecto::DbLogger.set_handler(STDOUT)
module Repo
extend Crecto::Repo
config do |conf|
conf.adapter = Crecto::Adapters::Postgres
conf.database = "issue_95"
conf.hostname = "localhost"
conf.port = 5432
end
end
class Food < Crecto::Model
schema "foods" do
field :upc, String
field :usda_db_id, String
field :name, String
field :description, String
field :protein, Float64
field :fat, Float64
field :carbohydrates, Float64
field :serving_unit, String
field :serving_label, String
field :serving_value, Int32
end
end
food = Food.new
food.upc = "123"
food.usda_db_id = "456"
food.name = "hey"
food.description = "description"
food.protein = 342.222
food.fat = 123.234
food.carbohydrates = 54.234
food.serving_unit = "g"
food.serving_label = "good food"
food.serving_value = 123
changeset = Repo.insert(food)
puts Repo.get_by(Food, id: changeset.instance.id)
@timkendall did that fix end up working for you? Or can you let me know if you're still getting this error? I know this is an old issue.
