gorse

gorse copied to clipboard

Gorse version 0.5.0-alpha, but also happens in 0.4.14

Describe the bug Every one hour or so we're encountering the following panic:

panic: floats: slice lengths do not match

goroutine 104 [running]:
github.com/zhenghaoz/gorse/base/floats.Dot(...)
 /go/gorse/base/floats/floats.go:169
github.com/zhenghaoz/gorse/base/search.(*DenseVector).Distance(0x21f5e80?, {0x21f4b38?, 0xc06508c140?})
 /go/gorse/base/search/index.go:58 +0x2d4
github.com/zhenghaoz/gorse/base/search.(*HNSW).distance(0xc10dc3a0b0, {0x21f4b38, 0xc06508c140}, {0xc04a8d9574, 0x1, 0xc000509000?})
 /go/gorse/base/search/hnsw.go:302 +0xc2
github.com/zhenghaoz/gorse/base/search.(*HNSW).efSearch(0xc10dc3a0b0, {0x21f4b38, 0xc06508c140}, 0xa?)
 /go/gorse/base/search/hnsw.go:443 +0x4d
github.com/zhenghaoz/gorse/base/search.(*HNSW).MultiSearch(0xc10dc3a0b0, {0x21f4b38, 0xc06508c140}, {0xc0217ad880, 0x1b, 0x4a8d96d8?}, 0x105, 0x0)
 /go/gorse/base/search/hnsw.go:421 +0x1fe
github.com/zhenghaoz/gorse/worker.(*Worker).collaborativeRecommendHNSW(0xc00056ca80, 0xc0216404d0?, {0xc018fd8ed8, 0x8}, {0xc0217ad880, 0x1b, 0x1b}, {0x220c4a8, 0xc055206de0}, 0xc01d81e8c0)
 /go/gorse/worker/worker.go:878 +0x1e5
github.com/zhenghaoz/gorse/worker.(*Worker).Recommend.func2(0x0, 0x4e1)
 /go/gorse/worker/worker.go:597 +0xb91
github.com/zhenghaoz/gorse/base/parallel.Parallel(0x48892, 0xc021c38de0?, 0xc04a89c280)
 /go/gorse/base/parallel/parallel.go:40 +0xf7
github.com/zhenghaoz/gorse/worker.(*Worker).Recommend(0xc00056ca80, {0xc0584ec000, 0x48892, 0x4b238})
 /go/gorse/worker/worker.go:550 +0xccf
github.com/zhenghaoz/gorse/worker.(*Worker).Serve.func1()
 /go/gorse/worker/worker.go:425 +0x405
github.com/zhenghaoz/gorse/worker.(*Worker).Serve(0xc00056ca80)
 /go/gorse/worker/worker.go:437 +0xddf
main.glob..func1.1()
 /go/gorse/cmd/gorse-in-one/main.go:109 +0x3f
created by main.glob..func1 in goroutine 1
 /go/gorse/cmd/gorse-in-one/main.go:107 +0xb1d

To Reproduce I'm unfortunately not able to reproduce the bug reliably, it occurs randomly inside woker.go:

if time.Since(tick) < w.Config.Recommend.Offline.CheckRecommendPeriod {
	loop()
}

Additional context

• gorse-in-one Docker image
• MongoDB for cache and MySQL for item storage
• 360K items, 700K feedback, ~30 categories

Config

[master]
n_jobs = 16
meta_timeout = "10s"

[server]
default_n = 10
clock_error = "5s"
auto_insert_user = true
auto_insert_item = true
cache_expire = "10s"

[recommend]
cache_size = 250
cache_expire = "12h"
active_user_ttl = 1

[recommend.data_source]
positive_feedback_types = ["comment","like","favorite"]
read_feedback_types = ["read"]
positive_feedback_ttl = 7
item_ttl = 30

[recommend.popular]
popular_window = "24h"

[recommend.user_neighbors]
neighbor_type = "related"
enable_index = true
index_recall = 0.8
index_fit_epoch = 3

[recommend.item_neighbors]
neighbor_type = "related"
enable_index = true
index_recall = 0.8
index_fit_epoch = 3

[recommend.collaborative]
enable_index = true
index_recall = 0.9
index_fit_epoch = 3
model_fit_period = "60m"
model_search_period = "360m"
model_search_epoch = 100
model_search_trials = 10
enable_model_size_search = true

[recommend.replacement]
enable_replacement = false
positive_replacement_decay = 0.8
read_replacement_decay = 0.6

[recommend.offline]
check_recommend_period = "15m"
refresh_recommend_period = "24h"
enable_latest_recommend = true
enable_popular_recommend = true
enable_user_based_recommend = true
enable_item_based_recommend = true
enable_collaborative_recommend = true
enable_click_through_prediction = true
explore_recommend = { popular = 0.2, latest = 0.1 }

[recommend.online]
fallback_recommend = ["item_based", "popular"]
num_feedback_fallback_item_based = 10

[tracing]
enable_tracing = false
exporter = "jaeger"
collector_endpoint = "http://localhost:14268/api/traces"
sampler = "always"
ratio = 1

Just ran into this today on latest main:

This is after running for a few hours. Not sure exactly what conditions would trigger this.

panic: floats: slice lengths do not match

goroutine 101 [running]:
github.com/zhenghaoz/gorse/common/floats.Dot({0x140013f5b00, 0x10, 0x10}, {0x1402fdc0300, 0x20, 0x20})
	/Users/andre/code/gorse/common/floats/floats.go:180 +0xa0
github.com/zhenghaoz/gorse/common/ann.(*HNSW[...]).distance(0x103ab95a0, {0x1402fdc0300, 0x20, 0x20}, {0x140042ae2cc, 0x1, 0x1})
	/Users/andre/code/gorse/common/ann/hnsw.go:229 +0xf4
github.com/zhenghaoz/gorse/common/ann.(*HNSW[...]).knnSearch(0x103ab95a0, {0x1402fdc0300, 0x20, 0x20}, 0x69, 0x69)
	/Users/andre/code/gorse/common/ann/hnsw.go:93 +0xac
github.com/zhenghaoz/gorse/common/ann.(*HNSW[...]).SearchVector(0x103ab95a0, {0x1402fdc0300, 0x20, 0x20}, 0x69, 0x0)
	/Users/andre/code/gorse/common/ann/hnsw.go:79 +0x8c
github.com/zhenghaoz/gorse/logics.(*MatrixFactorization).Search(0x1400b578440, {0x1402fdc0300, 0x20, 0x20}, 0x69)
	/Users/andre/code/gorse/logics/cf.go:58 +0x6c
github.com/zhenghaoz/gorse/worker.(*Worker).collaborativeRecommendHNSW(0x140000d4340, {0x103ab7368, 0x140202db440}, 0x1400b578440, {0x14003682798, 0x18}, {0x103abb010, 0x14013721700}, 0x14025b80c08)
	/Users/andre/code/gorse/worker/worker.go:901 +0x158
github.com/zhenghaoz/gorse/worker.(*Worker).Recommend.func2(0x0, 0x1)
	/Users/andre/code/gorse/worker/worker.go:653 +0xd94
github.com/zhenghaoz/gorse/common/parallel.Parallel(0x5e56, 0x1, 0x140000fd970)
	/Users/andre/code/gorse/common/parallel/parallel.go:40 +0x3bc
github.com/zhenghaoz/gorse/worker.(*Worker).Recommend(0x140000d4340, {0x14033842000, 0x5e56, 0x7471})
	/Users/andre/code/gorse/worker/worker.go:607 +0xff8
github.com/zhenghaoz/gorse/worker.(*Worker).Serve.func1()
	/Users/andre/code/gorse/worker/worker.go:486 +0x47c
github.com/zhenghaoz/gorse/worker.(*Worker).Serve(0x140000d4340)
	/Users/andre/code/gorse/worker/worker.go:498 +0x1348
main.init.func1.1()
	/Users/andre/code/gorse/cmd/gorse-in-one/main.go:109 +0x44
created by main.init.func1 in goroutine 1
	/Users/andre/code/gorse/cmd/gorse-in-one/main.go:107 +0xd88