analytics-ruby
analytics-ruby copied to clipboard
[Idea] Avoid silent failures or make more noise when maximum JSON message size reached
Hi there, I recently encountered a rather nasty error which has had some costly effects in terms of lost data. Essentially, our team discovered we had a gradual data drift in our downstream data science / product dashboards causal to lost messages sent to mixpanel.
After a bit of investigation, I discovered that there is a maximum JSON serialization limit for JSON generation. One of the challenges for us was that we didn't realize the gem caught this error and piped out to logs. Our log pipelines are quite busy so its hard to catch minor errors like this.
I wonder if it might make sense to simply raise an error saying "Limit of message reached" when you try to track
an event with a message size > 32k, instead of logging and failing silently.
https://github.com/segmentio/analytics-ruby/blob/c440393016b13c39716610143a1c309668e48d68/lib/segment/analytics/defaults.rb#L22
https://github.com/segmentio/analytics-ruby/blob/c440393016b13c39716610143a1c309668e48d68/lib/segment/analytics/message_batch.rb#L24-L31