stacks-blockchain-api icon indicating copy to clipboard operation
stacks-blockchain-api copied to clipboard

feat: ensure events are inserted into the raw event request table

Open zone117x opened this issue 1 year ago • 2 comments

Issue found by @csgui where the event_observer_requests table can be missing events even if the rest of the event ingestion was successful.

This appears to be because the function for inserting into event_observer_requests was ran after the event server http request has been completed. So when a large block was received, it may have been ingested into the regular tables, but could run into an OOM error on the event_observer_requests insert.

This PR fixes the issue by ensuring the http request is not completed until insert into event_observer_requests is completed. Also the amount of JSON parsing/stringifying has been reduced in an attempt to reduce likelihood of an OOM when receiving a large /new_block event payload.

zone117x avatar Apr 05 '24 11:04 zone117x

Vercel deployment URL: https://stacks-blockchain-i7qr60ei6-blockstack.vercel.app :rocket:

github-actions[bot] avatar Apr 05 '24 11:04 github-actions[bot]

Codecov Report

Attention: Patch coverage is 85.18519% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 72.02%. Comparing base (f522d79) to head (e5299b9). Report is 49 commits behind head on master.

:exclamation: Current head e5299b9 differs from pull request most recent head cb4016d

Please upload reports for the commit cb4016d to get more accurate results.

Files Patch % Lines
src/event-stream/event-server.ts 86.36% 3 Missing :warning:
src/datastore/pg-write-store.ts 80.00% 1 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1925      +/-   ##
==========================================
- Coverage   72.07%   72.02%   -0.06%     
==========================================
  Files          93       93              
  Lines       12026    12027       +1     
  Branches     2654     2654              
==========================================
- Hits         8668     8662       -6     
- Misses       3202     3209       +7     
  Partials      156      156              

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Apr 05 '24 11:04 codecov[bot]

@zone117x Just validated this fix against an archive synced from genesis. Please check the conversation on this devops issue for more context.

Blocks containing a large number of transactions are now included in the archive file.

For example, block 137843, which was the root cause of this issue, is now present in the archive file after the fix. Therefore, for a block to be present in the archive file, it must have also been present in the event_observer_requests table.

Fix LGTM.

csgui avatar May 31 '24 17:05 csgui

Awesome, thanks for the thorough testing @csgui!

zone117x avatar May 31 '24 20:05 zone117x

:tada: This PR is included in version 7.11.0-beta.8 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

blockstack-devops avatar Jun 05 '24 12:06 blockstack-devops

:tada: This PR is included in version 7.11.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

blockstack-devops avatar Jun 07 '24 14:06 blockstack-devops