Discussion: Grabbing and Saving Blocks Batch?

[gateway]

Hi, I'm thinking of using this block explore for a Go project I want to work on. After a few hours of playing around with this I kept hitting the a api limit which I understand.

I was looking at grabber.js and started to think of potential ways to batch up requests.

So currently we know a block is generated every 5 seconds give or take. Thats around 12 blocks a second which is pretty amazing in itself.

I haven't messed around with web3 much but I know they have a concept of batching (which my or may not work in this case), however if we could just call the api once very minute to retrieve the x amount of blocks (12) instead of watching for a block, grabbing it, writing it etc.

This could eliminate load on both the api and the servers running the explorer code. Mongo also has the concept of batch inserts as well. I'm not sure what whats going on in the api layer I would assume it wouldn't be hitting a mongo a database all the time? If blocks where done in batches the batches could be stored in redis as a key (block batch #) (value all the json data).

Anyhow Im just thinking out loud and maybe this approach may or may not work.

• Adjust listen for blocks function to grab 12 blocks, wait x time and grab the next 12 blocks
• If the api endpoint had the notion or code to have block batches the listen for blocks would be listen for batches
• Once data is returned from the api in batches mongo write that data as batch to the database.
• If someone tries to access or refresh a the front page for blocks then skip that and do things like normal.

I personally don't think anyone wants to stare at blocks flying by in a web browser, its pretty annoying and really doesn't give you any data. The front page could have a 1 min or 2 min refresh to return the last 12 or 24 blocks.

I think this could lower the rate of api calls and the load on the servers running the explorer. Also the api could be split into real time requests and some sort of redis in memory cluster that has data in batches.

thoughts?