[Feat] Better dev experience when accessing stream results on /crawl

[calebpeffer]

Currently, the stream on crawl sends back information in the partial data array, which is limited to 50 items according to the docs. The newer items replace older items in partial data.

@evanboyle suggested we give him a "cursor" basically one that allows him to paginate which part of the stream he can access. Another option would be an interator (call .next()) to get the data)

Whatever approach is best, we can improve the devx here.