go-reddit icon indicating copy to clipboard operation
go-reddit copied to clipboard

Add Subreddit Comment Retrieval and Stream Concurrency

Open RGood opened this issue 3 years ago • 4 comments

This PR currently sits on top of comment-support as it builds on top of some of it.

Changelog:

  • Move the set{} struct to its own helper file and refactor it to OrderedMaxSet
    • The goal was to give it a maximum size while still checking for duplicates in the latest ids
    • Helper since it's more abstract functionality and less go-reddit specific
  • Refactor streams to a request-callback approach, so requests don't block each other
    • The purpose of this is to make requests on pace with the Ticker set for the stream and not have requests block each other
    • The OrderedMaxSet should handle de-duping request responses

Considerations:

  • Perhaps make the streams configurable to either make requests sync or async

RGood avatar Mar 08 '21 18:03 RGood

This PR should be ready for review / merge

RGood avatar Jun 10 '21 03:06 RGood

panic happens sometimes with this pr (may exist in original code)

panic: send on closed channel

goroutine 93559 [running]:
github.com/vartanbeno/go-reddit/v2/reddit.(*StreamService).Posts.func2.1(0xc000081800, 0x64, 0x80, 0x0, 0x0)
       go-reddit/reddit/stream.go:91 +0xc5
github.com/vartanbeno/go-reddit/v2/reddit.(*StreamService).getPosts(0xc000286288, 0x9bc961, 0x3, 0xc00089c000)
       go-reddit/reddit/stream.go:193 +0x106
created by github.com/vartanbeno/go-reddit/v2/reddit.(*StreamService).Posts.func2
       go-reddit/reddit/stream.go:60 +0x198

possible fix based on this PR: https://github.com/darren/go-reddit/commit/39267077688fc9cd37974d6420563191894a17b6

darren avatar Jul 29 '21 05:07 darren

@darren Would you mind making a PR against this branch in my fork so I can easily add your code / give you credit for it?

RGood avatar Aug 10 '21 23:08 RGood

@RGood my fix above is still buggy, panic occasionally:

panic: send on closed channel

goroutine 347827 [running]: github.com/vartanbeno/go-reddit/v2/reddit.(*StreamService).Comments.func2.1(0xc001434800, 0x64, 0x80, 0x0, 0x0) go-reddit/reddit/stream.go:196 +0x1a5 github.com/vartanbeno/go-reddit/v2/reddit.(*StreamService).getComments(0xc0000a0700, 0xe26c10, 0x3, 0xc000138180) go-reddit/reddit/stream.go:218 +0xe3 created by github.com/vartanbeno/go-reddit/v2/reddit.(*StreamService).Comments.func2 go-reddit/reddit/stream.go:161 +0x1a5

So using a quitChan is still tricky and not fixing the problem, I opened another PR https://github.com/RGood/go-reddit/pull/1 by using context

darren avatar Aug 11 '21 02:08 darren