SwipeRefreshLayout breaks the ability to drag finger to the view

[Simolation]

I really like this library :) But I discovered an issue: I am using an swipeRefreshLayout around my recyclerview and when I am long pressing an item in this recyclerview, the popup shows up, but when I want to drag my finger down to the buttons of the view, the swipeRefreshLayout gets triggered and the popup closes. How can I disable the swipeRefreshLayout while the popup is open?