Prepend functionality

[iseletsk]

I am not sure you would want to merge this in - this adds a prepend functionality that I need for another project. I am sure it is not the best way to do prepend - but I expect the operation to be really infrequent. I totally understand if you don't want it as part of your dque package as it breaks size functionality, and some of the "expectations"

1. All objects will be added into a new single segment, and that segment will be the first segment (so, don't prepend too much in one goo)
2. Size will provide incorrect output

[joncrlsn]

Igor, Thank you for your PR. What is your use case? Is it so you can take something off the queue, then add it back near the front of the queue if you are not able to successfully handle it?

[iseletsk]

Thank you Jon,

That is exactly what I am doing. I might take 10 items of the queue, and then have to put some of them back in front of the queue because they "weren't processed".

On Thu, Nov 5, 2020, 17:13 Jon Carlson [email protected] wrote:

Igor, Thank you for your PR. What is your use case? Is it so you can take something off the queue, then add it back near the front of the queue if you are not able to successfully handle it?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joncrlsn/dque/pull/25#issuecomment-722742437, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7CWPZ7YFY2LNI6BVVCLSDSONESTANCNFSM4TH6XIKA .

[joncrlsn]

I like the idea, and I think it is needed. It is a better idea than locking the queue while you process the item.

However, for the implementation I'd rather see is a qSegment (with number zero) that contains all the prepended items that the Dequeue method would check first. Then the Size() method could be made to work right. That would take some more coding to clean it out occasionally, which I could help with.

[iseletsk]

The "zero" segment would make the Size() method work, and I do like that idea from a cleanliness perspective. I am worried a bit checking two segments vs one for each read - just to verify that segment 0 is empty, but I guess it can be done under a single lock, and wouldn't be as expensive.

What I cannot figure out is how to handle two prepands without major code rewrite on handling segments.

Here is an example: I read from the queue 1,2,3,4,5. I prepend the numbers, and now the zero segment has 1,2,3,4,5 in that order. I read 1,2,3, zero segment now has 4,5 I need to prepend again, but to put things back in 1,2,3,4,5 order I would need to redo the whole segment.

I mean - given that it is a rare operation, and would be done fully in-memory -- it might be worth it. What do you think?

On Thu, Nov 5, 2020 at 7:06 PM Jon Carlson [email protected] wrote:

I like the idea, and I think it is needed. It is a better idea than locking the queue while you process the item.

However, for the implementation I'd rather see a qSegment (with number zero) that contains all the prepended items that the Dequeue method would check first. Then the Size() method could be made to work right. That would take some more coding to clean it out occasionally, which I could help with.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joncrlsn/dque/pull/25#issuecomment-722781254, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7CWP4GIHV2DJ37S3SIT6DSONR3FANCNFSM4TH6XIKA .

[joncrlsn]

You could require people to enable the prepend functionality. That would eliminate the two reads for the people who don't use prepend.

Good thought. I'm wondering how important it is to put 1, 2, and 3 back in the right order. If you weren't able to handle it the first time (i.e. database unavailable), it's unlikely that an immediate second try would work either. In that case, putting it back to the end of the zero queue might be a good thing.

But I'm thinking it is going to be necessary to prepend with a wait period so you don't end up in an infinite loop handling things that not are working, while never getting to the new work in queues 1+.

[iseletsk]

In my case the prepand is not instant, nor infinite loop is possible. On the other hand - nor I have a hard rule of putting 1,2,3 in exact order ... as long as they are not going to the end of the queue, I should be fine.

Yet, something rubs me the wrong way in "unpredictability" of order, and double reads.

What if instead of doing zero segments, we would:

Take create a new segment, and fill it from some of the items from prepend & current first segment (in proper order), and make it precisely the max segment size Note that this will be in-memory operation (but depending on segment size, might be fairly large write). and create one before (if needed) that will be the new first segment.

Then whatever is left from prepend - would go in the new first segment. It will be expensive, but it will preserve the queue, order, size.

On Sun, Nov 8, 2020 at 5:22 PM Jon Carlson [email protected] wrote:

You could require people to enable the prepend functionality. That would eliminate the two reads for the people who don't use prepend.

Good thought. I'm wondering how important it is to put 1, 2, and 3 back in the right order. If you weren't able to handle it the first time (i.e. database unavailable), it's unlikely that an immediate second try would work either. In that case, putting it back to the end of the zero queue might be a good thing.

But I'm thinking it is going to be necessary to prepend with a wait period so you don't end up in an infinite loop handling things that not are working, while never getting to the new work in queues 1+.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joncrlsn/dque/pull/25#issuecomment-723702519, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7CWP6ZPCATAPXNKLDZNZTSO473TANCNFSM4TH6XIKA .

[joncrlsn]

Hi Igor, I've started thinking that new functionality like this should be in a wrapper/decorator object that enhances the current queue object without changing it. That would allow different features based on which wrapper/decorator you choose.

I would add links to the README documentation for projects that are wrapper/decorator objects.

https://en.wikipedia.org/wiki/Decorator_pattern

Thoughts?

[iseletsk]

Hi Jon,

Go is new for me, but I did work with Decorators a lot in python. Few things I don't get:

1. Which method would I decorate/wrap to implement Prepend?
2. I would not get access to unexported struct fields like mutex, fileLock, firstSegment, as well as unexported functions like newQueueSegment., or firstSegement.add/close

Or am I missing something?

On Tue, Nov 10, 2020 at 7:28 PM Jon Carlson [email protected] wrote:

Hi Igor, I've started thinking that new functionality like this should be in a wrapper/decorator object that enhances the current queue object without changing it. That would allow different features based on which wrapper/decorator you choose.

I would add links to the README file for projects that are wrapper/decorator objects.

https://en.wikipedia.org/wiki/Decorator_pattern

Thoughts?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joncrlsn/dque/pull/25#issuecomment-725119855, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7CWP2LD352EWPS7ZRGTJ3SPIAFJANCNFSM4TH6XIKA .

[joncrlsn]

My experience with OO is from Smalltalk and Java, so hopefully my terms will match yours enough.

You are right that it would not be a true decorator because it will actually extend the API with a new Prepend method that is in your wrapper class, and pass the other methods through to the Dque instance.

We'd probably need to make the qSegment class/struct public/external (first letter uppercase) so you can use that.

You might also need to add some methods to the wrapped DQue class/instance/struct that would allow you to integrate with the internals better. But you should try to just use what you have first, even if that means using your own mutexes in the wrapper class (might work depending on your implementation). Any new methods on Dque would need to be public if your wrapper is in it's own package. But could be private (starts with lowercase) if we include the decorator in the dque package.

Let me know if I didn't explain everything as well as I could.

I hope that's not too hard, but if it is, and you need the functionality as soon as possible, you could fork the project (as you've done) and make the changes in your own project, and use that.

[joncrlsn]

Using composition to add features usually adds some effort, but it pays off in being easier to test and allowing multiple implementations you can choose from (depending on your use-case).

[iseletsk]

I think exposing enough of DQue so it can be extendable via composition would be great. The only negative side effect is that you would need to export more things, which would make changes on how internals work more difficult - without breaking "externals"

On Wed, Nov 11, 2020 at 7:15 AM Jon Carlson [email protected] wrote:

Using composition to add features usually adds some effort, but it pays off in being easier to test and allowing multiple implementations you can choose from (depending on your use-case).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joncrlsn/dque/pull/25#issuecomment-725478058, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7CWP35GQPYODBXUCCYVL3SPKS3HANCNFSM4TH6XIKA .