jorgetavares
Any Chance We'll See External Sorts?
That was in my initial plans, to have some kind of external sort. I haven't worked in this library for quite some time due to multiple factors. Now, I could consider work on this if there is an actual demand for it (i.e., we really want to use it, not just glad to have it), otherwise I'm afraid I don't have much time. However, if someone also wants to add it, that's also great!
Well I think demand is a bit of a chicken and egg problem: if the capability were there, so would be demand. In my case, I do need to sort some very large vectors (on the order of terabytes or more), so external sorting is a must. This will almost certainly have to be a distributed sort, but as a start getting it working on large datasets on a single machine will be a good start. If the basics of external sort were there, I¹d be happy to extend it to shared-nothing architectures, but the problem, as it often is, is ramp-up time. It¹s difficult to justify ramp-up to add external sort to zsort when other libraries could do the sorting (Hive, Pig, etc.)
Perhaps if you can get some kind of basic structure in place, since you know the code, I can help fill in the details.
From: Jorge Tavares [email protected] Reply-To: jorgetavares/zsort <reply+0076a4f07699b62ad332ccee501d1df130c1255460f4fb1c92cf00000001107f5f7b9 [email protected]> Date: Saturday, November 15, 2014 at 10:37 To: jorgetavares/zsort [email protected] Cc: Steve Nunez [email protected] Subject: Re: [zsort] Any Chance We'll See External Sorts? (#1)
That was in my initial plans, to have some kind of external sort. I haven't worked in this library for quite some time due to multiple factors. Now, I could consider work on this if there is an actual demand for it (i.e., we really want to use it, not just glad to have it), otherwise I'm afraid I don't have much time. However, if someone also wants to add it, that's also great!
‹ Reply to this email directly or view it on GitHub https://github.com/jorgetavares/zsort/issues/1#issuecomment-63182949 .
CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Fair enough. I will try this week to take a look and see what's the best way to add an external sort, even if it's just a basic thing, to zsort.
By the way, is there a CL implementation that is preferable to use? In spite of trying to make things as portable as possible, sometimes there are issues. I assume you prefer/use SBCL but in the event of something else, let me know so that I use that particular implementation.
And thanks for your interest in zsort!