[Cylon] Arrow 14.0.2 Integration

[arupcsedu]

1. Add support for C++17
2. Update dependent library for conda
3. Update compatibility issue within arrow.

[cylon-manager]

@arupcsedu why not use arrow 15.0.x ?

UPDATE: never mind. I think latest cudf is at 14.x.x. So, this should be good

[arupcsedu]

cudf, pytorch default release version is compatible with 14.0.2

On Thu, Apr 18, 2024 at 12:12 PM cylon-manager @.***> wrote:

@arupcsedu https://github.com/arupcsedu why not use arrow 15.0.x ?

— Reply to this email directly, view it on GitHub https://github.com/cylondata/cylon/pull/688#issuecomment-2064397652, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOGXWR3PVXJAQ32YTAUVWTY57WGRAVCNFSM6AAAAABGNRPYHKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRUGM4TONRVGI . You are receiving this because you were mentioned.Message ID: @.***>

[laszewsk]

v 15 is a major new release, so i think you want to use 15

https://arrow.apache.org/release/15.0.0.html

[laszewsk]

is it that cudf, pytorch are not compatible with 15?

[arupcsedu]

right, Gregor. We should align with the released cudf and pytorch version. Arrow 14.0.2 comes with the released package of cudf and pytorch.

Despite this, it will not take that much time to upgrade to Arrow 15 after this upgrade. This one is the toughest one.

Best regards Arup Sarker

On Thu, Apr 18, 2024 at 12:45 PM Gregor von Laszewski < @.***> wrote:

is it that cudf, pytorch are not compatible with 15?

— Reply to this email directly, view it on GitHub https://github.com/cylondata/cylon/pull/688#issuecomment-2064517813, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOGXWWAUIX257MRGNRXYDDY572DXAVCNFSM6AAAAABGNRPYHKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRUGUYTOOBRGM . You are receiving this because you were mentioned.Message ID: @.***>

[mstaylor]

Arup,

Here are some initial suggestions:

1. Move the rivanna directory and update the necessary scripts under target
2. Change the conda environments to use cylon_dev as a name
3. The build completes successfully but there are many warnings that need to be cleaned up. We should have a build process that emits a minimal amount of warning output from the compiler

I'm testing with UCX now and therefore more suggestions may be forthcoming.

[arupcsedu]

I will do it. Thanks

On Sat, Aug 17, 2024 at 11:40 AM Mills Staylor @.***> wrote:

Arup,

Here are two suggestions:

1. move the rivanna directory and update the necessary scripts under target
2. change the conda environments to use cylon_dev as a name

I'm testing with UCX now and therefore more suggestions may be forthcoming.

— Reply to this email directly, view it on GitHub https://github.com/cylondata/cylon/pull/688#issuecomment-2294894347, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOGXWUYGB6T3DD2NNCAIMDZR5VHXAVCNFSM6AAAAABGNRPYHKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJUHA4TIMZUG4 . You are receiving this because you were mentioned.Message ID: @.***>