datasist
datasist copied to clipboard
risenW
Add advanced functionality to handle processing of big datasets
Adding pyspark module to be triggered in the event that the size of input dataset is bigger than 1mil rows. This capability should improve the runtime generally and give improved user experience.
Krisblarq, do you want to start writing the test cases, how do you think we can collaborate? Rising, I imagine that you'll like to retain existing code structure, which could mean that the pyspark code will be maintained as a separate module? If we agree on the structure, the development can be accelerated.
Very Best!
Get Outlook for Androidhttps://aka.ms/ghei36
From: Rising Odegua [email protected] Sent: Saturday, August 1, 2020 9:24:56 AM To: risenW/datasist [email protected] Cc: Iretioluwa Olawuyi [email protected]; Author [email protected] Subject: Re: [risenW/datasist] Add advanced functionality to handle processing of big datasets (#55)
Sure, please go ahead
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/risenW/datasist/issues/55#issuecomment-667494604, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGC2IF6RTXAWW6MRM3VATATR6PGNRANCNFSM4OZXKWRA.
Yes @iretex the module should be separate. Then it can be called from other modules when we intend to use it. This should be invisible to the user as well.
Very clear!
From: Rising Odegua [email protected] Sent: Saturday, August 1, 2020 1:31:56 PM To: risenW/datasist [email protected] Cc: Iretioluwa Olawuyi [email protected]; Mention [email protected] Subject: Re: [risenW/datasist] Add advanced functionality to handle processing of big datasets (#55)
Yes @iretexhttps://github.com/iretex the module should be separate. Then it can be called from other modules when we intend to use it. This should be invisible to the user as well.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/risenW/datasist/issues/55#issuecomment-667524811, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGC2IFZYRSCOOMDXIGENNKDR6QDLZANCNFSM4OZXKWRA.