PowerAligner() Takes too much RAM on large dataset

[taufique74]

I was using PowerAligner class to align roughly 30 million parallel sentences with multiprocessing. But it takes too much RAM. Any hint on how to reduce memory consumption? @NickRuiz

[NickRuiz]

I haven't worked on this in a while but I'm sure there are many opportunities to optimize the code. If anyone wants to work on this, let me know and I can guide you through the logic.

On Wed, Oct 19, 2022 at 5:17 AM Taufiquzzaman Peyash < @.***> wrote:

I was using PowerAligner class to align roughly 30 million parallel sentences with multiprocessing. But it takes too much RAM. Any hint on how to reduce memory consumption? @NickRuiz https://github.com/NickRuiz

— Reply to this email directly, view it on GitHub https://github.com/NickRuiz/power-asr/issues/5, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACUWSIHKLQLUPPHAHDOHPTWD64BNANCNFSM6AAAAAARI46EHU . You are receiving this because you were mentioned.Message ID: @.***>