Out of memory when using AWK

[fa1rid]

Hi, Using this command awk '!(count[$0]++)' old.txt > new.txt with huge dictionary is causing out of memory problem and command to be killed by the kernel.

[16346.370650] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-5.scope,task=awk,pid=17277,uid=0
[16346.370669] Out of memory: Killed process 17277 (awk) total-vm:31343560kB, anon-rss:31336728kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:61380kB oom_score_adj:0
[16347.841088] oom_reaper: reaped process 17277 (awk), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB