Python utf-8 read file uses a lot of memory

When I add utf-8 encoding, it uses about 3000 MB of memory (and freezes my computer) exmaple below works fine,

pool = ThreadPoolExecutor(max_workers=10)
with open("data.txt") as f:
        for line in f:
            pool.submit(thread, line)
pool.shutdown(wait=True)

but when I add utf-8 encoding, it basically causes my PC to crash

pool = ThreadPoolExecutor(max_workers=10)
with open("data.txt", encoding='utf-8') as f:
        for line in f:
            pool.submit(thread, line)
pool.shutdown(wait=True)
+4
source share

Source: https://habr.com/ru/post/1676362/


All Articles