How can I reduce memory consumption when processing millions of records in Python?

[codehaven175]

?what is your answer?

[talentUnion]

let's keep in touch via discord

[mayar-ariss]

Hard to be specific without knowing the data format or what you're computing.... don't load all the records at once (stream them with a generator or read in chunks), and keep only small running results in memory.
something like:

with open("data.txt") as f:
    total = sum(len(line) for line in f)  # one line in memory at a time

memory then stays flat no matter how many records there are, see: https://docs.python.org/3/howto/functional.html#generators