jschultz
Memory Leak
I am running a 64 bit machine with windows 10. I have 16gb ram.
I am converting a SQL Table that is ~ 40gb.
The program crashes everytime at ~ 9 gb.
You can watch the memory leak happen as the SQL Lite db is being created....there is a steady climb in memory until approximaly 80% ram is used...
The code needs to be refactored to manage this. Not sure of the solution as of yet.
I'm afraid I'm not using this code any longer since I discovered sqlalchemy allows you to write fast and friendly Python scripts to access MSSQL (plus SQLite, MySQL, PostgreSQL, etc.) databases using fast and friendly Python scripts.
But I presume there are memory leak detection tools along the lines of valgrind/memcheck that could help?
I guess not too many using this for larger systems. I ended up writing a different solution to take the output files, merge them..and use the sqllite3 command line to do the migration. Pretty simple and required really no code and did the 40gb in a few hours.