Memory leak and netCDF4

[fmaussion]

Hi, this is linux, python 3.5, latest numpy.

While profiling memory leaks in our software (which reads and writes a lot of NetCDF files), I have found what looks like memory leaks in netcdf4.

Here is a simple script to reproduce:

import netCDF4
import os
import psutil
import numpy as np
import matplotlib.pyplot as plt

process = psutil.Process(os.getpid())

f = 'dummy_data.nc'

if os.path.exists(f):
    os.remove(f)

with netCDF4.Dataset(f, 'w', format='NETCDF4') as nc:
    nc.createDimension('time', None)
    v = nc.createVariable('prcp', 'f4', ('time',))
    v[:] = np.arange(10000)
    v = nc.createVariable('temp', 'f4', ('time',))
    v[:] = np.arange(10000)


def dummy_read():
    with netCDF4.Dataset(f) as nc:
        prcp = nc.variables['prcp'][:]
        temp = nc.variables['temp'][:]


def dummy_numpy():
    prcp = np.arange(10000)
    temp = np.arange(10000)


out = []
for i in range(2000):
    dummy_read()
    out.append(process.memory_info().rss * 1e-6)

plt.plot(out)
plt.title(netCDF4.__version__)
plt.show()

The plots produced look like this with various netCDF4 versions:

Note that replacing dummy_read() with def dummy_numpy() (just creating some arrays) does not increase mem usage:

[fmaussion]

Sorry the plots in my original post did not show up. I corrected this now.

Note that the increase in memory usage (~5mb) is larger in 1.5.3 than 1.4.3 and 1.3.0 (~1.3mb).

cc @timoroth

[fmaussion]

Ups, sorry for not doing my homework, it seems related to https://github.com/Unidata/netcdf4-python/issues/986

The current Netcdf4 wheels still seem to ship with the old netcdf version though. We will try with an updated NetCDF and report back...

[jswhit]

Yes, please do report back when you have a chance to test with netcdf-c 4.7.4

[TimoRoth]

This is the result of running against netcdf-c 4.7.3 with https://github.com/Unidata/netcdf-c/commit/3bcdb5fbf11af85e63f7916768b13b1f7d02c860 backported. I could not use 4.7.4 because it made a major API and ABI change it seems, changing the soversion of the library. I made the patched library for Ubuntu 20.04 available via a PPA: https://launchpad.net/~btbn/+archive/ubuntu/netcdf

Edit: I massively increased the number of loops, and it seems like there is still a bit of leakage, though magnitudes less than in the unpatched versions, which might even just be the Python GC being unmotivated to act on such a low memory use.

Adding a gc.collect() before adding a final memory measurement did not show a drop in usage though.

[fmaussion]

Thanks @TimoRoth !

which might even just be the Python GC being unmotivated to act on such a low memory use. Adding a gc.collect() before adding a final memory measurement did not show a drop in usage though.

I don't know much about GC, but it sounds likely that something is leaking in C if the GC cannot pick it up...

[electricsam]

Is there any update on this? I have tracked down a memory leak in my application to some code in a third party library that reads a NetCDF file. I am using netCDF4-python v1.6.2.

[TimoRoth]

It's fixed in netcdf 4.7.4 and higher.

[electricsam]

@TimoRoth Sorry, I'm not a Python expert. I just used pip to install the Python library (pip install netCDF4). How do I check or change the netcdf-c version that comes with this?

In the meantime I managed to fork the thrid-party library I was using and refactored it to open the NetCDF file once, rather than opening and closing it in a loop. That workaround seems to work, but I would rather not have to do that.

[TimoRoth]

If you're using the wheels from pypi, I'd expect any of the last couple version to have something much more recent than that, so you don't have to do anything.

If you're building yourself, against your distros set of libraries, make sure you're using a modern enough distro, so it has the fixed version.

[electricsam]

All I did was this:

pip install netCDF4
Collecting netCDF4 Downloading netCDF4-1.6.2-cp39-cp39-macosx_11_0_arm64.whl (3.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 3.7 MB/s eta 0:00:00 Requirement already satisfied: numpy>=1.9 in /Users/cslater/.pyenv/versions/3.9.2/envs/eipy/lib/python3.9/site-packages (from netCDF4) (1.24.2) Requirement already satisfied: cftime in /Users/cslater/.pyenv/versions/3.9.2/envs/eipy/lib/python3.9/site-packages (from netCDF4) (1.6.2) Installing collected packages: netCDF4 Successfully installed netCDF4-1.6.2

[electricsam]

I modified @fmaussion 's script to perform a no-op after opening the test file and removed the graph generation since that would use memory. I simply print the memory usage.

import gc
import os

import netCDF4
import numpy as np
import psutil

process = psutil.Process(os.getpid())

f = 'dummy_data.nc'

if os.path.exists(f):
  os.remove(f)

with netCDF4.Dataset(f, 'w', format='NETCDF4') as nc:
  nc.createDimension('time', None)
  v = nc.createVariable('prcp', 'f4', ('time',))
  v[:] = np.arange(10000)
  v = nc.createVariable('temp', 'f4', ('time',))
  v[:] = np.arange(10000)


def dummy_read():
  with netCDF4.Dataset(f) as nc:
    pass
  gc.collect()

for i in range(20001):
  dummy_read()
  if i % 100 == 0:
    print("{}: {}".format(i, process.memory_info().rss * 1e-6))

The following results were printed:

0: 40.321024 100: 46.20288 200: 46.563328 300: 48.119808 400: 49.545215999999996 500: 49.905663999999994 600: 50.249728 700: 50.561023999999996 800: 50.888704 900: 51.249151999999995 1000: 51.625983999999995 1100: 51.937279999999994 1200: 52.379647999999996 1300: 52.723712 1400: 53.051392 1500: 53.444607999999995 1600: 53.73952 1700: 54.099968 1800: 54.444032 1900: 54.722559999999994 2000: 55.13216 2100: 55.45984 2200: 55.771136 2300: 56.131584 2400: 56.557567999999996 2500: 56.868863999999995 2600: 57.26208 2700: 57.524224 2800: 57.311232 2900: 54.607872 3000: 54.886399999999995 3100: 55.345152 3200: 55.640063999999995 3300: 55.967743999999996 3400: 56.426496 3500: 56.754175999999994 3600: 57.081855999999995 3700: 57.458687999999995 3800: 57.802752 3900: 58.146815999999994 4000: 58.523647999999994 4100: 58.867712 4200: 59.211776 4300: 59.260928 4400: 59.703295999999995 4500: 60.063744 4600: 60.391424 4700: 60.751872 4800: 60.669951999999995 4900: 61.046783999999995 5000: 59.686912 5100: 59.539455999999994 5200: 59.965439999999994 5300: 60.37504 5400: 60.719103999999994 5500: 60.899328 5600: 61.292544 5700: 61.620224 5800: 61.915136 5900: 62.062591999999995 6000: 61.685759999999995 6100: 61.079552 6200: 61.702144 6300: 62.275583999999995 6400: 62.652415999999995 6500: 62.34112 6600: 57.09824 6700: 57.163776 6800: 58.228736 6900: 59.129856 7000: 59.981823999999996 7100: 60.735488 7200: 61.571071999999994 7300: 62.078976 7400: 62.488575999999995 7500: 63.012864 7600: 63.307776 7700: 63.700992 7800: 64.258048 7900: 64.733184 8000: 65.20832 8100: 65.61792 8200: 66.011136 8300: 66.453504 8400: 66.977792 8500: 67.403776 8600: 67.846144 8700: 68.23935999999999 8800: 68.583424 8900: 68.87833599999999 9000: 68.829184 9100: 69.173248 9200: 69.46816 9300: 69.79584 9400: 70.090752 9500: 70.434816 9600: 70.877184 9700: 71.221248 9800: 71.66361599999999 9900: 72.040448 10000: 72.433664 10100: 72.777728 10200: 73.17094399999999 10300: 73.580544 10400: 74.006528 10500: 74.366976 10600: 74.71104 10700: 75.10425599999999 10800: 75.3664 10900: 75.72684799999999 11000: 76.16921599999999 11100: 76.59519999999999 11200: 76.955648 11300: 77.266944 11400: 77.4144 11500: 77.758464 11600: 78.168064 11700: 78.544896 11800: 78.6432 11900: 79.003648 12000: 79.38047999999999 12100: 70.402048 12200: 64.028672 12300: 62.275583999999995 12400: 60.53888 12500: 61.423615999999996 12600: 62.930944 12700: 64.258048 12800: 65.69984 12900: 66.7648 13000: 67.9936 13100: 69.074944 13200: 70.156288 13300: 70.991872 13400: 71.860224 13500: 72.66304 13600: 73.515008 13700: 74.039296 13800: 74.481664 13900: 75.005952 14000: 75.546624 14100: 75.988992 14200: 76.34944 14300: 76.972032 14400: 77.709312 14500: 78.20083199999999 14600: 78.790656 14700: 79.314944 14800: 79.888384 14900: 80.101376 15000: 80.52736 15100: 80.98611199999999 15200: 81.26464 15300: 81.641472 15400: 82.034688 15500: 82.395136 15600: 82.952192 15700: 83.34540799999999 15800: 83.853312 15900: 84.328448 16000: 84.688896 16100: 85.1968 16200: 85.68831999999999 16300: 86.34367999999999 16400: 86.638592 16500: 87.113728 16600: 87.62163199999999 16700: 88.14591999999999 16800: 88.752128 16900: 89.276416 17000: 89.636864 17100: 89.96454399999999 17200: 90.39052799999999 17300: 90.76736 17400: 91.127808 17500: 91.439104 17600: 91.848704 17700: 92.192768 17800: 92.585984 17900: 92.913664 18000: 93.356032 18100: 93.7984 18200: 94.208 18300: 94.093312 18400: 94.552064 18500: 94.86336 18600: 95.240192 18700: 95.64979199999999 18800: 96.09215999999999 18900: 96.50175999999999 19000: 96.878592 19100: 97.32096 19200: 97.763328 19300: 98.10739199999999 19400: 98.500608 19500: 98.84467199999999 19600: 99.270656 19700: 99.631104 19800: 99.958784 19900: 100.286464 20000: 100.630528

[electricsam]

I also ran the following to see if I could determine the version of netcdf-c used in the wheel: nm _netCDF4.cpython-39-darwin.so Which listed entries like:

0000000000098dac t ___pyx_setprop_7netCDF4_8_netCDF4_8Variable_ndim

Maybe it is using v4.8.x? Could there be another memory leak or a regression?