Connection Id and Parallel For Loop

[stpierrelab]

Hi DataJoint Team, I have set up a MySQL server on a Synology NAS with 6 GB of ram to work with DataJoint. I noticed that after having a prolonged time of usage (days), the ram usage on the server (used by the mariaDB) would gradually increase to a very high level, sometime resulting restarts. I check the issue 58 but also noticed an issue that I'm not sure if related: The connection_id() shown when establishing the link is not the same as what is displayed by calling dj.conn, under dj.Connection.connId.

 0:  10.###.###.### via TCP/IP           Server version 5.5.5-10.0.32-MariaDB

  connection_id() 
 +---------------+
  132

And close(dj.conn) always gives out prompt for id 0 connection:

>> close(dj.conn);
closing DataJoint connection #0

Could it be that the connection is not closed properly by calling close(dj.conn) thus being accumulated? My guess on this comes also from using the parfor loop like this:

parfor i = 1:4
    dj.conn();
    parpopulate(xxxx.yyyyy)
    close(dj.conn);
end

and after the run, the ram is not returned to the idle level. Please comment on my way of using parpopulate and the issue I found. Thanks for your support!

[ixcat]

I believe @eywalker is investigating the connection tracking -

As for the RAM usage- MySQL will cache data in RAM across connections, and conceivably may cache beyond the limit of the host system depending on usage patterns. You might review the output of show engine innodb status; as well as show variables like '%innodb%' and other related settings from within a mariadb root user login. If the crash is just the mariadb server and not the whole system, you might also review the system logs and the linux 'oom killer' settings to see if linux is terminating mariadb when the system runs low on memory, since this is a possibility when mariadb/mysql is caching large amounts of data.

[eywalker]

As confusing as it is, the connId on the dj.Connection object (as returned by dj.conn()) is actually not the same connection id as you would find on the MySQL server. The connId is more like a object pointer that is local to your MATLAB and is 0 most of the time.

As for the parfor and parpopulate, I have to check a few things before I can get back on this concretely. Running out of the memory issue is most likely related to the points brought up by @ixcat