DnsServer icon indicating copy to clipboard operation
DnsServer copied to clipboard

Extremely high RAM usage after upgrade to 14.1

Open aenima99x opened this issue 1 month ago • 14 comments

Technitium version 14.1, Running in a Debian LXC (2 vCPU and 12GB RAM) on Proxmox. Server was running fine with 4GB RAM before the update to 14.1 and then it started maxing out and I had to bump it to 12GB RAM. LXC has been restarted multiple times with no change, RAM will continually increase to about 7-8GB before leveling out there.

DNS Cache has always been set to save to disk. Cache max entries was always set to the default 10,000 - I've tried increasing to 100,000 with no changes seen Enable In-Memory stats has been enabled and no changes seen

Please let me know if I can provide any other info.

Image Image Image

aenima99x avatar Nov 19 '25 00:11 aenima99x

Thanks for the post. When the memory usage started maxing out, did you see any issues with the DNS service or any errors in the DNS logs? If yes then let me know the details and post any error logs here. If there were no issues then its nothing to worry about and the memory usage will get back to normal later. Providing more memory to the server will make it consume more as the .NET runtime will cache things and may delay freeing memory. The DNS server runs using .NET runtime which is a garbage collected runtime so used memory gets removed periodically unless its urgently required.

Are you using and DNS Apps? Please share screenshot or list them.

Do you have block lists configured? Too many lists will make it use more memory and it will increase usage when the server restarts or when the block lists are updated.

ShreyasZare avatar Nov 19 '25 06:11 ShreyasZare

- When the memory usage started maxing out, did you see any issues with the DNS service or any errors in the DNS logs? Yes that's how I knew there were issues, all my DNS traffic stopped working and I checked the server and saw it pegged at 100% RAM usage.

- Are you using any DNS Apps? Nothing besides the SQLite Query Logs

- Do you have block lists configured? Yes, lists are below.

https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt https://adaway.org/hosts.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/pro-onlydomains.txt

The RAM usage has been bouncing between 6-8GB over the last 24 hours and has not gone down. As I noted before these were the exact settings I had in place before upgrading to 14.1 and the RAM usage never went over 2GB.

Appreciate the quick response, let me know what other info you need.

aenima99x avatar Nov 19 '25 14:11 aenima99x

Thanks for the details. The block lists are fine and too just 200mb when I tried on my setup. The Query Logs app too should not use much memory unless its set to use in-memory db and has high limit set for record retention.

The only major difference with v14 release is that it uses .NET 9 Runtime which may have some different behavior compared to the .NET 8 runtime being used earlier.

Please share a screenshot of the Dashboard stats (including the stats above the Top Clients table) to that the load on the server is clear and helps understand the issue better. Also share a screenshot of the Settings > Cache section.

ShreyasZare avatar Nov 20 '25 07:11 ShreyasZare

Image Image Image

Also to note, I'm now seeing higher than normal cpu and very high Disk I/O writes.

Image Image

aenima99x avatar Nov 20 '25 16:11 aenima99x

Another follow-up on this. I spun up a new LXC with the same specs and then did a docker install of Technitium 14.1. As soon as it started up the RAM immediatley jumped up to using almost all 4GB and the server was almost unusable. I then spun up another LXC with a docker install but used the 13.6 image and everything is performing as expected. I imported all my zones, dhcp, etc and the RAM usage is sitting right around 200MB. So there is definitely something with dotnet 9 that is causing issues here.

aenima99x avatar Nov 20 '25 18:11 aenima99x

Thanks for all the details. The config looks fine. I am not really sure what could be the cause of this issue. I have received a couple of such feedback till now but yet to find a cause.

I would suggest that you give it a try using a VM on proxmox instead of LXC and see if that makes any difference. I have several deployments on VM with just 1GB RAM and the v14 update is roughly using the same amount of memory as that of the previous v13.x release.

ShreyasZare avatar Nov 21 '25 07:11 ShreyasZare

I run two Technitium systems on my internal home network on two different Proxmox servers in LXC. Both have the same hardware settings. Ich use Technitium 14.1.

I would like to share this information for informational purposes only. I have allocated 1 GB RAM, 1 core, and 4 GB disk space to the LXC system. I have 805,560 entries in the block lists. The system uses approximately 270 MB RAM and runs smoothly.

Image

Image

Image

roswitina avatar Nov 21 '25 18:11 roswitina

I've been trying to narrow this down and the only common denominator in it that I can find is dotnet9.
I've tried both Debian13 and Ubuntu25.10 as both LXC and VM and the same issue always appears. As soon as I run the Technitium install script the RAM usage dramatically increases and then eventually maxes out. The current install I kept running is a Debian 13.2 LXC running Technitium 14.2 You can see over the last 24 hours that everything is increasing over time and appears like a memory leak.

Image Image

aenima99x avatar Nov 23 '25 19:11 aenima99x

Hello,

is this issue fixed in 14.2?

0xChri2 avatar Nov 24 '25 09:11 0xChri2

It was not fixed for me in 14.2. I rolled back to 13.6 for the time being, but now I'm even seeing what appears to be a memory leak there also. I really would like to use Technitium since it is a great all-in-one solution for DNS/DHCP but I may need to go back to Unbound, Kea, etc if I can't get this stable.

Here's all the pertinent info - Proxmox 9.1.1 LXC - 1 CPU, 1GB RAM, 1GB swap, Debian 13.2 Technitium installed with the install script for 13.6

Ram usage over the last 24 hours has gone from an average of 150MB to 500MB and climbing. I don't see any errors in the logs, but I do see a lot of continual DHCP lease offers. The minimum lease times I have set are 7 days.

Image Image Image

aenima99x avatar Nov 24 '25 17:11 aenima99x

OK another update. So it looks like it is DHCP causing the issue. After seeing the continual DHCP messages in the logs, I disabled my 4 DHCP scopes as a test. The RAM usage instantly dropped from 550MB to 370 MB and is holding steady there. The Disk I/O also immediately dropped.

Image Image

Here you can see the flat areas are where DHCP scopes are disabled and then re-enabled and they spike again.

Image

aenima99x avatar Nov 24 '25 18:11 aenima99x

@aenima99x Thanks for these observations. I will check why DHCP scope is causing this. I have it deployed in production but not seeing similar issues with it though.

ShreyasZare avatar Nov 25 '25 11:11 ShreyasZare

Noob here. I have the same issue on a personal Windows 10 machine. Been running it since version 13.6, don't remember seeing high RAM usage, but after updating to 14 or 14.1 RAM usage became anywhere between 2 to 4GB. In v14.1 it used to start with 4GB and then gradually went down to 2GB and never below that. After updating to v14.2 it seems to be always at 2GB.

I use it mainly for blocking and forwarding to other resolvers. I have tried disabling blocking, removed all list links, even deleted the files which I guess wasn't necessary but have no effect on RAM usage.

Everything else is at default settings. No idea what DHCP Scopes do, tried disabling the default one that was there(as one user mentioned above), no effect on RAM usage whatsoever.

I have restarted the server after making any change, if that matters.

Very useful program BTW, that even a noob like me can use it. Thank you very much for making it.

GitCentre avatar Nov 29 '25 13:11 GitCentre

Noob here. I have the same issue on a personal Windows 10 machine. Been running it since version 13.6, don't remember seeing high RAM usage, but after updating to 14 or 14.1 RAM usage became anywhere between 2 to 4GB. In v14.1 it used to start with 4GB and then gradually went down to 2GB and never below that. After updating to v14.2 it seems to be always at 2GB.

Thanks for the feedback. My guess till now is that this is due to runtime version change. However, not everyone is observing this issue since it may not be significant for them. My own servers are using roughly same memory that they used with v13.6.

I use it mainly for blocking and forwarding to other resolvers. I have tried disabling blocking, removed all list links, even deleted the files which I guess wasn't necessary but have no effect on RAM usage.

Removing block lists wont immediately free memory. The DNS server runs over .NET runtime which has garbage collector and it can take a long time to free used memory. It cannot be predicted if it will free memory the next minute or do it in the next hour. Usually if you have memory available on the server, it wont rush to free it immediately.

Everything else is at default settings. No idea what DHCP Scopes do, tried disabling the default one that was there(as one user mentioned above), no effect on RAM usage whatsoever.

I have restarted the server after making any change, if that matters.

Restarting will cause the block lists to get loaded in memory again and it uses additional memory while doing this task which may linger for a while before it gets freed. So restarts wont really fix this issue.

Very useful program BTW, that even a noob like me can use it. Thank you very much for making it.

Thanks for the compliment.

ShreyasZare avatar Nov 29 '25 15:11 ShreyasZare

Agree, Technitium is super useful and with the clustering, now over the top. Looking forward to DHCP reservation being added to clustering.

I am having extremely high RAM usage as well since 14.1. With 14.1 is was not too bad, but when I upgraded to 14.2 the problem got much worse. This is a screen capture of the Proxmox LXC where you can see the RAM usage continue to increase, and once RAM is full, CPU goes to 100% and nothing works (DNS or SSH).

Image

Only have a few small block lists and the SQL log plugin is not set to use the in-memory db. I only have a single DHCP scope. Here is a screen capture showing the dotnet process with high memory.

Image

Let me know if you need any additional information to help narrow down the problem.

I tried to setup Technitium on a new LXC and import the config, but I'm getting an error. I'll open a new issue on this one.

Thanks again and keep up the good work.

rsb-developer avatar Dec 20 '25 20:12 rsb-developer

@rsb-developer Thanks for the feedback. If possible, then do export the config and share the zip file with [email protected]. This will help me to try to replicate this issue which is otherwise quite tough to figure out.

ShreyasZare avatar Dec 21 '25 10:12 ShreyasZare

Thanks for the quick reply.

I figured out the issue and I'm not sure how I missed the errors in the logs (maybe I need new glasses)... I was using the Query Logs (Sqlite) app and at some point post updating to version 14, the sqlite database became corrupted. With the sqlite db corruption, something is not released and causing unbounded memory usage which will exhaust all memory and hang the VM.

The quick fix was to remove the app (which removed the corrupt db) and restart. Memory usage is now stable at ~600M.

Is there a way to detect if this (or other exceptions) occurs in the future? I'm hesitant to re-install Query Logs since it could cause an issue in the future. Would the other apps have similar issues?

This is a snip from the log file:

[2025-12-20 00:00:07 UTC] System.IO.EndOfStreamException: Attempted to read past the end of the stream. at TechnitiumLibrary.Net.IPAddressExtensions.ReadFrom(Stream s) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\IPAddressExtensions.cs:line 54 at TechnitiumLibrary.Net.IPAddressExtensions.ReadFrom(BinaryReader bR) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\IPAddressExtensions.cs:line 36 at DnsServerCore.Dns.StatsManager.StatCounter..ctor(BinaryReader bR) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\StatsManager.cs:line 1908 at DnsServerCore.Dns.StatsManager.HourlyStats..ctor(BinaryReader bR) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\StatsManager.cs:line 1644 at DnsServerCore.Dns.StatsManager.LoadHourlyStats(DateTime dateTime, Boolean forceReload, Boolean ifNotExistsReturnEmptyHourlyStats) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\StatsManager.cs:line 414 [2025-12-20 00:02:39 UTC] [172.31.101.26:45184] [TCP] DNS Server received zone transfer request for zone: cluster-catalog.home [2025-12-20 00:02:44 UTC] DNS App [Query Logs (Sqlite)]: Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 11: 'database disk image is malformed'. at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db) at Microsoft.Data.Sqlite.SqliteDataReader.NextResult() at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior) at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery() at System.Data.Common.DbCommand.ExecuteNonQueryAsync(CancellationToken cancellationToken) --- End of stack trace from previous location --- at QueryLogsSqlite.App.<.ctor>b__17_0(Object state) in Z:\Technitium\Projects\DnsServer\Apps\QueryLogsSqliteApp\App.cs:line 87 at QueryLogsSqlite.App.<.ctor>b__17_0(Object state) in Z:\Technitium\Projects\DnsServer\Apps\QueryLogsSqliteApp\App.cs:line 87 at QueryLogsSqlite.App.<.ctor>b__17_0(Object state) in Z:\Technitium\Projects\DnsServer\Apps\QueryLogsSqliteApp\App.cs:line 109

rsb-developer avatar Dec 22 '25 21:12 rsb-developer

@rsb-developer Good to know that you found the issue and its working stable for you now.

The reason for the sqlite db file to get corrupt is ungraceful shutdown of the DNS server (i.e. process got killed) or due to power loss. Since the Query Logs app is frequently updating the db file, its prone to get the db file corrupt in such cases.

The other error log you shared is due to corrupt stats file so the stats module is failing to read it. This indicates that you have issue with file corruption in general and its not just the sqlite db file being corrupt.

There is no way that a software can handle such issues to prevent file corruption. So, you have to ensure that the DNS server always shuts down gracefully and that the server has UPS for power backup.

ShreyasZare avatar Dec 23 '25 09:12 ShreyasZare