peerProfiles too large

[khumarahn]

Hi. I ran out space today: the peerProfiles folder grew over 2.5GB. This isn't that much, but I run i2pd on a cheap vps with very limited storage. I wonder if this is normal, and if there's a way to optimize storage.

[Vort]

I suggest you to shut down node, delete peerProfiles folder and start node again.

I wonder if this is normal

This is side effect of ongoing attack.

[khumarahn]

Oh, an attack?

[Vort]

Yes, network is under attack. There were several waves of spreading fake floodfills routerinfos. That what lots of peerprofiles means. They describe fake nodes.

[khumarahn]

Is there an automatic way to remove peer profiles, or it is only manual?

[Vort]

They are cleared automatically. But in case of attack there are just too many of them.

[SilmorSenedlen]

I have noticed several attacks on ~ past week, in addition to huge number of peer profile files, I almost got exhausted from memory. Most of relays would probably have already fallen xD

After I had to move the I2Pd working directory from the RAM disk to the HDD, I have the following picture:

And relative "small" memory usage: about 500 MiB.

@Vort as you can see, now I have ~40k peers, ~30k floodfills and in total about of 1 million files in the I2Pd profile directory. Should I stop service and clean up NetDB and/or peerProfiles, or wait until I2Pd cleans up data itself due to its irrelevance?

[Vort]

@SilmorSenedlen that RAM consumption is a big problem. But I don't know how to track down why it happens. From my experience, it can happen because of lags. Something somewhere make system overload and i2pd start glitching because of it.

First of all, you need to update your node. I personally tested 2.47.0-67-gc1168c2a and it works relatively good. But there are more commits made after it, will test them too soon. If you will update node, then in the process it is better to clean peerProfiles. As for netDb, I think it will clean itself during few hours after update.

upd. It looks like 2.47.0-71-ge068a3cf works good too.

[LLE8]

i2pd version 2.47.0-68-g2f74e670 (0.9.58) floodfill = true uptime 1 d 4 h /var/lib/i2pd/peerProfiles/ folder grew from 0 to 2 GB, 507000 files 14000-14500 transit tunnels i2pd process memory usage: about 380 MB

[snaike00]

Where can download this new nodes?

[Vort]

For Windows, here: https://github.com/PurpleI2P/i2pd/actions/workflows/build-windows.yml https://github.com/PurpleI2P/i2pd/actions/runs/4903844731 https://github.com/PurpleI2P/i2pd/suites/12722999303/artifacts/683318292

[snaike00]

For Windows, here: https://github.com/PurpleI2P/i2pd/actions/workflows/build-windows.yml https://github.com/PurpleI2P/i2pd/actions/runs/4903844731 https://github.com/PurpleI2P/i2pd/suites/12722999303/artifacts/683318292

Thank you, but i need for linux (arm), some like this: i2pd_*bionic1_armhf.deb (maybe already have package?)

[SilmorSenedlen]

that RAM consumption is a big problem. But I don't know how to track down why it happens. From my experience, it can happen because of lags. Something somewhere make system overload and i2pd start glitching because of it.

Is it way to implement hard memory limit for i2pd, to save system and other apps from OOM(out of memory)? As last resort, of course.

[LLE8]

I think may be helpful ~prlimit --as~ ~prlimit --rss~

http://jlebar.com/2011/6/15/Limiting_the_amount_of_RAM_a_program_can_use.html

[SilmorSenedlen]

I think may be helpful

Sorry that I didn't indicate it right away, but I meant Windows version* Of course, that can be "resolved" by running I2Pd in container or VM, but it would be a huge overhead and cumbersome.

[wekoq]

Should be fixed after 1d8a91c commit. For now i2pd doesn't save peer profile if it values is near to default. Please test and close.

[r4sas]

For Windows, here: https://github.com/PurpleI2P/i2pd/actions/workflows/build-windows.yml https://github.com/PurpleI2P/i2pd/actions/runs/4903844731 https://github.com/PurpleI2P/i2pd/suites/12722999303/artifacts/683318292

Thank you, but i need for linux (arm), some like this: i2pd_*bionic1_armhf.deb (maybe already have package?)

Unfortunately there is still no armhf/arm64 packages built from trunk. If anybody have packaging skills and know how OBS works: I need help with creating automated builds. Here is working repository with release upload builds.

[snaike00]

r4sas, не совсем понял в чем проблема - в создании пакета *.deb, или в компилировании бинарника?

[r4sas]

r4sas, не совсем понял в чем проблема - в создании пакета *.deb, или в компилировании бинарника?

В написании скриптов для OBS, которые бы позволили автоматически собирать пакеты для кода из транка.

[snaike00]

r4sas, не совсем понял в чем проблема - в создании пакета *.deb, или в компилировании бинарника?

В написании скриптов для OBS, которые бы позволили автоматически собирать пакеты для кода из транка.

ХЗ. В общем пока не получится нормально юзать i2pd на arm (

[LLE8]

@snaike00 Raspberry Pi?

[snaike00]

@snaike00 Raspberry Pi?

No, but armv7

[LLE8]

Судя по "some like this: i2pd_*bionic1_armhf.deb" на этом устройстве Ubuntu 18.04?

[snaike00]

Судя по "some like this: i2pd_*bionic1_armhf.deb" на этом устройстве Ubuntu 18.04?

Да. Возможно есть скомпилированный бинарник? Этого будет вполне достаточно :)

[LLE8]

i2pd version 2.47.0-80-g623c3f46 Uptime 7 d 18 h Size of /var/lib/i2pd/peerProfiles/ folder is 196 MB, 31718 files i2pd process memory usage about 170 MB, 12000 transit tunnels

Либо атака прекратилась (временно), либо баг испрапвлен.

[SilmorSenedlen]

Либо атака прекратилась (временно), либо баг испрапвлен.

I don't noticed any new attacks from past week. And number of known Routers and Floodfills from my previous post decreased to the current amount by itself, within a few days. Version: 2.47.0

Uptime: 11 days, 10 hours, 25 minutes, 32 seconds
Network status: OK
Network status v6: OK
Tunnel creation success rate: 10%
Received: 2103.02 GiB (947.92 KiB/s)
Sent: 2207.30 GiB (997.02 KiB/s)
Transit: 1947.43 GiB (895.81 KiB/s)
Routers: 10870 Floodfills: 1333 LeaseSets: 214
Client Tunnels: 27 Transit Tunnels: 8777

Virt. mem: 545 MiB RAM: 400 MiB peerProfiles: ~34k files netDb: ~ 30k files

[LLE8]

Значит там есть все нужные средства разработки и библиотеки? Может быть попробовать собрать непосредственно на самом устройстве? 512 МБ ОЗУ + 1ГБ свопа найдется? Отсюда https://i2pd.readthedocs.io/en/latest/devs/building/unix/ можно извлечь простую и работоспособную (проверено) инструкцию из десятка команд. Русский вариант https://i2pd.readthedocs.io/ru/latest/build_notes_unix.html кажется немного неправильный, вероятно устарел.

[snaike00]

К сожалению там очень урезанная ОС, скомпилировать на ней получится.

[r4sas]

Сейчас попробую включить сборку armhf/arm64 в PPA репозитории: https://launchpad.net/~purplei2p/+archive/ubuntu/i2pd-git/+packages

add: ан, нет... не получится. Там нельзя выбирать архитектуру при сборке по "рецептам". add2: есть переключатель архитектур в самом репозитории, но не знаю повлияет ли он. Посмотрим...

[snaike00]

Сейчас попробую включить сборку armhf/arm64 в PPA репозитории: https://launchpad.net/~purplei2p/+archive/ubuntu/i2pd-git/+packages

add: ан, нет... не получится. Там нельзя выбирать архитектуру при сборке по "рецептам". add2: есть переключатель архитектур в самом репозитории, но не знаю повлияет ли он. Посмотрим...

Переключатель архитектур судя по всему повлиял, но вижу пакет armhf только для Ubuntu 23.10.1 Может выйдет собрать armhf под Ubuntu 18.04 (Bionic) ?

[LLE8]

А может для этого устройства есть OpenWrt? Транк собирается относительно просто
https://github.com/PurpleI2P/i2pd/issues/345 там есть соответствующий Makefile