XTLS
iOS packet tunnel provider limit
Original discussion: https://github.com/XTLS/Xray-core/discussions/4398
Good afternoon, dear members of our community!
Specifically, we know for certain that in the current versions of iOS, we have the following memory limitations:
| Provider | Limit (MiB) |
|---|---|
| packet tunnel | 50 |
| app proxy | 15 |
| filter control | 50 |
| filter data | 50 |
| DNS proxy | 15 |
| app push | 24 |
In our case, Xray is a network process limited to 50MB. It reads geo files and then operates with the read content.
Unfortunately, with the current implementation of geofiles in our core, where all categories are in one large file, we often encounter the inability to even simply start the tunnel on an Apple device. In particular, with the new geolocation files that we received from our partners, the file sizes have become significantly larger, which means that we cannot load them into memory.
A simple example (Loyalsoldier geosite):
None of the first four categories, if set in the routing rules, will allow the tunnel to start.
This also applies to geo files with CIDR (Loyalsoldier geoip)
Simple example:
@RPRX , considering the memory constraints, perhaps we should explore refactoring the geofile handling code. We could draw inspiration from neighboring teams and implement a system that utilizes multiple smaller files instead of one monolithic file. This approach could effectively resolve the memory issues on Apple devices. What you think about this?
on the iOS(with 8 GB ram) and in the "streisand" app, in addition to Loyalsoldier's geo-ip and geo-site, I added another geo-site(iran.dat), but still no problem occurred.
@patterniha The RAM here is irrelevant; this is specifically about the packet tunnel provider.
Are you sure you’re using the latest files? As of now, the old v2fly files are still included by default in Streisand.
Which categories are you using? For 100% of users in the control group, categories like geosite:category-ads-all, geosite:cn, and others caused the tunnel to crash.
geoip:ir, geosite:category-ads, geosite:category-ir, ext:iran.dat:tld-ir i replace default geoip and geosite with Loyalsoldier's geoip and geosite. also added a new geosite(iran.dat) and in the "streisand" settings i turn off memory limitation.
Although I did this about a month ago and now I don't have access to an iPhone to test again.
Do you mean that the problem is with the large size of the loyalsoldier and the previous default geo has no problem? or do you mean that even the previous default geo has a problem because it is a single file?
Xray-core 做过优化只解析有需要的 categories，不过前提是先把整个 geo 文件加载到内存中，我看了下新的是 21.3MB
如果这部分要继续优化就只能逐块读取文件
@patterniha I have created two simple visualizers for new files: https://geosite.marzban.dev https://geoip.marzban.dev
You can check the categories you specified.
In this case, with all due respect, your comparison is not relevant.
The geosite:ads category contains 749 rules, and there are no issues loading them.
However, the geosite:ads-all category contains 65,565(!!!!) rules, and loading them causes the tunnel to crash.
The geosite:category-ir category contains 170 rules.
The geosite:cn category contains 94,767(!!!) rules.
We could draw inspiration from neighboring teams and implement a system that utilizes multiple smaller files instead of one monolithic file.
我不懂这个 inspiration from neighboring teams 指的是什么，Xray 早在 v1.1.4 就有了优化机制，v2fly 是抄 Xray 的做法但只字未提
https://github.com/XTLS/Xray-core/releases/tag/v1.1.4
https://github.com/v2fly/v2ray-core/releases/tag/v4.39.0
不过我刚看了下 v2fly 这个是直接对着 file 读而不是对着内存中的 file 读，可以省点瞬时内存，但仍不敌 unmarshal 后的内存占用
要想 unmarshal 后占的内存少点，只能使用精简版的 geofiles
还是说你指的是 sing-box 的 rule-sets？那个我还没看
"utilizes multiple smaller files instead of one monolithic file" 很明显说的是singbox的rule-sets 它是切成多个小文件 每次只读取需要的((((
~~原来 rule-sets 就是这个，刚看到那两张图我还想着要不切成 1331+260 个文件算了~~
其实不切文件也能先用 1MB 的缓存遍历整个文件并建立 index 索引的，然后要哪部分就读哪部分，如果 Golang 方便这么做的话
但是这样也就顶多省个瞬时内存，比如减小 20MB 这样，对 unmarshal 后还是没有帮助，还是需要用精简版的 geofiles
如果是为了控制在 iOS 的 50MB 限制内，最终还是不能使用大而全的 geofiles，“进一步省瞬时内存”的必要性就存疑了
因为即使启动阶段省下了 20MB，后面代理跑起来也需要内存、缓存，给个 20MB 不过分吧
如果“进一步省瞬时内存”后，启动阶段都吃了顶格 50MB 了，后面还有什么内存跑代理
if the problem is related to premise load of geofile --> It should crash even for smaller categories like "ir".
so the problem is related to size of big categories like "cn" or "ads-all". and reading part by part doesn't help at all.
///
but 99% of the domains are less than 32 bytes length. so even if we have 200000 domains the size should not exceed 7 MB.
so why does it occupy 21.3 MB?
21.3MB 是现在 geoip.dat 14.7MB 加 geosite.dat 6.6MB，不过它俩本身就有“压缩”，unmarshal 后会膨胀一些，没测算过膨胀的比例
不过 Xray 不是整个 unmarshal，而是先把这 21.3MB 读进内存，找到目标 categories 再 unmarshal，然后释放这 21.3MB，再跑代理
继续优化就是不一次性把 21.3MB 读进内存，但如我所说这样没什么意义，即使能先在 iOS 上跑起来了，后面跑代理的内存也不富裕
这里有数据：https://github.com/XTLS/Xray-core/issues/4348#issuecomment-2644521036
但我也说过：https://github.com/XTLS/Xray-core/issues/4348#issuecomment-2644521759
我觉得倒是可以看看 unmarshal 后的结构有没有优化空间
When loading geoste:cn, one takes up 27m of memory and the other takes up 36m.
why 27m and 36m? according to https://github.com/XTLS/Xray-core/issues/4422#issuecomment-2674414972 we have less than 100000 rules.
So even if we assume that each domain is 32 bytes after decompression(although 99% of them are shorter) --> 32 * 100000 = 3.2m, so why 36m?
我觉得倒是可以看看 unmarshal 后的结构有没有优化空间
看了下，果然发现了大问题：
https://github.com/XTLS/Xray-core/blob/be43f66b63d5fdc7322675127672cb5cbe14a125/app/router/config.pb.go#L138-L150
https://github.com/XTLS/Xray-core/blob/be43f66b63d5fdc7322675127672cb5cbe14a125/app/router/config.pb.go#L203-L213
~~这下就很抽象了~~
@Fangliding 你试试把前三个 member 给删了并编译出来看看能少吃多少内存，还有 Domain Attribute 是干嘛的，实际有用到吗
而且它们的数组还是 []*router.Domain 和 []*router.CIDR，每条都多占个指针的内存
在路由使用 "geosite:cn" 的情况下，我的配置优化前静置内存占 15.1MB，优化后静置内存占 14.7MB
但是我发现了一个更严重的问题，如果 "geosite:cn" 出现了两次，内存占用会增加约 8MB
这应该是因为 IPCache[index] = &geoip 和 SiteCache[index] = &geosite 没有生效，我需要想下当初为啥是这样的决定。。。
我需要想下当初为啥是这样的决定。。。
应该是因为关缓存只需关 geofiles 的缓存，而不应该关 unmarshal 后的内容的缓存，结果当初一起关了
https://github.com/XTLS/Xray-core/issues/68#issuecomment-745165554
https://github.com/XTLS/Xray-core/issues/68#issuecomment-745231528
这个rule的大小是小问题吧 最后都要被编译成它AC自动机里的trie树的
不过按 @Fangliding 的说法，如果这里 rule 的大小无关，可能是 GC 不及时的问题？因为优化前后确实有区别
并且产生了新问题，如果像 SiteCache[index] = &geosite 这样的缓存没用，该怎么复用？~~需要深入研究下路由了~~
还有我在想 unmarshal 时 IP 那个 []byte 应该是 clone 而不是取现有文件的切片吧。。。
在 libXray 的早期版本中，实现了一种优化方案：在 packet tunnel 启动前，对用到的 geo 文件进行裁剪（只保留用到的规则）。具体可参见 geo_cut.go 。该文件已在后续更新中被删除。~~受限于当时的技术水平，此优化方案效果并不明显。~~
如果你们正在开发或改进你们的 iOS 客户端，此方案可以作为一种参考。
Domain Attribute 通常用于域名规则的特定地区分流，比如“geosite:apple@cn”，则只作用于 apple 相关的中国区域名。
In early versions of libXray, an optimization solution was implemented: before starting the packet tunnel, the geo files used were trimmed (only the rules used were retained). For details, please see geo_cut.go. This file has been removed in a subsequent update.
If you are developing or improving your iOS client, this solution can be used as a reference.
When I worked with loading large files (100+ GB) over the network in Java, I used file streaming to disk instead of loading into RAM. Then I processed those large compressed files without unpacking (the uncompressed ones were 600-1000 GB).
Is it possible to use the streaming principle in Xray?
To restructure the file so that it can be read not entirely into RAM, but in chunks consisting of byte arrays or whatever is inside the .dat file?
Store meta information about the sections inside the file and access specific areas instead of reading everything and then searching for the required part.
The CPU load will be higher, but why not?
Do I understand correctly that if the rules specify domain:ru, there will be no pupa.ru, zalupa.ru in the RAM?