[移动云盘][139云盘]PUT方法上传多个文件时出现空指针或者内存地址错误nil pointer dereference

[jibwf]

Please make sure of the following things

• [x] I have read the documentation. 我已经阅读了文档。

• [x] I'm sure there are no duplicate issues or discussions. 我确定没有重复的issue或讨论。

• [x] I'm sure it's due to AList and not something else(such as Network ,Dependencies or Operational). 我确定是AList的问题，而不是其他原因（例如网络，依赖或操作）。

• [x] I'm sure this issue is not fixed in the latest version. 我确定这个问题在最新版本中没有被修复。

AList Version / AList 版本

v3.37.4

Driver used / 使用的存储驱动

中国移动云盘

Describe the bug / 问题描述

批量上传文件时出现panic. debug log已经采集作为日志附件上传了。

[31m2024/09/25 09:28:27 [Recovery] 2024/09/25 - 09:28:27 panic recovered: PUT /mobile/p96f17e2785019958977bb8445983c075-s1e3e9ab4e386d28f12d HTTP/1.1 Host: 192.168.1.2:5246 Accept-Encoding: gzip Authorization: * Content-Length: 24880957 Content-Md5: 3h5q/xzn7Xccv4glHJUsnA== Content-Type: application/x-kopia User-Agent: MinIO (linux; amd64) minio-go/v7.0.69 X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD X-Amz-Date: 20240925T012807Z X-Amz-Decoded-Content-Length: 24846672

runtime error: invalid memory address or nil pointer dereference /opt/hostedtoolcache/go/1.23.1/x64/src/runtime/panic.go:262 (0x477fb8)

Reproduction / 复现链接

如果需要可以单独提供

Config / 配置

{ "force": false, "site_url": "", "cdn": "", "jwt_secret": "mNIstpFmBVEFR07R", "token_expires_in": 48, "database": { "type": "sqlite3", "host": "", "port": 0, "user": "", "password": "", "name": "", "db_file": "data/data.db", "table_prefix": "x_", "ssl_mode": "", "dsn": "" }, "meilisearch": { "host": "http://localhost:7700/", "api_key": "", "index_prefix": "" }, "scheme": { "address": "0.0.0.0", "http_port": 5244, "https_port": -1, "force_https": false, "cert_file": "", "key_file": "", "unix_file": "", "unix_file_perm": "" }, "temp_dir": "data/temp", "bleve_dir": "data/bleve", "dist_dir": "", "log": { "enable": true, "name": "data/log/log.log", "max_size": 10, "max_backups": 5, "max_age": 28, "compress": false }, "delayed_start": 0, "max_connections": 0, "tls_insecure_skip_verify": true, "tasks": { "download": { "workers": 5, "max_retry": 1, "task_persistant": true }, "transfer": { "workers": 5, "max_retry": 2, "task_persistant": true }, "upload": { "workers": 5, "max_retry": 0, "task_persistant": false }, "copy": { "workers": 5, "max_retry": 2, "task_persistant": true } }, "cors": { "allow_origins": [ "" ], "allow_methods": [ "" ], "allow_headers": [ "*" ] }, "s3": { "enable": true, "port": 5246, "ssl": false } }

Logs / 日志

[31m2024/09/25 09:28:27 [Recovery] 2024/09/25 - 09:28:27 panic recovered: PUT /mobile/p96f17e2785019958977bb8445983c075-s1e3e9ab4e386d28f12d HTTP/1.1 Host: 192.168.1.2:5246 Accept-Encoding: gzip Authorization: * Content-Length: 24880957 Content-Md5: 3h5q/xzn7Xccv4glHJUsnA== Content-Type: application/x-kopia User-Agent: MinIO (linux; amd64) minio-go/v7.0.69 X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD X-Amz-Date: 20240925T012807Z X-Amz-Decoded-Content-Length: 24846672 log-2024-09-25T01-30-27.705.log.zip

[pongfcnkl]

试试添加本机存储后，用复制功能上传

[jibwf]

试试添加本机存储后，用复制功能上传

您的意思是先把文件放到本机存储，然后复制到网盘？

[pongfcnkl]

试试添加本机存储后，用复制功能上传

您的意思是先把文件放到本机存储，然后复制到网盘？

[jibwf]

试试添加本机存储后，用复制功能上传

您的意思是先把文件放到本机存储，然后复制到网盘？

复制的方式上传没有遇到panic的问题. 不过这样操作，比较难自动化备份. 所以这个问题和139网盘驱动有关吗？还是和主程序fs.go 方法有关系？

之前panic时候的traceback. [31m2024/09/25 20:26:30 [Recovery] 2024/09/25 - 20:26:30 panic recovered: runtime error: invalid memory address or nil pointer dereference /opt/hostedtoolcache/go/1.23.1/x64/src/runtime/panic.go:262 (0x477fb8) /opt/hostedtoolcache/go/1.23.1/x64/src/runtime/signal_unix.go:900 (0x477f88) /opt/hostedtoolcache/go/1.23.1/x64/src/internal/bytealg/equal_amd64.s:114 (0x405b76) /home/runner/work/alist/alist/internal/op/fs.go:222 (0x9882b4) /home/runner/work/alist/alist/internal/op/fs.go:231 (0x9888b2) /home/runner/work/alist/alist/internal/op/fs.go:518 (0x98bfaa) /home/runner/work/alist/alist/internal/fs/put.go:70 (0xe33fb0) /home/runner/work/alist/alist/internal/fs/fs.go:97 (0xe321ae) /home/runner/work/alist/alist/server/s3/backend.go:325 (0x1ccd32a) /home/runner/go/pkg/mod/github.com/alist-org/[email protected]/gofakes3.go:728 (0x1cb9032) /home/runner/go/pkg/mod/github.com/alist-org/[email protected]/routing.go:88 (0x1cc135a) /home/runner/go/pkg/mod/github.com/alist-org/[email protected]/routing.go:61 (0x1cc1152) /opt/hostedtoolcache/go/1.23.1/x64/src/net/http/server.go:2220 (0x790908) /home/runner/go/pkg/mod/github.com/alist-org/[email protected]/cors.go:45 (0x1cb23d7) /home/runner/go/pkg/mod/github.com/alist-org/[email protected]/gofakes3.go:163 (0x1cb3d53) /opt/hostedtoolcache/go/1.23.1/x64/src/net/http/server.go:2220 (0x790908) /home/runner/go/pkg/mod/github.com/alist-org/[email protected]/gofakes3.go:144 (0x1cb36f2) /opt/hostedtoolcache/go/1.23.1/x64/src/net/http/server.go:2220 (0x790908) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/utils.go:49 (0x1cf792a) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 (0xe1f7ee) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102 (0xe1f7db) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 (0xe1e924) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:249 (0xe1e90b) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 (0xe1dd11) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:633 (0xe1d780) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:589 (0xe1d2b1) /opt/hostedtoolcache/go/1.23.1/x64/src/net/http/server.go:3210 (0x7afd8d) /opt/hostedtoolcache/go/1.23.1/x64/src/net/http/server.go:2092 (0x78f2af) /opt/hostedtoolcache/go/1.23.1/x64/src/runtime/asm_amd64.s:1700 (0x47e5a0) [0m [GIN] 2024/09/25 - 20:26:30 | 500 | 51.071551543s | 172.17.0.1 | PUT "/mobile/pbce9e795bfb82a679892633a906fa188-sefdfd621c4f20b3412d" 2024/09/25 20:26:36 INFO worker execute task worker=1 task=IozDyXhs2SrEMXpLiJhSh

[jibwf]

我限制了同时上传2个，也还是会panic

[pongfcnkl]

试试添加本机存储后，用复制功能上传

您的意思是先把文件放到本机存储，然后复制到网盘？

复制的方式上传没有遇到panic的问题. 不过这样操作，比较难自动化备份. 所以这个问题和139网盘驱动有关吗？还是和主程序fs.go 方法有关系？

之前panic时候的traceback. �[31m2024/09/25 20:26:30 [Recovery] 2024/09/25 - 20:26:30 panic recovered: runtime error: invalid memory address or nil pointer dereference /opt/hostedtoolcache/go/1.23.1/x64/src/runtime/panic.go:262 (0x477fb8) /opt/hostedtoolcache/go/1.23.1/x64/src/runtime/signal_unix.go:900 (0x477f88) /opt/hostedtoolcache/go/1.23.1/x64/src/internal/bytealg/equal_amd64.s:114 (0x405b76) /home/runner/work/alist/alist/internal/op/fs.go:222 (0x9882b4) /home/runner/work/alist/alist/internal/op/fs.go:231 (0x9888b2) /home/runner/work/alist/alist/internal/op/fs.go:518 (0x98bfaa) /home/runner/work/alist/alist/internal/fs/put.go:70 (0xe33fb0) /home/runner/work/alist/alist/internal/fs/fs.go:97 (0xe321ae) /home/runner/work/alist/alist/server/s3/backend.go:325 (0x1ccd32a) /home/runner/go/pkg/mod/github.com/alist-org/[email protected]/gofakes3.go:728 (0x1cb9032) /home/runner/go/pkg/mod/github.com/alist-org/[email protected]/routing.go:88 (0x1cc135a) /home/runner/go/pkg/mod/github.com/alist-org/[email protected]/routing.go:61 (0x1cc1152) /opt/hostedtoolcache/go/1.23.1/x64/src/net/http/server.go:2220 (0x790908) /home/runner/go/pkg/mod/github.com/alist-org/[email protected]/cors.go:45 (0x1cb23d7) /home/runner/go/pkg/mod/github.com/alist-org/[email protected]/gofakes3.go:163 (0x1cb3d53) /opt/hostedtoolcache/go/1.23.1/x64/src/net/http/server.go:2220 (0x790908) /home/runner/go/pkg/mod/github.com/alist-org/[email protected]/gofakes3.go:144 (0x1cb36f2) /opt/hostedtoolcache/go/1.23.1/x64/src/net/http/server.go:2220 (0x790908) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/utils.go:49 (0x1cf792a) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 (0xe1f7ee) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102 (0xe1f7db) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 (0xe1e924) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:249 (0xe1e90b) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:185 (0xe1dd11) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:633 (0xe1d780) /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:589 (0xe1d2b1) /opt/hostedtoolcache/go/1.23.1/x64/src/net/http/server.go:3210 (0x7afd8d) /opt/hostedtoolcache/go/1.23.1/x64/src/net/http/server.go:2092 (0x78f2af) /opt/hostedtoolcache/go/1.23.1/x64/src/runtime/asm_amd64.s:1700 (0x47e5a0) �[0m [GIN] 2024/09/25 - 20:26:30 | 500 | 51.071551543s | 172.17.0.1 | PUT "/mobile/pbce9e795bfb82a679892633a906fa188-sefdfd621c4f20b3412d" 2024/09/25 20:26:36 INFO worker execute task worker=1 task=IozDyXhs2SrEMXpLiJhSh

你可以用rclone进行自动化操作，无非就是把put换成命令。 rclone用相同的配置名称就是复制了，比如rclone -P copy alist:本地 alist:139 ...

除非你的文件和alist不在一个地方，否则建议走复制，如果不在一个地方，可以考虑搭建在有文件的设备上，然后在走复制。

[jibwf]

明白了感谢。之前考虑过用rclone，但是我这个应用是kopia备份，它对rclone的支持也是实验性的，所以就没用rclone，想减少一层. 希望有大佬能看看修复一下

[pongfcnkl]

明白了感谢。之前考虑过用rclone，但是我这个应用是kopia备份，它对rclone的支持也是实验性的，所以就没用rclone，想减少一层. 希望有大佬能看看修复一下

其实rclone的同名称复制对alist来说才是减少一层。 另外可以先用kopia备份到本地，然后在用rclone复制到云盘

[jibwf]

@pongfcnkl 其实rclone的同名称复制对alist来说才是减少一层。 能不能详细说一下，我之前理解rclone就是通过webdav方法连接alist使用网盘和我直接用alist webdav方法有区别吗？

[pongfcnkl]

@pongfcnkl 其实rclone的同名称复制对alist来说才是减少一层。 能不能详细说一下，我之前理解rclone就是通过webdav方法连接alist使用网盘和我直接用alist webdav方法有区别吗？

意思是用相同的webdav配置名称进行copy复制，这样就和你网页上复制一样了 下面这个相当于网页上的复制 rclone --timeout 0 --retries 9 --transfers 1 --ignore-existing --tpslimit 1 -P copy alist:本机存储路径 alist:139 ... 下面这个就和你平常用webdav没区别了 rclone -P copy 本地路径 alist:139 ...

[Mmx233]

Taking look at this issue and #7127. @jibwf. If you still willing to help, please provider a testing environment by following steps:

1. 下载或复制最新的 Alist 本体到空目录
2. 启动并记录初始账号
3. 添加有问题的 Dirver，并确认可以使用
4. 将整个目录打包，和初始账号一起通过电子邮件发送到 [email protected]

[jibwf]

Taking look at this issue and #7127. @jibwf. If you still willing to help, please provider a testing environment by following steps:

1. 下载或复制最新的 Alist 本体到空目录
2. 启动并记录初始账号
3. 添加有问题的 Dirver，并确认可以使用
4. 将整个目录打包，和初始账号一起通过电子邮件发送到 [email protected]

感谢大佬国庆期间帮忙排查，环境已经发送.

[Mmx233]

是通过前端批量上传吗，我这边没有触发 panic

[jibwf]

@Mmx233 是通过alist开启S3服务，然后通过S3批量上传.

这种方式和前端页面上传有区别吗？前端页面我也来测试一下

[Mmx233]

但是根据 #7071 ，s3 不支持上传

[Mmx233]

此环境亦无法复现 #7127 内存泄漏

[jibwf]

但是根据 #7071 ，s3 不支持上传

我看官网的介绍是支持的，我也可以上传文件，之前就是间歇性的panic，所以速度降低很多.

刚才我用S3drive测试了同时上传多个大文件，也没复现出来. 目前kopia+rclone+alist形式了，使用正常.

感谢大佬花费时间.

[foxxorcat]

错误已修复 #7865

[catfishlty]

目前kopia+rclone+alist形式了

您好，想问问，kopia+rclone+alist具体是怎么连接了？ kopia 创建repository到rclone这一步是直接在物理机上部署的？

[pongfcnkl]

目前kopia+rclone+alist形式了

您好，想问问，kopia+rclone+alist具体是怎么连接了？ kopia 创建repository到rclone这一步是直接在物理机上部署的？

放弃kopia，直接用rclone的复制

[catfishlty]

放弃kopia，直接用rclone的复制

我的是Docker容器架构（rclone和alist都在容器里面），rclone复制需要配置前置执行数据库备份命令，然后过滤后才能复制，这两步感觉需要自己实现逻辑才行。