aliyunpan
aliyunpan copied to clipboard
tickstep
unraid系统docker运行CPU\IO写入的很高的问题
版本：v0.2.1 模式：update
上传没报错，但CPU一直在28%左右运行，磁盘读写一直在30M/s，上传并发2，速度也不是很快，但是就是读写和CPU很高。
另外就是，update模型下，之前用其它工具做过上传备份，现在改用这个工具，已存在的文件会自动替换，不能做到跳过已存在的文件。
A7C326EB4FE169D2A7BE411","path":"/home/app/data/F-Organized/2019/04/20190412_145719_723D0D7B.jpg","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.678] update pan file to db: {"driveId":"19863","domainId":"bj29","fileId":"62bb747c631ea4dfeef84a8da2feef7118bc3cf0","fileName":"20210529_142821_FE9C0E1D.png","fileSize":819988,"fileType":"file","createdAt":"2022-06-29 05:37:00","updatedAt":"2022-06-29 05:37:04","fileExtension":"png","uploadId":"C1F006CD242744CEA3545BDAD475157E","parentFileId":"62a7940586faa64090b54403957283843c3142e2","crc64Hash":"18040427096584971391","sha1Hash":"CD54DC847E33117C931EF468F6364741E461090F","path":"/Unriad-Photo-Backup/B-Organized/2021/05/20210529_142821_FE9C0E1D.png","category":"image","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.687] update local file to db: {"fileName":"20190412_145739_31AE9475.jpg","fileSize":12026895,"fileType":"file","createdAt":"2022-04-02 00:26:21","updatedAt":"2022-04-02 00:26:21","fileExtension":".jpg","sha1Hash":"FA3398E5CA5D3CC5AC90A98852E7E538CA252FA2","path":"/home/app/data/F-Organized/2019/04/20190412_145739_31AE9475.jpg","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.691] update pan file to db: {"driveId":"19863","domainId":"bj29","fileId":"62bb738c324e6e9dfdcd4f4bb420c4404b653eca","fileName":"20210529_142554_7561BA1E.png","fileSize":795020,"fileType":"file","createdAt":"2022-06-29 05:33:00","updatedAt":"2022-06-29 05:33:02","fileExtension":"png","uploadId":"AE3AAC8C0C2042C48E9AF384AFB91DA2","parentFileId":"62a7940586faa64090b54403957283843c3142e2","crc64Hash":"13849846887943846758","sha1Hash":"75C6D262F84F1901597E7DA96F82E4C93A3E26DD","path":"/Unriad-Photo-Backup/B-Organized/2021/05/20210529_142554_7561BA1E.png","category":"image","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.700] update local file to db: {"fileName":"20190412_145755_EC398390.jpg","fileSize":11584743,"fileType":"file","createdAt":"2022-04-02 00:26:25","updatedAt":"2022-04-02 00:26:25","fileExtension":".jpg","sha1Hash":"BFDD6837FBF0C022999CE73B6111995671759098","path":"/home/app/data/F-Organized/2019/04/20190412_145755_EC398390.jpg","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.703] update pan file to db: {"driveId":"19863","domainId":"bj29","fileId":"62bb71ebfd9558fc229e4c8091e0485807304d57","fileName":"20210529_060927_A794AB31.mp4","fileSize":358407,"fileType":"file","createdAt":"2022-06-29 05:26:03","updatedAt":"2022-06-29 05:26:11","fileExtension":"mp4","uploadId":"88058D5B15E04816AA9B83F90D2E2C25","parentFileId":"62a7940586faa64090b54403957283843c3142e2","crc64Hash":"5855694800580578096","sha1Hash":"4F9E6457FFC2E92F786B130709174F2C4563A4ED","path":"/Unriad-Photo-Backup/B-Organized/2021/05/20210529_060927_A794AB31.mp4","category":"video","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.712] update local file to db: {"fileName":"20190412_153656_A8A2EFC5.jpg","fileSize":2098734,"fileType":"file","createdAt":"2022-04-02 00:26:35","updatedAt":"2022-04-02 00:26:35","fileExtension":".jpg","sha1Hash":"1410071AD1E9B91CFAB134E6CED21C9706DBFB2F","path":"/home/app/data/F-Organized/2019/04/20190412_153656_A8A2EFC5.jpg","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.715] update pan file to db: {"driveId":"19863","domainId":"bj29","fileId":"62bb709aa3eb578fe16348a6a63ee57a70de973b","fileName":"20210529_060038_C51DEB28.mp4","fileSize":424038,"fileType":"file","createdAt":"2022-06-29 05:20:26","updatedAt":"2022-06-29 05:20:38","fileExtension":"mp4","uploadId":"F4E6DF97DF9941418A0BA1A6B45A46E4","parentFileId":"62a7940586faa64090b54403957283843c3142e2","crc64Hash":"4788624967240189861","sha1Hash":"DBC9F701F6BBBC77D6EAC5F54622E3B0B7F63004","path":"/Unriad-Photo-Backup/B-Organized/2021/05/20210529_060038_C51DEB28.mp4","category":"video","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.724] update local file to db: {"fileName":"20190412_153658_4A06D9AB.jpg","fileSize":2183267,"fileType":"file","createdAt":"2022-04-02 00:26:40","updatedAt":"2022-04-02 00:26:40","fileExtension":".jpg","sha1Hash":"B2E3F2AAFC83FFE0D27BC7F0F3444858B659B5D3","path":"/home/app/data/F-Organized/2019/04/20190412_153658_4A06D9AB.jpg","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.728] update pan file to db: {"driveId":"19863","domainId":"bj29","fileId":"62bb6e37a470e16272444ccaa1ab1dfc77fb291e","fileName":"20210528_191443_B064F932.png","fileSize":1286998,"fileType":"file","createdAt":"2022-06-29 05:10:15","updatedAt":"2022-06-29 05:10:20","fileExtension":"png","uploadId":"E0DA5C6C8D10475D8CCB5A67A82AF36B","parentFileId":"62a7940586faa64090b54403957283843c3142e2","crc64Hash":"10901026750831965725","sha1Hash":"7BAB5FEC109D003DB0CFC5420A0CDB56E3BD6AE5","path":"/Unriad-Photo-Backup/B-Organized/2021/05/20210528_191443_B064F932.png","category":"image","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.737] update local file to db: {"fileName":"20190412_153701_5DD895C2.jpg","fileSize":2101348,"fileType":"file","createdAt":"2022-04-02 00:26:43","updatedAt":"2022-04-02 00:26:43","fileExtension":".jpg","sha1Hash":"6AD8BEFE5CD80B32814D4BAD77D29B715E57BD00","path":"/home/app/data/F-Organized/2019/04/20190412_153701_5DD895C2.jpg","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.741] update pan file to db: {"driveId":"19863","domainId":"bj29","fileId":"62bb6cc52c18d9525c2f4f9dbabc8f7b393dab83","fileName":"20210528_103646_7C277223.png","fileSize":1716033,"fileType":"file","createdAt":"2022-06-29 05:04:05","updatedAt":"2022-06-29 05:04:10","fileExtension":"png","uploadId":"34054DC1A30D4B3DA7E6CD45DEA11EEA","parentFileId":"62a7940586faa64090b54403957283843c3142e2","crc64Hash":"6198659529789535939","sha1Hash":"436CEF27D16362896F36BAFA1EDD4F31A8674ED4","path":"/Unriad-Photo-Backup/B-Organized/2021/05/20210528_103646_7C277223.png","category":"image","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.749] update local file to db: {"fileName":"20190412_153704_DECA06C9.jpg","fileSize":2167085,"fileType":"file","createdAt":"2022-04-02 00:26:46","updatedAt":"2022-04-02 00:26:46","fileExtension":".jpg","sha1Hash":"82C21BAB832AEE5FEE967520BA11D8F8514AFFC6","path":"/home/app/data/F-Organized/2019/04/20190412_153704_DECA06C9.jpg","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.751] update pan file to db: {"driveId":"19863","domainId":"bj29","fileId":"62bb69c59d49310a97d2415a9f366ec2b9cb730d","fileName":"20210528_020430_8AC2F2FD.mp4","fileSize":1348358,"fileType":"file","createdAt":"2022-06-29 04:51:17","updatedAt":"2022-06-29 04:51:20","fileExtension":"mp4","uploadId":"46AF3D08D78F4198A7754FE082F3EDEA","parentFileId":"62a7940586faa64090b54403957283843c3142e2","crc64Hash":"14525150562902578260","sha1Hash":"E02BF64399DE9410563CB709A72C315573706B55","path":"/Unriad-Photo-Backup/B-Organized/2021/05/20210528_020430_8AC2F2FD.mp4","category":"video","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.761] update local file to db: {"fileName":"20190412_153705_1B88B5D3.jpg","fileSize":2216649,"fileType":"file","createdAt":"2022-04-02 00:26:49","updatedAt":"2022-04-02 00:26:49","fileExtension":".jpg","sha1Hash":"E918726187E3539E7824EE4340619E61FECF1A4A","path":"/home/app/data/F-Organized/2019/04/20190412_153705_1B88B5D3.jpg","scanTimeAt":"2022-09-08 15:40:48","scanStatus":"normal"} [2022-9-8 15:40:48.765] update pan file to db: {"driveId":"19863","domainId":"bj29","fileId":"62bb687580a548ee5f0e4ca78465edb6690b3c08","fileName":"20210525_185849_8E58756C.png","fileSize":158800,"fileType":"file","createdAt":"2022-06-29 04:45:41","updatedAt":"2022-06-29 04:45:55","fileExtension":"png","uploadId":"E40241B860344B1
你这个备份目录下文件数量是不是很多呀？大概有多少个？ 完成首次备份后，CPU还是很高的么？
是的，是我历史所有的照片文件，按年、月自动分组好了，大概一万以上的照片。还没有首次备份完，因为我的之前用其它工具备份过，但是工具还是会自动上传已经存在的文件，不能跳过。上传速度也不快，2秒一张左右的速度在传，我的上行带宽10s/M的速度，也不至于这么难。
是需要带什么 参数给程序才会自动跳过已存在的文件吗？
CPU高的问题，这个之前逻辑有点问题，已经修复 历史数据同步的，这个增加扫描文件的步骤，你可以先扫描好文件，然后再进行同步任务的启动。这样就可以自动跳过已经存在网盘的文件了。
这个两个修复会在下个版本合入，预计这两天会发布，到时候帮忙验证下。
还没有首次备份完，因为我的之前用其它工具备份过，但是工具还是会自动上传已经存在的文件，不能跳过。
还没有首次备份完，因为我的之前用其它工具备份过，但是工具还是会自动上传已经存在的文件，不能跳过。 ==》你可以先使用scan建立同步数据库，然后在启动同步命令，这样就会能精确跳过你之前用其他工具备份到云盘的文件
例如： 当你本地同步目录文件非常多，或者云盘同步目录文件非常多，为了后期更快更精准同步文件，可以先进行文件扫描并构建同步数据库，然后再正常启动同步任务。如下所示：
aliyunpan sync start -step scan
aliyunpan sync start
上传速度问题以后时间再优化
`扫描本地文件：/home/app/data/B-Organized/2013/04/20130428_081727_BAD3953A.jpg 扫描本地文件：/home/app/data/B-Organized/2013/04/20130428_081727_F50DDD94.jpg 扫描本地文件：/home/app/data/B-Organized/2013/04/20130428_081728_1CA9A381.jpg 扫描本地文件：/home/app/data/B-Organized/2013/04/20130428_081728_6DE787E2.jpg 扫描本地文件：/home/app/data/B-Organized/2013/04/20130428_081732_2DF7BE5C.jpg 扫描云盘文件：/Unriad-Photo-Backup/F-Organized/2020/11 扫描云盘文件：/Unriad-Photo-Backup/F-Organized/2020/03 扫描云盘文件：/Unriad-Photo-Backup/F-Organized/2020/02 扫描云盘文件：/Unriad-Photo-Backup/F-Organized/2020/01 扫描云盘文件：/Unriad-Photo-Backup/F-Organized/2020/08 扫描云盘文件：/Unriad-Photo-Backup/F-Organized/2020/09 扫描云盘文件：/Unriad-Photo-Backup/F-Organized/2020/10 扫描云盘文件：/Unriad-Photo-Backup/F-Organized/2020/12 扫描云盘文件：/Unriad-Photo-Backup/F-Organized/2020/05 扫描云盘文件：/Unriad-Photo-Backup/F-Organized/2020/07 扫描云盘文件：/Unriad-Photo-Backup/F-Organized/2020/04 扫描云盘文件：/Unriad-Photo-Backup/F-Organized/2020/06 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x1d0 pc=0xaa9b44]
goroutine 7 [running]: github.com/tickstep/bolt.(*DB).beginRWTx(0x0, 0x0, 0x0, 0x0) /Users/tickstep/Applications/go/pkg/mod/github.com/tickstep/[email protected]/db.go:506 +0x44 github.com/tickstep/bolt.(*DB).Begin(0x0, 0x1, 0xc0003b76c0, 0x3f, 0xc00031e180) /Users/tickstep/Applications/go/pkg/mod/github.com/tickstep/[email protected]/db.go:461 +0x38 github.com/tickstep/aliyunpan/internal/syncdrive.(*BoltDb).Update(0xc00048df00, 0xc0003b76c0, 0x3f, 0xc00031e300, 0x162, 0x0, 0x0, 0x0) /Users/tickstep/Documents/Workspace/go/projects/aliyunpan/internal/syncdrive/bolt_db.go:302 +0xe6 github.com/tickstep/aliyunpan/internal/syncdrive.(*LocalSyncDbBolt).Update(0xc0000705c0, 0xc000276e60, 0x0, 0x0, 0x0) /Users/tickstep/Documents/Workspace/go/projects/aliyunpan/internal/syncdrive/sync_db_bolt.go:333 +0x26d github.com/tickstep/aliyunpan/internal/syncdrive.(*SyncTask).scanLocalFile(0xc0000d9a20, 0xdba258, 0xc00007c5c0, 0xc00034b201) /Users/tickstep/Documents/Workspace/go/projects/aliyunpan/internal/syncdrive/sync_task.go:493 +0x8da created by github.com/tickstep/aliyunpan/internal/syncdrive.(*SyncTask).Start /Users/tickstep/Documents/Workspace/go/projects/aliyunpan/internal/syncdrive/sync_task.go:185 +0x1d7`
使用-step scan 创建同步数据库时，内存溢出。
然后，我依旧是update模式，目前观察没有出现覆盖以前同步过的文件的情况。但是磁盘写入还是非常高，30MB/s，且上传期间资源消耗也非常大。
我可以提供运行日志，请问需要提供哪些类型的日志给你分析问题？
补充一些信息，我的上传备份目录
名称: Photo 位置: disk2 最后修改时间: 9 个月前 总占用空间: 299 GB in 232 folders and 54,270 files
文件结构是"根目录/年/月/files"
开了debug信息看了一下，满屏都在跑截图的日志。观察了一下，是不是每次启动，或者每次上传/同步，程序都会去全量写数据库呢？只要开启，我的docker缓存磁盘写入就非常高。
