support for baidu pan (1 TB free)

[avigard]

Hello guys!

Old post 2018

Baidu used to offer 2TB free cloud storage. I registered lately and it is "just" 1TB now. Nevertheless it's 1 terabyte free cloud space. I wouldn't store my stuff on it unencrypted because it's a chinese provider but with rclone encryption i would love to use it.

I also found a go library for baidu netdisk

https://github.com/suconghou/netdisk

Update 2021

Maintained BaiduPCS go client (confirmed to work by @ericma15) - https://github.com/qjfoidnh/BaiduPCS-Go

Unfinished backend code for rclone - https://github.com/Cnly/rclone/commits/baidupcs

Important note by @Cnly

In a word, the users will be always forced to purchase a premium plan to have an acceptable transfer speed. Not saying that its payment method only targeted Chinese users (Baidu baifubao, Alipay, Wechat Pay), unfriendly for the majority.

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[ncw]

Is baidu netdisk different from baidu pan?

[avigard]

No, I think pan means something like disk or cloud. There were other chinese cloud providers like http://www.kuaipan.cn/ and https://yunpan.360.cn/, but the first is no more and the seconde isn't free, anymore.

[franklndh]

Yes please, im user of baidu, i have 1TB free and use it for replication, I have the same files of my gdrive account.

[ivan98]

According to Baidu Annoucement, effective 08 Sep 2015, no new developer accounts are allowed; existing accounts can continue to be used; Baidu developer site will no longer contain PCS related services. --- as translated from --- “个人云存储(PCS)”：不再支持新用户接入，老用户可继续使用；9月8日起，百度开发者中心不再提供个人云存储(PCS)相关链接和服务。

[avigard]

That is very unfortunate

[ivan98]

Nevertheless, I think rclone can work with Baidu PCS the same way rclone works with Amazon Cloud. I notice rclone do not supply its own client-id for Amazon Cloud, but to ask the user for one.

[jianershi]

that will be extremely sweet if possible.

[Cnly]

Based on people's remarks, it seems https://github.com/iikira/BaiduPCS-Go is a working and actively maintained golang CLI client for BaiduPCS. But it should be noted that this client probably works by mocking Baidu's official clients, rather than by using documented APIs, etc.

[ncw]

@Cnly - it appears to have an API abstraction too which says all the right things for rclone use in it.

But it should be noted that this client probably works by mocking Baidu's official clients, rather than by using documented APIs, etc.

It wouldn't be the first rclone backend to do that (Mega, Jottacloud).

[Cnly]

@ncw I've had a look at its code. You're right that there is an API abstraction, but some logic (E.g. how to login) is written in the command part. So if we want to use it, we may need to convert some code. Not much though, it seems.

Baidu offers 1 TB or 2 for many users for free, and the speed is pretty good. Maybe this backend is worth adding. This backend also has some interesting features, for example, it allows users to add files to their accounts by specifying some hashes of those files. If those hashes match some files already on the server, no upload is required.

[ncw]

@Cnly it looks possible to do a backend certainly - do you want to work on it?

This backend also has some interesting features, for example, it allows users to add files to their accounts by specifying some hashes of those files. If those hashes match some files already on the server, no upload is required.

So instead of uploading the file you say I've got file length x, md5sum y and they say OK and stuff it in your account? Sounds interesting if a little scary!

[Cnly]

it looks possible to do a backend certainly - do you want to work on it?

Would like to have a try, but not sure how long this will take. I'll keep things in my baidupcs branch.

So instead of uploading the file you say I've got file length x, md5sum y and they say OK and stuff it in your account?

That's it. Here's what's actually needed: https://github.com/iikira/BaiduPCS-Go/blob/master/baidupcs/prepare.go#L268

Another interesting feature would be remote download, which allows you to submit a link (HTTP or even magnet, etc.) and ask the server to download the file for you.

[ncw]

Would like to have a try, but not sure how long this will take. I'll keep things in my baidupcs branch.

Great :-)

Another interesting feature would be remote download, which allows you to submit a link (HTTP or even magnet, etc.) and ask the server to download the file for you.

Some of the other backends support that and it is a feature often requested by users! It would require some optional interfaces in rclone I think to support and probably another command or two.

[Cnly]

I have got the very basic things to work now. One thing very important for this backend is that we must find a way in rclone to achieve multithreaded upload/download, since there can be a 100x performance difference. But because Update and Open in rclone are both concerned with readers/writers rather than simple local files, this seems not that easy. :/

[lzjluzijie]

I have written some related code, I don't know if this can help you. (I am just a beginner, the code is able to run but not well written.

https://github.com/lzjluzijie/SecFiles/blob/master/storage/baiduwangpan/baiduwangpan.go

[Cnly]

@lzjluzijie thanks for offering to help! I've had a look at your code, but I'm afraid it could not do much for this particular situation because Update and Open in rclone are both concerned with readers/writers rather than simple local files. I'm looking for a way to parallelize transfers without direct access to local files and I'm testing with some self-implemented in-mem pipes for this currently. Temp files may be another idea, though. Please feel free to speak if you have other good suggestions.

[lzjluzijie]

Sorry I don't know much about rclone. Maybe I have some misunderstandings but I think I use aes streams rather than directly read/write files that should be readers/writers.

https://github.com/lzjluzijie/SecFiles/blob/master/storage/baiduwangpan/baiduwangpan.go#L88

[Cnly]

@lzjluzijie Maybe I should have make it more obvious: the two important interfaces needed for parallelization that rclone's readers and writers don't implement are io.ReaderAt and io.WriterAt. In your code, your upload is not parallelized, so it does not rely on io.ReaderAt but the performance isn't the best. And your download relies on io.WriterAt, which rclone doesn't provide.

[ncw]

Open can be given an offset, or a range which works with all remotes so parallelization is possible. however that doesn't make it all the way into the parameters to Update so isn't useful right now.

[mdstoll]

Is there maybe an update on this issue? I am a premium Baidu user and have the 5 TB account. Would love rclone support if possible! Thanks!

[Cnly]

A sad update: Based on my evaluation on the case for this backend, I will not continue the work on it unless some large helpful changes occur; sorry for having kept everyone waiting. :( If anyone else is interested in working on it, it'll be great and I'll be looking forward to their contribution. Discussions are also welcome.

I stopped working on the backend because of the following reasons:

1. Baidu does not provide official APIs for developers. And even though there are unofficial projects to provide APIs, they are unstable as Baidu is continuously suppressing these projects. They may need frequent updates to keep up with the changes, or some behaviours may need to be changed frequently in order to bypass the suppression.
2. Baidu Pan has a serious speed limitation (~50 kB/s?) for each data transfer thread used by the client, if the user is not on a premium plan. Therefore, in order to reach a reasonable upload/download speed, multithreaded transfer for every single file is essential (unless it is really small). And that will have some conflicts with the current design of rclone.

[ncw]

A sad update: Based on my evaluation on the case for this backend, I will not continue the work on it unless some large helpful changes occur; sorry for having kept everyone waiting. :( If anyone else is interested in working on it, it'll be great and I'll be looking forward to their contribution. Discussions are also welcome.

Thanks for working on it to this point. Is where you've got to available somewhere?

I stopped working on the backend because of the following reasons:

1. Baidu does not provide official APIs for developers.

That is hard work. We have one backend (mega) with no official API and that is quite hard work to keep working, and mega don't try to suppress it, so that is understandable.

2. Baidu Pan has a serious speed limitation (~50 kB/s?) for each data transfer thread used by the client, if the user is not on a premium plan. Therefore, in order to reach a reasonable upload/download speed, multithreaded transfer for every single file is essential (unless it is really small). And that will have some conflicts with the current design of rclone.

Multithreaded uploads are possible (quite a lot of backends do that), rclone isn't laid out for multithreaded downloads at the moment, though I do have some ideas!

[Cnly]

Some 800 lines of code are available on my baidupcs branch, and some others are still on my local machine. The code I've pushed had a very basic working single-threaded framework for the backend, but looking at BaiduPCS-Go's changelog, I'm pretty sure it won't work anymore :( And on my local machine there're some effort and attempts to parallelise things in rclone, which is mainly an RWAtPipe (Reader/Writer-At Pipe). I didn't push that part of code because it was incomplete and I don't think it's responsible to push it. Also, it's been a long time since I worked on it.

P.S. I just saw a notification and discovered https://github.com/ncw/rclone/issues/2252. It would definitely help if that issue gets solved.

[ttimasdf]

Some 800 lines of code are available on my baidupcs branch, and some others are still on my local machine. The code I've pushed had a very basic working single-threaded framework for the backend, but looking at BaiduPCS-Go's changelog, I'm pretty sure it won't work anymore :( And on my local machine there're some effort and attempts to parallelise things in rclone, which is mainly an RWAtPipe (Reader/Writer-At Pipe). I didn't push that part of code because it was incomplete and I don't think it's responsible to push it. Also, it's been a long time since I worked on it.

Thanks for all of your work! I was hoping Baidu netdisk to be a backend for rclone as well. But I don't think it's something worthy to develop at the time being.

First the original project (iikira/BaiduPCS-Go) has stopped its maintenance. And the whole project is relying on PCS API that is not officially supported (the API doc page is 404 now) that could removed at any moment.

And the speed limit is a thing.

Baidu Pan has a serious speed limitation (~50 kB/s?) for each data transfer thread used by the client, if the user is not on a premium plan. Therefore, in order to reach a reasonable upload/download speed, multithreaded transfer for every single file is essential (unless it is really small). And that will have some conflicts with the current design of rclone.

As stated in iikira/BaiduPCS-Go#643, the limit has upgraded to 100kB/s per account. From my (not so intensive) test, the multithreaded download will just fail with no download progress at all. I upgraded a yearly plan and the limit was removed completely. The speed issue will be worse outside of mainland China.

In a word, the users will be always forced to purchase a premium plan to have an acceptable transfer speed. Not saying that its payment method only targeted Chinese users (Baidu baifubao, Alipay, Wechat Pay), unfriendly for the majority.

[juukee]

agree, and i think improve the transfer speed is important for user. rclone will be a better way to use it, then rclone can get more fans and stars.

[raymanzhang]

Baidu Pan reopens their public api now. Everyone can apply for the access via https://pan.baidu.com/union/apply . And there are docments for the api too : https://pan.baidu.com/union/document . Really wish rclone can support it. Thanks.

[jcx70100]

Really wish rclone can support Baidu pan. Thanks.😄

[ivandeex]

I searched for a well starred and recently updated bidu-go project and stumbled upon https://github.com/qjfoidnh/BaiduPCS-Go

@raymanzhang

Baidu Pan reopens their public api now. Everyone can apply for the access via https://pan.baidu.com/union/apply .

Can you run a file upload/download test of the API with https://github.com/qjfoidnh/BaiduPCS-Go ?

@mdstoll

Is there maybe an update on this issue? I am a premium Baidu user and have the 5 TB account. Would love rclone support if possible! Thanks!

Can you run a file upload/download test of your account with https://github.com/qjfoidnh/BaiduPCS-Go ?

p.s. I cleaned up mee tooo comments a little. Please put your :+1: on the top post rather than leaving those :)

[ericma15]

I searched for a well starred and recently updated bidu-go project and stumbled upon https://github.com/qjfoidnh/BaiduPCS-Go

@raymanzhang

Baidu Pan reopens their public api now. Everyone can apply for the access via https://pan.baidu.com/union/apply .

Can you run a file upload/download test of the API with https://github.com/qjfoidnh/BaiduPCS-Go ?

@mdstoll

Is there maybe an update on this issue? I am a premium Baidu user and have the 5 TB account. Would love rclone support if possible! Thanks!

Can you run a file upload/download test of your account with https://github.com/qjfoidnh/BaiduPCS-Go ?

hi，i use this now. It works fine. And i also use a tool named bypy you can find it on github. I’m a premium Baidu user.

[ericma15]

Baidu’s develope manual says,you can only save files in /apps, but baiduPCS can upload to any dir. But Bypy obey this. May be Bypy use official api but baiduPCS not?