feat(services/pcloud): setup test for pcloud

[hoslo]

[tisonkun]

Please avoid request a large range of reviewers which cause noise.

IIRC we have a script to randomly pick up reviewers from volunteers already.

[Xuanwo]

Please avoid request a large range of reviewers which cause noise.

I'm guessing reviewers are requested as code owner.

[Zheaoli]

Please avoid request a large range of reviewers which cause noise.

IIRC we have a script to randomly pick up reviewers from volunteers already.

Code owners are requested for review automatically. Also we have two random reviewer for each PR

[Zheaoli]

Python binding part LGTM. But I think there may get some throttle here, so we can't run 4 pcloud test success as the same time

[Xuanwo]

Python binding part LGTM. But I think there may get some throttle here, so we can't run 4 pcloud test success as the same time

I believe they are bugs to fix instead of throttle.

[Zheaoli]

I believe they are bugs to fix instead of throttle.

The throttle may come from the pcloud, not ours. Need to dive into the action failure

[Xuanwo]

The throttle may come from the pcloud, not ours. Need to dive into the action failure

Could you share the clues that led you to believe it's related to the throttle?

The errors I found in logs are just bugs:

---- behavior::test_delete_file ----
test panicked: assertion failed: !op.is_exist(&path).await?

---- behavior::test_list_dir_with_metakey ----
test panicked: file should be found in list

---- behavior::test_list_dir_with_metakey_complete ----
test panicked: file should be found in list

---- behavior::test_list_prefix ----
test panicked: assertion `left == right` failed
  left: 0
 right: 1

---- behavior::test_blocking_read_full ----
Unexpected (permanent) at read, context: { service: pcloud, path: 220f33bd-4bda-439a-98f3-552c4605e1ef, range: 0-2139880 } => GetFileLinkResponse { result: 2009, path: None, hosts: None }

---- behavior::test_blocking_rename_file ----
NotFound (persistent) at rename, context: { service: pcloud, from: d260fb1f-2b36-43b1-b8d4-b9e0e2647ef8, to: fedaf28d-d51a-4539-826a-fc21cca69d01 } => PcloudError { result: 2009, error: Some("File not found."), .. }

[hoslo]

@Xuanwo I didn't find a place for the throttle, but I ran the behavior test locally separately and it was ok.

running 123 tests
test behavior::test_copy_source_dir                         ... ok
test behavior::test_copy_non_existing_source                ... ok
test behavior::test_copy_self                               ... ok
test behavior::test_create_dir                              ... ok
test behavior::test_copy_target_dir                         ... ok
test behavior::test_delete_empty_dir                        ... ok
test behavior::test_create_dir_existing                     ... ok
test behavior::test_delete_file                             ... ok
test behavior::test_delete_not_existing                     ... ok
test behavior::test_fuzz_issue_2717                         ... ok
test behavior::test_fuzz_pr_3395_case_1                     ... ok
test behavior::test_fuzz_pr_3395_case_2                     ... ok
test behavior::test_copy_overwrite                          ... ok
test behavior::test_copy_nested                             ... ok
test behavior::test_check                                   ... ok
test behavior::test_delete_with_special_chars               ... ok
test behavior::test_copy_file_with_ascii_name               ... ok
test behavior::test_copy_file_with_non_ascii_name           ... ok
test behavior::test_list_prefix                             ... ok
test behavior::test_list_dir_with_metakey_complete          ... ok
test behavior::test_list_dir_with_metakey                   ... ok
test behavior::test_list_non_exist_dir                      ... ok
test behavior::test_remove_one_file                         ... ok
test behavior::test_list_with_start_after                   ... ok
test behavior::test_list_dir                                ... ok
test behavior::test_list_sub_dir                            ... ok
test behavior::test_list_nested_dir                         ... ok
test behavior::test_list_dir_with_file_path                 ... ok
test behavior::test_read_range                              ... ok
test behavior::test_read_large_range                        ... ok
test behavior::test_reader_range                            ... ok
test behavior::test_reader_range_with_buffer                ... ok
test behavior::test_reader_from                             ... ok
test behavior::test_reader_from_with_buffer                 ... ok
test behavior::test_reader_tail                             ... ok
test behavior::test_reader_tail_with_buffer                 ... ok
test behavior::test_read_not_exist                          ... ok
test behavior::test_read_with_if_match                      ... ok
test behavior::test_read_with_if_none_match                 ... ok
test behavior::test_read_with_dir_path                      ... ok
test behavior::test_list_root_with_recursive                ... ok
test behavior::test_read_with_override_cache_control        ... ok
test behavior::test_read_with_override_content_disposition  ... ok
test behavior::test_read_with_override_content_type         ... ok
test behavior::test_list_empty_dir                          ... ok
test behavior::test_read_full                               ... ok
test behavior::test_read_with_invalid_seek                  ... ok
test behavior::test_rename_non_existing_source              ... ok
test behavior::test_rename_source_dir                       ... ok
test behavior::test_read_with_special_chars                 ... ok
test behavior::test_rename_self                             ... ok
test behavior::test_list_with_recursive                     ... ok
test behavior::test_rename_target_dir                       ... ok
test behavior::test_stat_file                               ... ok
test behavior::test_stat_dir                                ... ok
test behavior::test_rename_file                             ... ok
test behavior::test_rename_nested                           ... ok
test behavior::test_stat_not_cleaned_path                   ... ok
test behavior::test_stat_with_if_match                      ... ok
test behavior::test_stat_with_if_none_match                 ... ok
test behavior::test_stat_with_override_cache_control        ... ok
test behavior::test_stat_with_override_content_disposition  ... ok
test behavior::test_stat_with_override_content_type         ... ok
test behavior::test_stat_root                               ... ok
test behavior::test_stat_nested_parent_dir                  ... ok
test behavior::test_write_with_empty_content                ... ok
test behavior::test_write_with_dir_path                     ... ok
test behavior::test_stat_not_exist                          ... ok
test behavior::test_write_with_cache_control                ... ok
test behavior::test_write_with_content_type                 ... ok
test behavior::test_write_with_content_disposition          ... ok
test behavior::test_writer_write                            ... ok
test behavior::test_writer_write_with_concurrent            ... ok
test behavior::test_writer_sink                             ... ok
test behavior::test_writer_sink_with_concurrent             ... ok
test behavior::test_writer_copy                             ... ok
test behavior::test_writer_copy_with_concurrent             ... ok
test behavior::test_stat_with_special_chars                 ... ok
test behavior::test_writer_abort                            ... ok
test behavior::test_writer_futures_copy                     ... ok
test behavior::test_writer_futures_copy_with_concurrent     ... ok
test behavior::test_writer_abort_with_concurrent            ... ok
test behavior::test_blocking_copy_non_existing_source       ... ok
test behavior::test_write_with_special_chars                ... ok
test behavior::test_write_only                              ... ok
test behavior::test_remove_all                              ... ok
test behavior::test_blocking_copy_source_dir                ... ok
test behavior::test_rename_overwrite                        ... ok
test behavior::test_blocking_copy_self                      ... ok
test behavior::test_blocking_copy_target_dir                ... ok
test behavior::test_blocking_create_dir                     ... ok
test behavior::test_blocking_copy_file                      ... ok
test behavior::test_blocking_delete_file                    ... ok
test behavior::test_blocking_create_dir_existing            ... ok
test behavior::test_blocking_copy_nested                    ... ok
test behavior::test_blocking_remove_one_file                ... ok
test behavior::test_blocking_list_non_exist_dir             ... ok
test behavior::test_blocking_copy_overwrite                 ... ok
test behavior::test_blocking_list_dir                       ... ok
test behavior::test_blocking_read_range                     ... ok
test behavior::test_blocking_read_large_range               ... ok
test behavior::test_blocking_read_not_exist                 ... ok
test behavior::test_blocking_list_dir_with_metakey_complete ... ok
test behavior::test_blocking_list_dir_with_metakey          ... ok
test behavior::test_blocking_rename_non_existing_source     ... ok
test behavior::test_blocking_rename_source_dir              ... ok
test behavior::test_blocking_read_full                      ... ok
test behavior::test_blocking_rename_target_dir              ... ok
test behavior::test_blocking_rename_file                    ... ok
test behavior::test_blocking_stat_file                      ... ok
test behavior::test_blocking_rename_nested                  ... ok
test behavior::test_blocking_stat_dir                       ... ok
test behavior::test_blocking_stat_not_exist                 ... ok
test behavior::test_blocking_stat_with_special_chars        ... ok
test behavior::test_blocking_write_with_dir_path            ... ok
test behavior::test_blocking_scan                           ... ok
test behavior::test_blocking_rename_overwrite               ... ok
test behavior::test_blocking_write_file                     ... ok
test behavior::test_blocking_write_with_special_chars       ... ok
test behavior::test_blocking_remove_all                     ... ok
test behavior::test_blocking_rename_self                    ... ok
test behavior::test_delete_stream                           ... ok
test behavior::test_list_rich_dir                           ... ok

test result: ok. 123 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 472.91s

[Xuanwo]

Take test async_copy as an example:

  2024-01-25T04:16:32.466966Z DEBUG opendal::services: service=pcloud operation=copy from=2ef78cef-304b-44d2-901e-b73547de22a1 to=f74a7d1d-5e3c-470d-b483-9e3283d5dd89/7cd74ba0-296c-41b1-9836-b67a1d35921f/0c4541e6-0d0d-4309-ad28-4b47566e2062 -> finished
    at core/src/layers/logging.rs:385

  2024-01-25T04:16:32.467052Z DEBUG opendal::services: service=pcloud operation=stat path=f74a7d1d-5e3c-470d-b483-9e3283d5dd89/7cd74ba0-296c-41b1-9836-b67a1d35921f/0c4541e6-0d0d-4309-ad28-4b47566e2062 -> started
    at core/src/layers/logging.rs:454

  2024-01-25T04:16:32.669307Z DEBUG opendal::services: service=pcloud operation=stat path=f74a7d1d-5e3c-470d-b483-9e3283d5dd89/7cd74ba0-296c-41b1-9836-b67a1d35921f/0c4541e6-0d0d-4309-ad28-4b47566e2062 -> finished: RpStat { meta: Metadata { metakey: FlagSet(Complete | Mode | ContentLength | LastModified), mode: FILE, cache_control: None, content_disposition: None, content_length: Some(2442332), content_md5: None, content_range: None, content_type: None, etag: None, last_modified: Some(2024-01-25T04:16:32Z), version: None } }
    at core/src/layers/logging.rs:466

...

  2024-01-25T04:16:32.669437Z DEBUG opendal::services: service=pcloud operation=read path=f74a7d1d-5e3c-470d-b483-9e3283d5dd89/7cd74ba0-296c-41b1-9836-b67a1d35921f/0c4541e6-0d0d-4309-ad28-4b47566e2062 range=0-2442331 -> started
    at core/src/layers/logging.rs:287

  2024-01-25T04:16:32.669480Z DEBUG opendal::services: service=pcloud operation=read path=f74a7d1d-5e3c-470d-b483-9e3283d5dd89/7cd74ba0-296c-41b1-9836-b67a1d35921f/0c4541e6-0d0d-4309-ad28-4b47566e2062 range=0-2442331 -> got reader
    at core/src/layers/logging.rs:302

  2024-01-25T04:16:32.669839Z TRACE opendal::services: service=pcloud operation=Reader::read path=f74a7d1d-5e3c-470d-b483-9e3283d5dd89/7cd74ba0-296c-41b1-9836-b67a1d35921f/0c4541e6-0d0d-4309-ad28-4b47566e2062 read=0 -> buf size: 2442332B, read pending
    at core/src/layers/logging.rs:1030

  2024-01-25T04:16:32.672076Z TRACE opendal::services: service=pcloud operation=Reader::read path=f74a7d1d-5e3c-470d-b483-9e3283d5dd89/7cd74ba0-296c-41b1-9836-b67a1d35921f/0c4541e6-0d0d-4309-ad28-4b47566e2062 read=0 -> buf size: 2442332B, read pending
    at core/src/layers/logging.rs:1030

  2024-01-25T04:16:32.707461Z TRACE opendal::services: service=pcloud operation=Reader::read path=f74a7d1d-5e3c-470d-b483-9e3283d5dd89/7cd74ba0-296c-41b1-9836-b67a1d35921f/0c4541e6-0d0d-4309-ad28-4b47566e2062 read=0 -> buf size: 2442332B, read pending
    at core/src/layers/logging.rs:1030

  2024-01-25T04:16:32.742807Z TRACE opendal::services: service=pcloud operation=Reader::read path=f74a7d1d-5e3c-470d-b483-9e3283d5dd89/7cd74ba0-296c-41b1-9836-b67a1d35921f/0c4541e6-0d0d-4309-ad28-4b47566e2062 read=0 -> buf size: 2442332B, read pending
    at core/src/layers/logging.rs:1030

  2024-01-25T04:16:32.742978Z TRACE opendal::services: service=pcloud operation=Reader::read path=f74a7d1d-5e3c-470d-b483-9e3283d5dd89/7cd74ba0-296c-41b1-9836-b67a1d35921f/0c4541e6-0d0d-4309-ad28-4b47566e2062 read=0 -> buf size: 2442332B, read pending
    at core/src/layers/logging.rs:1030


  2024-01-25T04:16:32.892685Z ERROR opendal::services: service=pcloud operation=Reader::read path=f74a7d1d-5e3c-470d-b483-9e3283d5dd89/7cd74ba0-296c-41b1-9836-b67a1d35921f/0c4541e6-0d0d-4309-ad28-4b47566e2062 read=0 -> read failed: Unexpected (permanent) at read => GetFileLinkResponse { result: 2009, path: None, hosts: None }

thread '<unnamed>' panicked at core/tests/behavior/async_copy.rs:180:54:
read must succeed: Unexpected (permanent) at read => GetFileLinkResponse { result: 2009, path: None, hosts: None }

Context:
   service: pcloud
   path: f74a7d1d-5e3c-470d-b483-9e3283d5dd89/7cd74ba0-296c-41b1-9836-b67a1d35921f/0c4541e6-0d0d-4309-ad28-4b47566e2062
   range: 0-2442331

Backtrace:
   0: opendal::types::error::Error::new
             at ./src/types/error.rs:338:24
   1: opendal::services::pcloud::core::PcloudCore::get_file_link::{{closure}}
             at ./src/services/pcloud/core.rs:95:32
   2: <opendal::services::pcloud::backend::PcloudBackend as opendal::raw::accessor::Accessor>::read::{{closure}}
             at ./src/services/pcloud/backend.rs:299:50

The copy returns ok, stat returns it's metadata, but read returns 2009 which means file not found. Do you have idea why this happened here? Do we need to retry it?

[Xuanwo]

And is it a good idea for us to use file_read and file_pread instead?

ref: https://docs.pcloud.com/methods/fileops/index.html

[hoslo]

And is it a good idea for us to use file_read and file_pread instead?

ref: https://docs.pcloud.com/methods/fileops/index.html

The implementation of upload and download is based on the official javascript sdk https://github.com/pCloud/pcloud-sdk-js

[hoslo]

@Xuanwo I have encountered an issue while implementing Upyun services. After uploading a file, immediately requesting the file results in a 'Not Found' response. The same issue occurs with the list api. I'm not sure if pcloud has a similar issue, Because I can't reproduce it locally.

[Xuanwo]

pcloud said those in docs: https://docs.pcloud.com/protocols/http_json_protocol/single_connection.html

The API server will not be able to respond correctly to two interleaved requests and data corruption may occur. While writes of small amount of data on a socket MIGHT be atomic on some operating systems it is preferable to use locks or dedicated thread responsible for all the reading/writing to the socket.

Seems pcloud doesn't support writing data concurrently.

[hoslo]

Seems pcloud doesn't support writing data concurrently.

So, should we change Arc<PcloudCore> to Arc<Mutex<PcloudCore>>?

[Xuanwo]

So, should we change Arc<PcloudCore> to Arc<Mutex<PcloudCore>>?

I'm not sure about that. Maybe we can drop copy and rename support of pcloud first?

[hoslo]

So, should we change Arc<PcloudCore> to Arc<Mutex<PcloudCore>>?

I'm not sure about that. Maybe we can drop copy and rename support of pcloud first?

It doesn't seem to work either.

[Xuanwo]

I will take a look into this today.

[Xuanwo]

@Xuanwo I have encountered an issue while implementing Upyun services. After uploading a file, immediately requesting the file results in a 'Not Found' response. The same issue occurs with the list api. I'm not sure if pcloud has a similar issue, Because I can't reproduce it locally.

I now agree with your observation that pcloud has some issues with metadata consistency. plcoud service seems to work but can't pass our behavior tests. We might need help from the plcoud.

[hoslo]

@Xuanwo I have encountered an issue while implementing Upyun services. After uploading a file, immediately requesting the file results in a 'Not Found' response. The same issue occurs with the list api. I'm not sure if pcloud has a similar issue, Because I can't reproduce it locally.

I now agree with your observation that pcloud has some issues with metadata consistency. plcoud service seems to work but can't pass our behavior tests. We might need help from the plcoud.

Yes, Upyun also has such an issue, and it is easily reproducible.

[Xuanwo]

Yes, Upyun also has such an issue, and it is easily reproducible.

I have sent ticket to pcloud (which cc your ASF email too). Let's work with them to find out how to address this issue.

Converting this PR to draft now.

[Xuanwo]

Hi, @hoslo, thanks a lot for your work! I'm afraid that there's nothing we can do for pcloud. I created an issue to track this: https://github.com/apache/opendal/issues/4441 instead.

I'm going to close this PR, and will come back once pcloud is ready.