paws-r
list_objects & list_objects_v2 : objects missing
I was doing some comparisons between jupyter and r notebooks while accessing S3 and found out paws currently has issues listing files:
With prefix "A/B", I get 0 R files :
But with prefix "A/B/C", I get multiple R files (indeed supposed to be there):
sessionInfo():
R version 4.1.1 (2021-08-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.3 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8
[4] LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] digest_0.6.27 paws.common_0.3.12 R6_2.5.1 evaluate_0.14
[5] httr_1.4.2 rlang_0.4.11 curl_4.3.2 paws_0.1.11
[9] xml2_1.3.2 rmarkdown_2.10 tools_4.1.1 paws.storage_0.1.12
[13] xfun_0.25 yaml_2.2.1 fastmap_1.1.0 compiler_4.1.1
[17] htmltools_0.5.2 knitr_1.33
Sorry about that. Can you give more details about the key names? I am not able to reproduce the issue using names exactly like "A/B/C/1.R". The only difference in my software versions is I am using R 4.1.0 instead of 4.1.1, and I have paws.common 0.3.13 instead of 0.3.12, but this has only two minor unrelated changes.
On Thu, Oct 21, 2021 at 2:43 PM lpalbou @.***> wrote:
I was doing some comparisons between jupyter and r notebooks while accessing S3 and found out paws currently has issues listing files:
With prefix "A/B", I get 0 R files : [image: Capture d’écran 2021-10-21 à 20 40 37] https://user-images.githubusercontent.com/24249870/138337456-a36e8493-4bd6-420c-9ce3-ce3877b88321.png
But with prefix "A/B/C", I get multiple R files (indeed supposed to be there):
[image: Capture d’écran 2021-10-21 à 20 40 19] https://user-images.githubusercontent.com/24249870/138337418-a6593fca-fd3c-4aab-8632-13d644cc33ba.png
sessionInfo():
R version 4.1.1 (2021-08-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.3 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8
[4] LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] digest_0.6.27 paws.common_0.3.12 R6_2.5.1 evaluate_0.14
[5] httr_1.4.2 rlang_0.4.11 curl_4.3.2 paws_0.1.11
[9] xml2_1.3.2 rmarkdown_2.10 tools_4.1.1 paws.storage_0.1.12
[13] xfun_0.25 yaml_2.2.1 fastmap_1.1.0 compiler_4.1.1
[17] htmltools_0.5.2 knitr_1.33
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/paws-r/paws/issues/465, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAV3AQKKWHXQRTK3UT35GDUIBNNNANCNFSM5GOZ4UUQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
Hi David, unfortunately I can't share the full keys for confidentiality reasons. But it seems there are some issues with the recursivity while looking for keys under different "sub-folders" (sub-prefixes since there are no folders in S3).
I admit it's intriguing as it doesn't happen all the time. It has nothing to do with the .endsWith("R") but it was my test to limit the number of files. So the issue is definitely in list_objects_v2 and in the use of "Prefix" param.
This is now fixed in paws.common version 0.3.15 on CRAN, which you can install with install.packages("paws.common"). Thank you!