reva icon indicating copy to clipboard operation
reva copied to clipboard

Add a jsoncs3 share manager

Open aduffeck opened this issue 3 years ago • 1 comments

This PR adds a new jsoncs3 share manager which splits the json file per storage space and caches data locally.

aduffeck avatar Aug 11 '22 08:08 aduffeck

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

update-docs[bot] avatar Aug 11 '22 08:08 update-docs[bot]

adding a stat afted upload introduces a lost read. That is not the right solution. I dropped the stat and ran the ginko tests locally until they failed: ginkgo --until-it-fails pkg/share/manager/jsoncs3

Running Suite: Json Suite - /home/vscode/repositories/reva/pkg/share/manager/jsoncs3
====================================================================================
Random Seed: 1661158281

Will run 39 of 39 specs
•••••••••••••••••••••••••••••••••
------------------------------
• [FAILED] [0.027 seconds]
Jsoncs3
/home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:45
  with an existing share
  /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:249
    GetReceivedShare
    /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:761
      with a group share [BeforeEach]
      /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:790
        syncs the cache
        /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:806

  Unexpected error:
      <errtypes.PreconditionFailed>: resource has been modified
      error: precondition failed: resource has been modified
  occurred
  In [BeforeEach] at: /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:793
------------------------------
•••••

Ran 39 of 39 Specs in 0.241 seconds
FAIL! -- 38 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestJson (0.24s)
FAIL

Tests failed on attempt #494


Ginkgo ran 1 suite in 59.350250234s

Test Suite Failed

the second run took 1093 attempts ... for the same error.

the third stopped at attempt no 8:

------------------------------
• [FAILED] [0.008 seconds]
Jsoncs3
/home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:45
  with an existing share
  /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:249
    UpdateShare
    /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:439
      [It] updates an existing share
      /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:455

  Expected
      <bool>: false
  to be true
  In [It] at: /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:476
------------------------------

and after 3410 attempts

------------------------------
• [FAILED] [0.003 seconds]
Jsoncs3
/home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:45
  with an existing share
  /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:249
    ListReceivedShares
    /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:627
      with a group share [BeforeEach]
      /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:711
        syncronizes the group received cache before listing
        /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:728

  Unexpected error:
      <errtypes.PreconditionFailed>: resource has been modified
      error: precondition failed: resource has been modified
  occurred
  In [BeforeEach] at: /home/vscode/repositories/reva/pkg/share/manager/jsoncs3/jsoncs3_test.go:714
------------------------------

In a redundant setup the persist call may fail at any time. especially for the providercache when two users create or update a share in the same space at the same time.

AFAICT when a persist fails the code should try to download the correspondting file, reapply the changes end retry writing.

🤔

butonic avatar Aug 22 '22 08:08 butonic

adding a syscall.Sync() at the beginning of metadata disk.go Upload() makes the test fail more reliably. I think using the oldMtime in the IfUnmodifiedSince header is wrong ... it will never change for a service ...

butonic avatar Aug 22 '22 10:08 butonic

This pull request introduces 2 alerts when merging 285e35cc16816ad49073599ca49a5ab0464150bf into 15d06664261445641ad1d5bf341faae1e2ad49b1 - view on LGTM.com

new alerts:

  • 2 for Useless assignment to local variable

lgtm-com[bot] avatar Aug 22 '22 11:08 lgtm-com[bot]

This pull request introduces 2 alerts when merging 18f2d669d470ac9df7035601f0b1dc2851476be9 into 15d06664261445641ad1d5bf341faae1e2ad49b1 - view on LGTM.com

new alerts:

  • 2 for Useless assignment to local variable

lgtm-com[bot] avatar Aug 22 '22 12:08 lgtm-com[bot]

This pull request introduces 2 alerts when merging 9f3c7b3f783dd19df551015fda8b2873ffa86f4f into 15d06664261445641ad1d5bf341faae1e2ad49b1 - view on LGTM.com

new alerts:

  • 2 for Useless assignment to local variable

lgtm-com[bot] avatar Aug 22 '22 12:08 lgtm-com[bot]

This pull request introduces 2 alerts when merging b41b20fa3c08fd20969733154c38e285e45de55f into 15d06664261445641ad1d5bf341faae1e2ad49b1 - view on LGTM.com

new alerts:

  • 2 for Useless assignment to local variable

lgtm-com[bot] avatar Aug 22 '22 13:08 lgtm-com[bot]

flaky concurrency test tracked in https://github.com/cs3org/reva/issues/3166

butonic avatar Aug 22 '22 13:08 butonic

This pull request introduces 2 alerts when merging 5b7a0945b73d44d444f0a358c22d3c5fde4a96ca into 15d06664261445641ad1d5bf341faae1e2ad49b1 - view on LGTM.com

new alerts:

  • 2 for Useless assignment to local variable

lgtm-com[bot] avatar Aug 22 '22 13:08 lgtm-com[bot]

This pull request introduces 1 alert when merging dc166513562871c50c6302ad4fbefa4668052d00 into 15d06664261445641ad1d5bf341faae1e2ad49b1 - view on LGTM.com

new alerts:

  • 1 for Useless assignment to local variable

lgtm-com[bot] avatar Aug 22 '22 14:08 lgtm-com[bot]