vircadia-native-core icon indicating copy to clipboard operation
vircadia-native-core copied to clipboard

Published 20 hours ago • verified publisher icon vircadia

Experiment with Qt HTTP caching options

Open HifiExperiments opened this issue 4 years ago • 24 comments

potentially closes https://github.com/vircadia/vircadia/issues/92 and https://github.com/vircadia/vircadia/issues/990 fixes https://github.com/vircadia/vircadia/issues/1062

should theoretically

  1. remove the need for cachebusting
  2. improve loading times

also adds a reload button next to model URL in create

Test plan/open questions:

  • confirm that cachebusting scripts/models is no longer necessary (links won't update automatically, you'll still need to refresh the script or modelURL).
  • verify that the reload button next to model URL in create works
  • compare loading times (initial/loading from cache) between this PR and master. is this PR faster?
  • does it respect HTTP expiry times? can someone smarter than me make a test for this

HifiExperiments avatar Feb 09 '21 21:02 HifiExperiments

The following links are available: build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-2c4fd58_b8b9cdc5.sh
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-2c4fd58.tar_b8b9cdc5.gz
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-2c4fd58.tar_b8b9cdc5.Z

build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-2c4fd58_b8b9cdc5.dmg

build (ubuntu-18.04, android)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-Quest-Alpha-PR1012-2c4fd58_b8b9cdc5.apk
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-2c4fd58_b8b9cdc5.apk

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-2c4fd58_b8b9cdc5.exe

vircadia-build-notifier avatar Feb 09 '21 23:02 vircadia-build-notifier

So originally when we discussed cachebusting the idea was turned down because of concerns about web servers being set up properly to support that feature. While I was always for cachebusting, has the discussion happened again and the result changed this time?

JulianGro avatar Feb 11 '21 07:02 JulianGro

This is implementing I think exactly what we were talking about. Webservers do have cache information in them, but currently we do not respect that. This gets us a bit closer to respecting that where possible.

digisomni avatar Feb 11 '21 07:02 digisomni

yeah I think this is a pretty standard feature? this PR basically amounts to “use more default Qt values, and use them more correctly”

HifiExperiments avatar Feb 11 '21 07:02 HifiExperiments

The following links are available: build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-3879882_b9b801f8.dmg

build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-3879882.tar_b9b801f8.gz
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-3879882_b9b801f8.sh
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-3879882.tar_b9b801f8.Z

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-3879882_b9b801f8.exe

build (ubuntu-18.04, android)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-3879882_b9b801f8.apk
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-Quest-Alpha-PR1012-3879882_b9b801f8.apk

vircadia-build-notifier avatar Feb 22 '21 09:02 vircadia-build-notifier

Seems to work. A refresh button would definitely be useful. I uploaded a model,then modified is slightly in blender, reuploaded to my S3, then removed the link and repasted and the model. The updated model appeared without cache-busting.

Aitolda avatar Feb 24 '21 04:02 Aitolda

One way to do this would be to have the refresh button empty the field and then paste back in the URL quickly (hacky method). The other method might be to have an actual scripting API call that allows you to just trigger a reload for a resource. Given that the system handles that pretty well...

It's possible that multiple parts of the system are looking at the URL for a change, but I'm hoping that it's simply the case that the resource being refeteched would update all layers. As for falling out of sync with others, it can send a message for others' Create app's to do the same call on that model?

digisomni avatar Feb 24 '21 04:02 digisomni

yeah for now the best way to do this would be to set the URL to "" and then set it back to the actual URL, just to make sure everything picks up on the change. we might just need to copy how it works for the script field? someone who understands create more than me might need to take that over

HifiExperiments avatar Feb 24 '21 04:02 HifiExperiments

Theoretically you could lose the URL if the reload button removes the URL and adds it back. On Linux this happens often enough.

JulianGro avatar Feb 24 '21 11:02 JulianGro

Note: In the Create app, the client script and server entity script fields both have a "reload" button.

ctrlaltdavid avatar Feb 25 '21 19:02 ctrlaltdavid

ok added a reload button, ready for testing

HifiExperiments avatar Mar 21 '21 06:03 HifiExperiments

The following links are available: build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-fee508d-.tar_2fed5124.Z
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-fee508d-.tar_2fed5124.gz
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-fee508d-_2fed5124.sh

build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-fee508d-_2fed5124.dmg

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-fee508d-_2fed5124.exe

build (ubuntu-18.04, android)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-fee508d_2fed5124.apk
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-Quest-Alpha-PR1012-fee508d_2fed5124.apk

vircadia-build-notifier avatar Mar 21 '21 07:03 vircadia-build-notifier

Refresh seems to work, but its worth noting the the new object will retain the dimensions or the original (reset dimensions will correct this). This is not a bug, and is desirable in many cases, but worth considering in the future.

Aitolda avatar Mar 21 '21 18:03 Aitolda

Compared loading The Hub content on this PR versus 2021.1.0. There doesn't seem to be any noticeable difference for me,

ctrlaltdavid avatar Mar 28 '21 01:03 ctrlaltdavid

Compared loading The Hub content on this PR versus 2021.1.0. There doesn't seem to be any noticeable difference for me,

Is there any way to make downloads work even if links redirect? I'm not sure how the system works or how things like that work in general.

However, between caching improvements and redirecting users to downloads that are on edge servers I think we can get fast loading times.

digisomni avatar Mar 28 '21 01:03 digisomni

Compared loading The Hub content on this PR versus 2021.1.0. There doesn't seem to be any noticeable difference for me,

My understanding is that, load times should be same or worse. Old behaviour is to just never update the cache. So whatever you downloaded would not change even if the source file changed. My understanding is that this PR contacts the server to find out if the cache is outdated.

JulianGro avatar Mar 28 '21 02:03 JulianGro

The following links are available: build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-d6b3159-_2fed5124.dmg

build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-d6b3159-_2fed5124.sh
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-d6b3159-.tar_2fed5124.Z
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-d6b3159-.tar_2fed5124.gz

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-d6b3159-_2fed5124.exe

build (ubuntu-18.04, android)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-d6b3159_2fed5124.apk
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-Quest-Alpha-PR1012-d6b3159_2fed5124.apk

vircadia-build-notifier avatar Apr 08 '21 17:04 vircadia-build-notifier

Test with webserver caching options set to no cache...

digisomni avatar Apr 08 '21 21:04 digisomni

So, I've found that the refresh button doesn't update for file:/// or https:// cases, did not try others but basically I see the model blink as if it's reloading except the new model doesn't show up.

I test this by...

  • renaming my model on the server after loading it once
  • hitting refresh in the Create app, but the model stays the same it was even though it should be a whole new model, just with the exact same filename and path

However, the test for cachebusting in general worked successfully...

  • rename the model on the server after loading it once
  • go to the tutorial world under "Help" -> "Tutorial" then return back to the world
  • notice that the model is now the new model

I do not believe my caching options on the webserver affected either case much.

digisomni avatar Apr 29 '21 23:04 digisomni

hmmm how is that different than what @Aitolda tried originally and reported success for?

HifiExperiments avatar Apr 30 '21 00:04 HifiExperiments

So I finally got to digging into this again and now I'm simply confused. I tried the newest version of this PR and couldn't get it working, so I started backtracking all the way to the first release on Feb 9, which was on the old protocol and it worked suddenly. Got excited, tried again, and it didn't work. Frustrated, I tried again and this time it worked. Seems to be inconsistent.

Here's what I've been doing.

I have two models that are both simply named "cube.glb" One of these is rotated off axis. I load up one, confirm it's working. Go to the model box in the properties dialogue and remove the link or add some gibberish. SOMETIMES the box becomes a green outline as expected. sometimes there is no effect on screen and the model remains.

Next I replace the original file on my S3 with the alternate cube.glb. In the properties dialogue I re-paste the link (identical). If the model had turned into the green outline previously, I now see the new model as expected, but if it remained visible, nothing happens.

Again so far I've only got this working on the original Feb 9th version but it's worth trying the others again. Will keep investigating.

Aitolda avatar May 15 '21 17:05 Aitolda

Going by Aitolda's comments it seems this needs more testing still

daleglass avatar Jul 22 '21 21:07 daleglass

The following links are available: build (macOS-latest, client)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-Interface-PR1012-a2125fb-_8e472fc4.dmg

build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-a2125fb-_8e472fc4.dmg

build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-a2125fb-.tar_8e472fc4.gz
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-a2125fb-.tar_8e472fc4.Z
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-a2125fb-_8e472fc4.sh

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1012/Vircadia-PR1012-a2125fb-_8e472fc4.exe

vircadia-build-notifier avatar Jul 31 '21 20:07 vircadia-build-notifier

Hello! Is this still an issue?

stale[bot] avatar Jun 18 '22 01:06 stale[bot]