Watcher keeps going after composer install or branch switch

[rikwillems]

During development I often have the cache-clean watcher running. When I switch branch and do a composer install the watcher registers the file changes, but then keeps cleaning and cleaning. It looks like it registers all file changes and cleans related caches one by one. The only way to stop it is to close the terminal window running the watcher.

If this is the case maybe it is possible to implement a limiter of some sort?

[Vinai]

Hi @rikwillems Currently every file change event continues to be processed, regardless whether the cache already is empty or not. Maybe I could add some kind of check "if this cache type or id already was cleared within the last 3 seconds, ignore"... That might help. Any thoughs?

[rikwillems]

Hi @Vinai I'm unsure about the watcher internals or what is possible. If each file is a separate event it might be hard to collect changes to do the right cache clear once. It all events come in a few milliseconds maybe a short delay could do a collection over all events?

[Vinai]

I was thinking of this (pseudocode):

function fileChanged(event)
{
    cacheTypesToClear = determineCacheTypesToClear(event);
    cacheTypesToClear.map(cacheType => {
        if (now() > lastClearedCacheTypeTimestamp(cacheType) + 3000) {
            clearCacheType(cacheType);
            setLastClearedCacheTypeTimestamp(cacheType, now());
        }
    });
}

...and the same for cache ids. This would skip cleaning cache types that where recently cleaned.

If I understand you correctly, you suggest buffering incoming change events and cleaning them periodically or after a delay. Something like this pseudocode:

function fileChanged(event)
{
    cacheTypesToClear = determineCacheTypesToClear(event);
    cacheTypesToClear.map(cacheTypesToClearSet.add);
    t = t ? t : setTimeout(() => { cleanCacheTypes(cacheTypesToClearSet); t = false; }, 2000);
}

To me skipping looks more digestable than buffering... It also would mean having to wait for a cache to be cleaned until the timeout is reached, even if there is no clean-event flood. Did I understand you correctly though?

[rikwillems]

Yeah that is exactly what I mean. The event flood happens during a large change like branch switch or composer install. But while developing I run the watcher together with Browsersync. That means my browser is refreshed straight after the cache invalidation. A few saves short after each other could fall within the 3 seconds. The delay/buffer looks good but would delay normal changes too. The question I cannot answer is how quickly the events come into the watcher. Is a branch switch 10000 events in 100ms or does it take longer? If it is only a small timeframe the 2000 timeout could be 100 and that is very workable. Maybe you can also ignore invalidation for cache types that are already scheduled or in progress?

[Vinai]

How long it would take for the all the fs events to register with the watcher process after a checkout or install depends on the filesystem and environment.

Since each event is processed without in isolation it would be tricky to check against other pending events.

Since this is written in ClojureScript, a core.async channel with a distinct transducer to remove incoming events comes to mind quickly, but that won't work because that transducer is stateful and would need to wait until all events have arrived (i.e. eternity).

A custom stateful transducer might probably work, but it has been years since I wrote one.

As a quick workaround I'll add a pause to cleaning the same cache/cache ID within 500 milliseconds. That isn't a great solution but it might be good enough - or at least better than the current state.

[rikwillems]

As I'm not at all familiar with these internal processes or ClojureScript at all I can't really add anything here.

I think this quick workaround will suffice. As it's only on specific actions. It's just that I can't stop the processing but by closing the window.

[Vinai]

FYI, I've been working with a version containing the quick-fix for 2 days now and it seems to work. If all continues to work I'll release it on Monday.

[rikwillems]

Sweet, thank you Vinai. If I can test anything please let me know.

[Vinai]

I've just tagged release 1.0.36 that includes the "workaround" with a "grace time" of 500ms. So duplicate clean types/ids are ignored for half a second. Curious to hear if that makes a difference to you.

[rikwillems]

I'm afraid the update doesn't make any difference. The same happens, it keeps clearing caches and I can't stop it.

[Vinai]

I would like to gain a bit of additional visibility on what is happening exactly. I've added a little debug logging and pushed my branch flood-protection to github.

Could you please install that version with composer global require mage2tv/magento-cache-clean:dev-flood-protection (if you have a global install, otherwise without the global option). Then please run the command with cache-clean.js -w -vv to enable debug output. Then switch a branch and paste the output here (there will be a lot of log messages). Hopefully that will help. Thank you!

[rikwillems]

This is a short extract from the output on the specified branch:

23:07:02 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/list/toolbar/amount.phtml
23:07:02 Cleaning cache type(s) block_html full_page
23:07:02 Using :default cache_backend
Reading app config by shelling out to php
23:07:02 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:02 WAIT grace period  block_html -497ms
23:07:02 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:02 WAIT grace period  block_html -497ms
23:07:02 WAIT grace period  full_page -195ms
Reading app config by shelling out to php
23:07:02 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/list/toolbar/sorter.phtml
23:07:02 Cleaning cache type(s) block_html full_page
23:07:02 Using :default cache_backend
Reading app config by shelling out to php
23:07:02 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:02 WAIT grace period  block_html -384ms
23:07:02 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:02 WAIT grace period  block_html -384ms
23:07:02 WAIT grace period  full_page -81ms
Reading app config by shelling out to php
23:07:02 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/view/addtocart.phtml
23:07:02 Cleaning cache type(s) block_html full_page
23:07:02 Using :default cache_backend
Reading app config by shelling out to php
23:07:02 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:02 WAIT grace period  block_html -284ms
23:07:02 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:02 WAIT grace period  block_html -284ms
23:07:02 OK to clean  full_page since 1018 ms
23:07:02 Using :page_cache cache backend
Reading app config by shelling out to php
23:07:02 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/page_cache}, :id_prefix b8d_}
23:07:02 Cleaning dir /Users/rikwillems/Sites/project/var/page_cache/
Reading app config by shelling out to php
Reading app config by shelling out to php
23:07:02 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/list/toolbar
Reading app config by shelling out to php
23:07:02 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/list
Reading app config by shelling out to php
23:07:02 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/view/attributes.phtml
23:07:02 Cleaning cache type(s) block_html full_page
23:07:02 Using :default cache_backend
Reading app config by shelling out to php
23:07:02 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:02 OK to clean  block_html since 1040 ms
23:07:02 Cleaning tag BLOCK_HTML
23:07:02 Tag-file /Users/rikwillems/Sites/project/var/cache/mage-tags/mage---b8d_BLOCK_HTML
23:07:02 WAIT grace period  block_html -1000ms
23:07:02 WAIT grace period  full_page -675ms
Reading app config by shelling out to php
23:07:02 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/view/authors.phtml
23:07:02 Cleaning cache type(s) block_html full_page
23:07:02 Using :default cache_backend
Reading app config by shelling out to php
23:07:02 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:02 WAIT grace period  block_html -889ms
23:07:02 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:02 WAIT grace period  block_html -888ms
23:07:02 WAIT grace period  full_page -564ms
Reading app config by shelling out to php
23:07:02 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/view/gallery.phtml
23:07:02 Cleaning cache type(s) block_html full_page
23:07:02 Using :default cache_backend
Reading app config by shelling out to php
23:07:02 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:02 WAIT grace period  block_html -779ms
23:07:02 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:02 WAIT grace period  block_html -779ms
23:07:02 WAIT grace period  full_page -455ms
Reading app config by shelling out to php
23:07:02 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/view/pdf-sample.phtml
23:07:02 Cleaning cache type(s) block_html full_page
23:07:02 Using :default cache_backend
Reading app config by shelling out to php
23:07:03 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:03 WAIT grace period  block_html -668ms
23:07:03 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:03 WAIT grace period  block_html -668ms
23:07:03 WAIT grace period  full_page -344ms
Reading app config by shelling out to php
23:07:03 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/view/product-info.phtml
23:07:03 Cleaning cache type(s) block_html full_page
23:07:03 Using :default cache_backend
Reading app config by shelling out to php
23:07:03 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:03 WAIT grace period  block_html -557ms
23:07:03 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:03 WAIT grace period  block_html -556ms
23:07:03 WAIT grace period  full_page -232ms
Reading app config by shelling out to php
23:07:03 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/view/subscription/product-info.phtml
23:07:03 Cleaning cache type(s) block_html full_page
23:07:03 Using :default cache_backend
Reading app config by shelling out to php
23:07:03 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:03 WAIT grace period  block_html -452ms
23:07:03 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:03 WAIT grace period  block_html -452ms
23:07:03 WAIT grace period  full_page -128ms
Reading app config by shelling out to php
23:07:03 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/view/usp-product.phtml
23:07:03 Cleaning cache type(s) block_html full_page
23:07:03 Using :default cache_backend
Reading app config by shelling out to php
23:07:03 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:03 WAIT grace period  block_html -358ms
23:07:03 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:03 WAIT grace period  block_html -358ms
23:07:03 WAIT grace period  full_page -34ms
Reading app config by shelling out to php
23:07:03 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Customer/templates/header/customer-menu.phtml
23:07:03 Cleaning cache type(s) block_html full_page
23:07:03 Using :default cache_backend
Reading app config by shelling out to php
23:07:03 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:03 WAIT grace period  block_html -260ms
23:07:03 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:03 WAIT grace period  block_html -260ms
23:07:03 OK to clean  full_page since 1064 ms
23:07:03 Using :page_cache cache backend
Reading app config by shelling out to php
23:07:03 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/page_cache}, :id_prefix b8d_}
23:07:03 Cleaning dir /Users/rikwillems/Sites/project/var/page_cache/
Reading app config by shelling out to php
Reading app config by shelling out to php
23:07:03 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/view/subscription
Reading app config by shelling out to php
23:07:03 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product/view
Reading app config by shelling out to php
23:07:03 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates/product
Reading app config by shelling out to php
23:07:03 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog/templates
Reading app config by shelling out to php
23:07:03 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Catalog
Reading app config by shelling out to php
23:07:03 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_LayeredNavigation/templates/layer/state.phtml
23:07:03 Cleaning cache type(s) block_html full_page
23:07:03 Using :default cache_backend
Reading app config by shelling out to php
23:07:03 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:03 OK to clean  block_html since 1169 ms
23:07:03 Cleaning tag BLOCK_HTML
23:07:03 Tag-file /Users/rikwillems/Sites/project/var/cache/mage-tags/mage---b8d_BLOCK_HTML
23:07:03 WAIT grace period  block_html -999ms
23:07:03 WAIT grace period  full_page -570ms
Reading app config by shelling out to php
23:07:03 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Customer/templates/header
Reading app config by shelling out to php
23:07:03 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Customer/templates
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Customer
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_LayeredNavigation/templates/layer/view.phtml
23:07:04 Cleaning cache type(s) block_html full_page
23:07:04 Using :default cache_backend
Reading app config by shelling out to php
23:07:04 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:04 WAIT grace period  block_html -761ms
23:07:04 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:04 WAIT grace period  block_html -761ms
23:07:04 WAIT grace period  full_page -332ms
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Newsletter/templates/subscribe.phtml
23:07:04 Cleaning cache type(s) block_html full_page
23:07:04 Using :default cache_backend
Reading app config by shelling out to php
23:07:04 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:04 WAIT grace period  block_html -665ms
23:07:04 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:04 WAIT grace period  block_html -665ms
23:07:04 WAIT grace period  full_page -236ms
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_LayeredNavigation/templates/layer
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_LayeredNavigation/templates
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_LayeredNavigation
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Swatches/layout/catalog_product_view_type_configurable.xml
23:07:04 Cleaning cache type(s) layout full_page
23:07:04 Using :default cache_backend
Reading app config by shelling out to php
23:07:04 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:04 OK to clean  layout since 3080 ms
23:07:04 Cleaning tag LAYOUT_GENERAL_CACHE_TAG
23:07:04 Tag-file /Users/rikwillems/Sites/project/var/cache/mage-tags/mage---b8d_LAYOUT_GENERAL_CACHE_TAG
23:07:04 WAIT grace period  layout -999ms
23:07:04 WAIT grace period  full_page -1ms
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Newsletter/templates
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Newsletter
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Swatches/templates/product/view/renderer.phtml
23:07:04 Cleaning cache type(s) block_html full_page
23:07:04 Using :default cache_backend
Reading app config by shelling out to php
23:07:04 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:04 WAIT grace period  block_html -241ms
23:07:04 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:04 WAIT grace period  block_html -240ms
23:07:04 OK to clean  full_page since 1189 ms
23:07:04 Using :page_cache cache backend
Reading app config by shelling out to php
23:07:04 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/page_cache}, :id_prefix b8d_}
23:07:04 Cleaning dir /Users/rikwillems/Sites/project/var/page_cache/
Reading app config by shelling out to php
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Swatches/layout
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Theme/layout/cms_index_index.xml
23:07:04 Cleaning cache type(s) layout full_page
23:07:04 Using :default cache_backend
Reading app config by shelling out to php
23:07:04 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:04 WAIT grace period  layout -566ms
23:07:04 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:04 WAIT grace period  layout -565ms
23:07:04 WAIT grace period  full_page -756ms
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Swatches/templates/product/view
Reading app config by shelling out to php
23:07:04 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Swatches/templates/product
Reading app config by shelling out to php
23:07:05 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Swatches/templates
Reading app config by shelling out to php
23:07:05 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Swatches
Reading app config by shelling out to php
23:07:05 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Theme/layout/default.xml
23:07:05 Cleaning cache type(s) layout full_page
23:07:05 Using :default cache_backend
Reading app config by shelling out to php
23:07:05 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:05 WAIT grace period  layout -283ms
23:07:05 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:05 WAIT grace period  layout -282ms
23:07:05 WAIT grace period  full_page -473ms
Reading app config by shelling out to php
23:07:05 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Theme/layouts.xml
Reading app config by shelling out to php
23:07:05 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Theme/page_layout/1column-narrow.xml
23:07:05 Cleaning cache type(s) layout full_page
23:07:05 Using :default cache_backend
Reading app config by shelling out to php
23:07:05 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:05 WAIT grace period  layout -139ms
23:07:05 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:05 WAIT grace period  layout -138ms
23:07:05 WAIT grace period  full_page -329ms
Reading app config by shelling out to php
23:07:05 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Theme/layout
Reading app config by shelling out to php
23:07:05 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Theme/templates/html/breadcrumbs.phtml
23:07:05 Cleaning cache type(s) block_html full_page
23:07:05 Using :default cache_backend
Reading app config by shelling out to php
23:07:05 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:05 OK to clean  block_html since 1571 ms
23:07:05 Cleaning tag BLOCK_HTML
23:07:05 Tag-file /Users/rikwillems/Sites/project/var/cache/mage-tags/mage---b8d_BLOCK_HTML
23:07:05 WAIT grace period  block_html -1000ms
23:07:05 WAIT grace period  full_page -189ms
Reading app config by shelling out to php
23:07:05 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Theme/page_layout
Reading app config by shelling out to php
23:07:05 Processing /Users/rikwillems/Sites/project/app/design/frontend/Project/base/Magento_Theme/templates/html/footer.phtml
23:07:05 Cleaning cache type(s) block_html full_page
23:07:05 Using :default cache_backend
Reading app config by shelling out to php
23:07:05 Cache storage  {:backend Cm_Cache_Backend_File, :backend_options {:cache_dir /Users/rikwillems/Sites/project/var/cache}, :id_prefix b8d_}
23:07:05 WAIT grace period  block_html -859ms
23:07:05 Cleaning dir /Users/rikwillems/Sites/project/var/cache/
23:07:05 WAIT grace period  block_html -859ms
23:07:05 WAIT grace period  full_page -48ms
Reading app config by shelling out to php

[Vinai]

Thanks for all that. Sorry for the late response, I haven't forgotten about this issue. To be honest, I'm currently feel a bit stuck though. I can't think of a way to collect cache clean events without introducing a global delay at the same time. What also doesn't help is that I'm so far unable to reproduce the issue. To test I've switched from one branch to another one that introduces a couple of modules. In my case the watcher runs a few seconds and then is done.

[rikwillems]

OK let's leave it at that. Closing the terminal stops the continued running. No need to invest more time.

[Vinai]

I'll keep the issue open. Maybe I can revisit in if I have a new idea...

[Vinai]

I think I finally implemented a fix in release 1.0.44 :). The file watcher will only trigger events for the same cache type of ID after a guard period of 5 seconds.
Please try it when you get a chance and let me know how it works for you.

[hostep]

I'm also running into this problem using 1.0.48 of the tool.

For me what would be a good fix, is just having the opportunity to press Ctrl+C to stop the process, but that keypress isn't being caught by the tool during this seemingly endless amount of cache flushing that happens. I have to open a new terminal window, find the process id and kill it, which is a bit inconvenient.

An easy way to reproduce:

• have one git branch with an old version of Magento
• have a second git branch with a new version of Magento
• be on the first branch and have everything running as it should
• start the file watching
• switch to the second branch and execute composer install

[Wohlie]

Hi, I also run into this Problem with v1.0.48. I removed my vendor folder and perform a composer install. The watcher took about 10 minutes to finish all the file changes. I even can't stop the watcher with [Ctrl] + [C].

It is normal that the process takes 10 minutes, or is there any problem with my OS/configuration?

It looks like that the same cache will be purge multiple time for the same logged timestamp, see full_page at the end of the following sample:

...
21:10:09 Processing /var/www/default/src/vendor/magento/module-staging/Model
21:10:09 Processing /var/www/default/src/vendor/magento/module-staging/Api/UpdateRepositoryInterface.php
21:10:09 Cleaning id(s): serviceInterfaceMethodsMap-d11120fafadce667b44efe490f7ba9b4 service_method_params_c5e8ab41c6cfb703a9e9fd936d5e300e service_method_params_824ed9ec53fe001684f05ee44c2becee service_method_params_499bbaa9304d9b969baafbea4585fd34 service_method_params_cb1e44ecbb2250a68592bff60821b38d service_method_params_b762c1d764a964f1530c96d595e04e1b
21:10:09 Processing /var/www/default/src/vendor/magento/module-staging/Api/Data/UpdateSearchResultInterface.php
21:10:09 Cleaning id(s): serviceInterfaceMethodsMap-17ad06685da24a618e8b82eb2cf0be69 service_method_params_6be357f7abcea8635ba5c0be8b8c60b3 service_method_params_82aaee7300a9ce6c0501b9918c38753f
21:10:09 Processing /var/www/default/src/vendor/magento/module-staging/Api/Data/UpdateInterface.php
21:10:09 Cleaning id(s): serviceInterfaceMethodsMap-2180413387784593238d78681f61274b service_method_params_bca913cfc97f0935d8ee95a1a3b8472f service_method_params_282ed5e7443fa1ed37437bef53b9f9e8 service_method_params_39cf6f92383aba4161c4f28cf813f363 service_method_params_970fefefbe9820d114e40230884ce316 service_method_params_50702bec5488dc5dcf14e977aa1b3d33 service_method_params_4bd849ecb1562cebd5ffa953a9616967 service_method_params_8c6b5955f3c17c53be597b382d24f550 service_method_params_ae535083a676691ebbcab3e824ecdf83 service_method_params_8c2cfc2342e97195828c734eac93e880 service_method_params_5a1432cf3d10aafc332774e5ad20c07b service_method_params_0ff257ea1264dfd2e5ea8e210608ff25 service_method_params_fb73c808c062fc533c3304db82143f6f service_method_params_f20ed7f35f9f6a8d7935d4879772fed2 service_method_params_95c254985e8706a240559fcbf39808ad service_method_params_28d49da0a2fc1998c3ecfa35e8f636dc service_method_params_a338100c1a88cb5c9a1000dd81c9054c service_method_params_655261dd8f843e0f4089c83d2ba7ac76 service_method_params_ba8592f1c3081a1323925681fabce372 service_method_params_0a3d54523767f3fe8b81a8e459d348f4 service_method_params_4ac8f677fda039ce496d6f87195a0ce1
21:10:09 Processing /var/www/default/src/vendor/magento/module-staging/Api/Data/Data
21:10:09 Processing /var/www/default/src/vendor/magento/module-staging/Api/Data
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/Api/Api
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/Api
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/LICENSE_EE.txt
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/templates/preview.phtml
21:10:10 Cleaning cache type(s) block_html full_page
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/templates/schedule-title.phtml
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/templates/templates
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/templates
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/layout/staging_update_preview.xml
21:10:10 Cleaning cache type(s) layout
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/layout/staging_update_index.xml
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/layout/staging_update_edit.xml
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/layout/layout
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/layout
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/ui_component/staging_update_preview_grid.xml
21:10:10 Cleaning id(s): ui_component_configuration_data_staging_update_preview_grid
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/ui_component/staging_update_edit.xml
21:10:10 Cleaning id(s): ui_component_configuration_data_staging_update_edit
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/ui_component/staging_update_grid.xml
21:10:10 Cleaning id(s): ui_component_configuration_data_staging_update_grid
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/ui_component/ui_component
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/ui_component
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid/listing-disabled.html
21:10:10 Removing compiled frontend js-translation.json files
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid/toolbar.html
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid/includes.html
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid/cells/singleselect.html
21:10:10 Removing compiled frontend js-translation.json files
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid/cells/cell-date.html
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid/cells/actions.html
21:10:10 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid/cells/cells
21:10:11 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid/cells
21:10:11 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid/listing.html
21:10:11 Removing compiled frontend js-translation.json files
21:10:11 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid/paging/paging-total.html
21:10:11 Removing compiled frontend js-translation.json files
21:10:11 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid/paging/paging
21:10:11 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid/paging
21:10:11 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid/grid
21:10:11 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/grid
21:10:11 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/timeline/details.html
21:10:11 Removing compiled frontend js-translation.json files
21:10:11 Processing /var/www/default/src/vendor/magento/module-staging/view/adminhtml/web/template/timeline/record.html
21:10:11 Removing compiled frontend js-translation.json files
...
21:10:32 Processing /var/www/default/src/vendor/magento/module-versions-cms/Test/Unit/Controller/Adminhtml/Cms/Page/InlineEdit/PluginTest.php
21:10:32 Cleaning id(s): app_action_list
21:10:32 Cleaning cache type(s) full_page
21:10:32 Processing /var/www/default/src/vendor/magento/module-versions-cms/Test/Unit/Controller/Adminhtml/Cms/Page/InlineEdit/InlineEdit
21:10:32 Processing /var/www/default/src/vendor/magento/module-versions-cms/Test/Unit/Controller/Adminhtml/Cms/Page/InlineEdit
21:10:32 Processing /var/www/default/src/vendor/magento/module-versions-cms/Test/Unit/Controller/Adminhtml/Cms/Page/Page
21:10:32 Processing /var/www/default/src/vendor/magento/module-versions-cms/Test/Unit/Controller/Adminhtml/Cms/Page
21:10:32 Processing /var/www/default/src/vendor/magento/module-versions-cms/Test/Unit/Controller/Adminhtml/Cms/Hierarchy/Widget/ChooserTest.php
21:10:32 Cleaning id(s): app_action_list
21:10:32 Cleaning cache type(s) full_page
21:10:32 Processing /var/www/default/src/vendor/magento/module-versions-cms/Test/Unit/Controller/Adminhtml/Cms/Hierarchy/Widget/Widget
21:10:32 Processing /var/www/default/src/vendor/magento/module-versions-cms/Test/Unit/Controller/Adminhtml/Cms/Hierarchy/Widget
21:10:32 Processing /var/www/default/src/vendor/magento/module-versions-cms/Test/Unit/Controller/Adminhtml/Cms/Hierarchy/SaveTest.php
21:10:32 Cleaning id(s): app_action_list
21:10:32 Cleaning cache type(s) full_page
...

Maybe we can also exclude the Test folder to speed up the process.

[esprit-dev]

Hi @Vinai Since i upgraded from a very old version to the lastest, i'm discovering your fix on the release 1.0.44. On my side, i don't want to wait 5 seconds between each file update : often I modify a file several times in a very short time, and now i have to to check the console output each time to verify the cache cleaning job :( Killing the process when switching git branch is not a problem for me. Could you add a cli parameter so that we can disable or reduce the guard period with our own value ? (def cache-clean-guard-period 5000) Thank you, and many thanks for that great tool !

[Vinai]

Hey @esprit-dev, it should be possible to make it a command line argument. Something like --no-flood-guard or -n?

[Vinai]

I've just tagged release 1.0.51 which:

• process each file change as a macro-task to allow keyboard events to be processed (i.e. ctrl-c while many changes are being processed)
• add --no-flood-guard or the short form -n flags to disable the 5s timeframe per cache type

Please feel free to reopen this issue (or open a new one) if any of you don't consider this done.

[esprit-dev]

Hi @Vinai it's perfect now ! Thank you :)