Some pages not being synced to Amazon

[jordelver]

I have a problem with some pages not being uploaded to Amazon S3.

I'm using the blog extension. Everything uploads correctly except for the blog category pages which are served at blog/categories/. See output below.

The site is public here if it helps.

I thought it was because the category pages were not in the sitemap, but when I visit http://localhost:4567/__middleman/sitemap/ they seem to be there and everything builds correctly locally.

Any ideas what's up?

Processing remote files |Time: 00:00:01 | ===================================================================================== | Time: 00:00:01
     s3_sync  Ready to apply updates to jordanelver.co.uk.
     s3_sync  Putting bucket website: {"IndexDocument":"index.html","ErrorDocument":"404.html"}
     s3_sync  Deleting blog/categories/
     s3_sync  Deleting blog/categories/admin/
     s3_sync  Deleting blog/categories/amazon/
     s3_sync  Deleting blog/categories/api/
     s3_sync  Deleting blog/categories/arq/
     s3_sync  Deleting blog/categories/backups/
     s3_sync  Deleting blog/categories/commandline/
     s3_sync  Deleting blog/categories/curl/
     s3_sync  Deleting blog/categories/deployment/
     s3_sync  Deleting blog/categories/amazon/index.html
     s3_sync  Deleting blog/categories/design/
     s3_sync  Deleting blog/categories/backups/index.html
     s3_sync  Deleting blog/categories/development/
     s3_sync  Deleting blog/categories/api/index.html
     s3_sync  Deleting blog/categories/development/page/
     s3_sync  Deleting blog/categories/development/page/2/
     s3_sync  Deleting blog/categories/curl/index.html
     s3_sync  Deleting blog/categories/ergodox/
     s3_sync  Deleting blog/categories/arq/index.html
     s3_sync  Deleting blog/categories/firmware/
     s3_sync  Deleting blog/categories/deployment/index.html
     s3_sync  Deleting blog/categories/frontend/
     s3_sync  Deleting blog/categories/design/index.html
     s3_sync  Deleting blog/categories/git/
     s3_sync  Deleting blog/categories/development/page/2/index.html
     s3_sync  Deleting blog/categories/google-nearline/
     s3_sync  Deleting blog/categories/development/index.html
     s3_sync  Deleting blog/categories/heroku/
     s3_sync  Deleting blog/categories/admin/index.html
     s3_sync  Deleting blog/categories/hosting/
     s3_sync  Deleting blog/categories/commandline/index.html
     s3_sync  Deleting blog/categories/html/
     s3_sync  Deleting blog/categories/firmware/index.html
     s3_sync  Deleting blog/categories/json/
     s3_sync  Deleting blog/categories/ergodox/index.html
     s3_sync  Deleting blog/categories/linux/
     s3_sync  Deleting blog/categories/frontend/index.html
     s3_sync  Deleting blog/categories/middleman/
     s3_sync  Deleting blog/categories/git/index.html
     s3_sync  Deleting blog/categories/rest/
     s3_sync  Deleting blog/categories/google-nearline/index.html
     s3_sync  Deleting blog/categories/ruby/
     s3_sync  Deleting blog/categories/html/index.html
     s3_sync  Deleting blog/categories/s3/
     s3_sync  Deleting blog/categories/heroku/index.html
     s3_sync  Deleting blog/categories/software/
     s3_sync  Deleting blog/categories/hosting/index.html
     s3_sync  Deleting blog/categories/json/index.html
     s3_sync  Deleting blog/categories/sort/
     s3_sync  Deleting blog/categories/middleman/index.html
     s3_sync  Deleting blog/categories/superduper/
     s3_sync  Deleting blog/categories/timemachine/
     s3_sync  Deleting blog/categories/rest/index.html
     s3_sync  Deleting blog/categories/tmk_keyboard/
     s3_sync  Deleting blog/categories/s3/index.html
     s3_sync  Deleting blog/categories/vim/
     s3_sync  Deleting blog/categories/software/index.html
     s3_sync  Deleting blog/categories/zsh/
     s3_sync  Deleting blog/categories/linux/index.html
     s3_sync  Deleting blog/categories/superduper/index.html
     s3_sync  Deleting blog/categories/ruby/index.html
     s3_sync  Deleting blog/categories/sort/index.html
     s3_sync  Deleting blog/categories/tmk_keyboard/index.html
     s3_sync  Deleting blog/categories/vim/index.html
     s3_sync  Deleting blog/categories/timemachine/index.html
     s3_sync  Deleting blog/categories/zsh/index.html

[fredjean]

Could you post your config.rb file (minus any AWS credentials), Gemfile and Gemfile.lock so I can see what versions you have?

[fredjean]

Nevermind... I didn't realize that the repo was public. Looking into it...

[fredjean]

Try moving the s3_sync configuration block after the blog one.

[jordelver]

Thanks, it does have an impact but still wants to delete the category folders.

$ be middleman s3_sync --dry_run
     s3_sync  Let's see if there's work to be done...
Processing sitemap |Time: 00:00:01 | ==================================================================================================================== | Time: 00:00:01
Processing remote files |Time: 00:00:00 | =============================================================================================================== | Time: 00:00:00
     s3_sync  Ready to apply updates to jordanelver.co.uk.
     s3_sync  Putting bucket website: {"IndexDocument":"index.html","ErrorDocument":"404.html"}
     s3_sync  Creating blog/2016/index.html
     s3_sync  Creating blog/2016/01/index.html
     s3_sync  Creating blog/2016/01/25/index.html
     s3_sync  Creating blog/2015/index.html
     s3_sync  Creating blog/2015/05/index.html
     s3_sync  Creating blog/2015/05/27/index.html
     s3_sync  Creating blog/2015/04/index.html
     s3_sync  Creating blog/2015/04/29/index.html
     s3_sync  Creating blog/2015/03/index.html
     s3_sync  Creating blog/2015/03/30/index.html
     s3_sync  Creating blog/2015/02/index.html
     s3_sync  Creating blog/2015/02/02/index.html
     s3_sync  Creating blog/2015/01/index.html
     s3_sync  Creating blog/2015/01/27/index.html
     s3_sync  Creating blog/2014/index.html
     s3_sync  Creating blog/2014/07/index.html
     s3_sync  Creating blog/2014/07/13/index.html
     s3_sync  Creating blog/2014/06/index.html
     s3_sync  Creating blog/2014/06/25/index.html
     s3_sync  Creating blog/2014/03/index.html
     s3_sync  Creating blog/2014/03/12/index.html
     s3_sync  Creating blog/2014/02/index.html
     s3_sync  Creating blog/2014/02/17/index.html
     s3_sync  Creating blog/2014/02/10/index.html
     s3_sync  Creating blog/2014/02/04/index.html
     s3_sync  Creating blog/2014/01/index.html
     s3_sync  Creating blog/2014/01/31/index.html
     s3_sync  Creating blog/2011/index.html
     s3_sync  Creating blog/2011/07/index.html
     s3_sync  Creating blog/2011/07/30/index.html
     s3_sync  Creating blog/2011/07/24/index.html
     s3_sync  Creating blog/2010/index.html
     s3_sync  Creating blog/2010/11/index.html
     s3_sync  Creating blog/2010/11/27/index.html
     s3_sync  Creating blog/2009/index.html
     s3_sync  Creating blog/2009/11/index.html
     s3_sync  Creating blog/2009/11/27/index.html
     s3_sync  Creating blog/2009/06/index.html
     s3_sync  Creating blog/2009/06/03/index.html
     s3_sync  Creating blog/2008/index.html
     s3_sync  Creating blog/2008/12/index.html
     s3_sync  Creating blog/2008/12/31/index.html
     s3_sync  Creating blog/2008/12/16/index.html
     s3_sync  Creating blog/2008/08/index.html
     s3_sync  Creating blog/2008/08/07/index.html
     s3_sync  Creating blog/2008/08/04/index.html
     s3_sync  Creating blog/2008/07/index.html
     s3_sync  Creating blog/2008/07/24/index.html
     s3_sync  Creating blog/2008/07/10/index.html
     s3_sync  Creating blog/2008/06/index.html
     s3_sync  Creating blog/2008/06/30/index.html
     s3_sync  Creating blog/2008/06/18/index.html
     s3_sync  Creating blog/2008/06/17/index.html
     s3_sync  Deleting blog/categories/
     s3_sync  Deleting blog/categories/admin/
     s3_sync  Deleting blog/categories/amazon/
     s3_sync  Deleting blog/categories/api/
     s3_sync  Deleting blog/categories/arq/
     s3_sync  Deleting blog/categories/backups/
     s3_sync  Deleting blog/categories/commandline/
     s3_sync  Deleting blog/categories/curl/
     s3_sync  Deleting blog/categories/deployment/
     s3_sync  Deleting blog/categories/design/
     s3_sync  Deleting blog/categories/development/
     s3_sync  Deleting blog/categories/development/page/
     s3_sync  Deleting blog/categories/development/page/2/
     s3_sync  Deleting blog/categories/ergodox/
     s3_sync  Deleting blog/categories/firmware/
     s3_sync  Deleting blog/categories/frontend/
     s3_sync  Deleting blog/categories/git/
     s3_sync  Deleting blog/categories/google-nearline/
     s3_sync  Deleting blog/categories/heroku/
     s3_sync  Deleting blog/categories/hosting/
     s3_sync  Deleting blog/categories/html/
     s3_sync  Deleting blog/categories/json/
     s3_sync  Deleting blog/categories/linux/
     s3_sync  Deleting blog/categories/middleman/
     s3_sync  Deleting blog/categories/rest/
     s3_sync  Deleting blog/categories/ruby/
     s3_sync  Deleting blog/categories/s3/
     s3_sync  Deleting blog/categories/software/
     s3_sync  Deleting blog/categories/sort/
     s3_sync  Deleting blog/categories/superduper/
     s3_sync  Deleting blog/categories/timemachine/
     s3_sync  Deleting blog/categories/tmk_keyboard/
     s3_sync  Deleting blog/categories/vim/
     s3_sync  Deleting blog/categories/zsh/

[fredjean]

I was able to (somewhat) reproduce this morning.

I changed the blog taglink setting and immediately did an middleman s3_sync. The articles were not updated and the existing tags based path were removed.

However, the behavior was what I expected once I ran a middleman build before the sync. The tags path were changed to categories and were sync'ed as I expected them to.

Can you give it a try and see what happens?

[jordelver]

Sorry for the late reply. I just tried this and it doesn't change anything unfortunately :(

[tlindsay]

I'm having a similar issue to this. My favicons aren't getting pushed up. I'm using follmann/middleman-favicon-generator, and I've confirmed the favicons are in the build dir.

Edit: Whoops! Forgot to leave my repo in case you want to take a look. tlindsay/thatdarnpat

Let me know if there's anything I can do to help fix this problem! Not very familiar with this gem, but happy to contribute in any way I can.

[matiasgarciaisaia]

I had this very same issue but regarding blog pages - /blog would sync, but not /blog/pages/2.

"Fixed" it by activating the s3_sync extension after the blog one.

[mathie]

FWIW, I think I've seen the same behaviour this morning on one of my sites. I think what's happening is that the proxy resource configuration isn't triggering the sitemap to be refreshed, so s3sync isn't becoming aware of pages declared with it, unless the sitemap is refreshed later in the configuration file. (Does that even make sense?)

In my particular case, I've got a ready block that adds a pile of category & tag pages (which I suspect is much like what the blog extension does):

https://github.com/wossname/woss.name/blob/master/config.rb#L69-L94

Without the call to all_tags at the end, the tags pages weren't getting generated. I'm slightly confused as to why iterating through the sitemap makes it get refreshed, but adding a proxy page doesn't!

[rsharrott]

Had the same problem with pages that I created with proxy that came from data downloaded prior to build.

Wrapping the proxy code in a ready do block makes the files get ignored in sync.

This is ignored:

ready do
  data.downloaded.each do |id, item|
    proxy "/some_data/#{item.title}.html", "/data.html"
  end  
end

This is included:

  data.downloaded.each do |id, item|
    proxy "/some_data/#{item.title}.html", "/data.html"
  end  

[matiasgarciaisaia]

Sorry, forgot to comment on this issue. You can check my last comment on #135:

I found I had to manually call sitemap.ensure_resource_list_updated! after applying the redirects. It may be relevant that I was doing that on a ready block in my config.rb - I'm redirecting blogposts and a few other pages, so I needed the sitemap to be fully populated before doing the redirects.

So my config.rb is something like:

activate :blog do |blog|
  # ...
end

activate :s3_sync do |s3_sync|
  # ...
end

ready do
  blog.articles.each do |article|
    author = article.data['author']
    redirect article.url.sub('/blog/', "blog/#{author}/").gsub(/\.html$/, '/index.html'), to: article
  end

  sitemap.ensure_resource_list_updated! # <-- This "fixes" the issue
end

I'm not sure if there's anything that can be done in either Middleman or middleman-s3_sync to avoid that extra call.

[maxiloc]

I had a similar issue with redirects not being pushed to S3. Adding sitemap.ensure_resource_list_updated! fixed it as well.