Duplicating folder/image during build

[jarodtaylor]

I'm not sure if this is an issue with permalinks or if it's a product of how metalsmith works. It's quite possible that it's just the way mine is structured. I'm hoping you can shed some light on it.

Okay, I am using Gulp/Metalsmith (gulpsmith). Metalsmith is only going to be handling the images and markdown files. I'm using gulp to process the js, css, etc.

I have a src directory like this:

├─ src
|   ├── javascripts (handled by Gulp)
|   ├── metalsmith (handled by metalsmith/gulpsmith)
|   |   ├── blog
|   |   |   ├── 09-17-2015
|   |   |   |   ├── first-post.md
|   |   |   |   ├── first-post-specific-image.png
|   |   |   ├── index.md
|   |   ├── about
|   |   ├── images
|   |   ├── index.md
|   ├── stylesheets (handled by Gulp)

here's my metalsmith.js file:

import gulp from 'gulp'
import gulpsmith from 'gulpsmith'
import markdown from 'metalsmith-markdown'
import layouts from 'metalsmith-layouts'
import collections from 'metalsmith-collections'
import permalinks from 'metalsmith-permalinks'
import gulpFrontMatter from 'gulp-front-matter'
import connect from 'gulp-connect'
import lodash from 'lodash'
import elevate from 'metalsmith-elevate'

module.exports = function() {
  gulp.src('./src/metalsmith/**/*')
    .pipe(gulpFrontMatter()).on('data', function(file) {
      lodash.assign(file, file.frontMatter);
      delete file.frontMatter;
    })
    .pipe(
      gulpsmith()
        .use(collections({
          articles: {
            pattern: 'blog/!(index.md)**/*.md',
            sortBy: 'date',
            reverse: true
          }
        }))
        .use(markdown())
        .use(layouts({
          engine: 'handlebars',
          directory: 'layouts',
          partials: {
            header: 'partials/header',
            footer: 'partials/footer'
          }
        }))
        .use(permalinks({
          pattern: 'blog/:title'
        }))
    )
    .pipe(gulp.dest('./build'))
    .pipe(connect.reload());
};

So when I run my gulp build task it does its three jobs (process my stylesheets, process my js files, and run the metalsmith task). I have the permalinks pattern referring to blog/:title, and the title is set in the .md file (e.g. My First Post). So metalsmith-permalinks does build it with blog/first-post/index.html and blog/first-post/first-post-specific-image.png, but the image (but not the index.html) is also put in a 09-17-2015 directory.

This is what my /build/blog directory ends up looking like:

├─ build
|   ├── blog
|   |  ├── 09-17-2015
|   |  |  ├── first-post-specific-image.png
|   |  ├── first-post
|   |  |  ├── first-post-specific-image.png
|   |  |  ├── index.html
|   |  ├── index.html

Like I said, I'm not sure if this is metalsmith-permalinks or if it's metalsmith. I'm just hoping someone can shed some light.

Thanks for all the great plugins, by the way!

[jarodtaylor]

I've tried to dig into the source and I think it has something to do with line 57 where you're deleting the files. It's not including the dupes.

[jarodtaylor]

For anyone else that runs into this issue, I thought I'd share my solution for it.

I couldn't quite figure out how to fix the plugin so for now I just created a custom metalsmith plugin that I call AFTER permalinks.

So, when I went into my function (custom plugin I was creating) and looped over the files object returned by metalsmith after permalinks did its thing, I did a console.log(file) and this was returned:

blog/09-17-2015/jt_headshot.jpg
blog/09-23-2015/phpbb_jt.jpg
index.html
about/index.html
blog/index.html
blog/first-post/index.html
blog/second-post/index.html
blog/first-post/jt_headshot.jpg
blog/second-post/phpbb_jt.jpg

The top two were the ones that were duplicating (as you can see by the bottom two). So I just had to use some regex magic to filter them out and delete them. Here's the plugin code:

function removeDupes() {
 return function(files, metalsmith, done) {
    var duplicate = /^blog\/\d{2}\-\d{2}\-\d{4}\/.*$/ ;
    Object.keys(files).forEach(function(file) {
      if(duplicate.test(file)){
        delete files[file];
      }
    });
    done();
  }
}

then you just have to make sure you include your plugin after permalinks:

gulpsmith()
        .use(collections({
          articles: {
            pattern: 'blog/!(index.md)**/*',
            sortBy: 'date',
            reverse: true
          }
        }))
        .use(markdown())
        .use(layouts({
          engine: 'handlebars',
          directory: 'layouts',
          partials: {
            header: 'partials/header',
            footer: 'partials/footer'
          }
        }))
        .use(permalinks({
          pattern: 'blog/:title'
        }))
        .use(removeDupes())

Obviously you'll need to alter the regex to fit your needs, but I hope this helps someone else until the permalinks plugin is fixed.

[alex-ketch]

It seems that this is actually the intended functionality of the plugin, one that I'm also trying to work around.

From the docs: Notice how the css directory is duplicated and is in both build and post folders.

For example for this source directory:

src/
  css/
    style.css
  post.html

Here's what the build directory would look like with relative on:

build/
  post/
    index.html
    css/  // <- HERE
      style.css
  css/  // <- AND HERE
    style.css

For me this is a problem as I'm working on my design portfolio site which is very image heavy. I'm going to see if I can add an option that will remove allow for the removal of the original if desired. Thinking optional as it could be destructive for non permalinked files if the top level stylesheet were removed, as per the example.

[alex-ketch]

Right now I've solved this by adding the following code at the end of the Permalinks main return function. Entirely not efficient, convoluted and probably not the right approach, but solves my need. I'll see if I can rethink it and add tests before I make a PR, but I'll leave this here in case anyone else needs it.

// iterate over all files
// check if an entry with the same dir and base name exists in dupes
// if in dupes and key not same delete from files
Object.keys(files).forEach(function(file){
  var check = files[file];
  Object.keys(dupes).forEach(function(dupe){
    var duplicate = dupes[dupe].path;
    if (
      check // Due to double iterating and deleting entries, need to check if key still exists
      && check.path // Some entries do not have path
      && (check.path.dir + check.path.base) === (duplicate.dir + duplicate.base) // Check to ensure they're indeed duplicates, and not files with same name in different locations
      && file !== dupe // Check keys are different, as permalink assigns same key for it's relative files function
    ) delete files[file]; // Delete entry
  });
});

[tomyam1]

I came here because I had the same issue. This is actually the expected and documented behavior of the plugin. Just set the relative: false when initializing the plugin. https://github.com/segmentio/metalsmith-permalinks#relative-files

[jarodtaylor]

@tepez That's a different "issue". Setting relative:false will not prevent the following:

├── blog
|   |   |   ├── 09-17-2015
|   |   |   |   ├── first-post.md
|   |   |   |   ├── first-post-specific-image.png

Compiles to:

|   ├── blog
|   |  ├── 09-17-2015
|   |  |  ├── first-post-specific-image.png
|   |  ├── first-post
|   |  |  ├── first-post-specific-image.png
|   |  |  ├── index.html

[webketje]

With the relative option removed, this issue is no longer relevant