i18nliner-js

i18nliner-js copied to clipboard

running the check command is quite slow, depending on the number of files in your directory... even though we auto-ignore node_modules, we still recursively walk everything (and non-js stuff too!) before selecting/excluding stuff

some ideas:

1. be able to specify a directory/directories where your js files live and only look in there
2. reimplement globby to do lazy enumeration. a big win there is we could break up exclusion patterns and not walk directories we know we don't need to

might be worth looking into the other globby to see if we can build on top of that instead

"extract" task is taking about 350secs for me, with aprox 77 files,.. too slow :(

even I executed is a SSD drive, I don't want to think how many time it will take in a normal hard disk. there is something that I can do to minimize that time?

Best Regards.

Yikes, that's terrible :(

I actually implemented item 1. from the description a while back, but forgot to update this issue. If you create a .i18nrc file at the root of your project and you specify the directories you care about, it should dramatically speed things up. For example:

{
  "directories": ["app", "lib"]
}

i'm guessing that 99% of the slowness you're seeing is due to globby traversing all the node_modules (or other directories that don't matter), and inefficiencies related to that. I've got a project with over 1600 js files, and i18nliner check takes about ~22 seconds (also on a SSD). Before I added the directories setting, it was completely unusable.

So hopefully in your case if you add that, you should see the runtime drop down to just a second or two.

Thank you for your sooner reply, I will try it.

yeap it reduce dramatically,.. I was set a list of exclude files into .i18nignore file,.. but still it was very slower,..

but setting up the
{ "directories": ["app", "lib"] } in the .i18nrc worked,..

Thank you for your help.

@jenseng I'm seeing the same slowness, but specifying directories in the root does not help... I assume that's because I'm trying to run the i18nliner export command and it does not pick up the setting... I've hacked around it by manually setting this.directories to my directories, hardcoded in the sources, but obviously that's not the solution. Any ideas on why the setting is not working for me?

I have sprinkeled the source with console.log statements. Here is my command and the logs I'm seeing:

Frank@TOAD c:\ws\ba
> node node_modules\i18nliner\bin\i18nliner check --only=src/ui/app/**/*
starting
argv= { _: [ 'check' ], only: 'src/ui/app/**/*' }
Commands= { run: [Function: run],
  Check: { [Function: Check] processors: { JsProcessor: [Function: JsProcessor] } },
  Export: [Function: Export] }
run check { _: [ 'check' ], only: 'src/ui/app/**/*' }        <=== NOTE i also tried `only` option...
run: creating new command...
Check: calling GenericCommand...
Check: called GenericCommand.
Check: translations= TranslationHash { translations: {} }
setUpProcessors: starting...
AbstractProcessor TranslationHash { translations: {} } { translations: TranslationHash { translations: {} },
  checkWrapper: [Function: bound ],
  only: 'src/ui/app/**/*',
  directory: undefined }
AbstractProcessor: this.directories= undefined   <== .i18nrc file has `directories` option but still undefined
setUpProcessors: done
run: running command
Check.run: started at 1478025207200
Check.checkFiles: checking 1 processors...
Check.checkFiles: checking processor 0:  JsProcessor {
  translations: TranslationHash { translations: {} },
  translationCount: 0,
  fileCount: 0,
  checkWrapper: [Function: bound ],
  pattern: '**/*.js',
  file: undefined,
  only: 'src/ui/app/**/*',
  directories: [ 'src/ui/app' ] }        <== Here, i've set it hardcoded
AbstractProcessor.checkFiles
AbstractProcessor.checkFiles: checking 1 directories...
Check.checkFiles: done.


Finished in 0.014 seconds

0 files, 0 strings, 0 failures
run: command completed. result= true

If I don't add the directories setting, it takes forever (I always quit it because I thought it hung)

I think the issue is here:

https://github.com/jenseng/i18nliner-js/blob/master/lib/processors/abstract_processor.js#L36

var fileScope = Globby.
      select(pattern).
      reject(["/node_modules", "/bower_components"]).
      reject(I18nliner.ignore());

It first does select on everything (which returns a giant list of all files I think), then rejects some of those afterwards... but by now it's already scanned through all of node_modules.

I don't think it's possible with gglobby to do it the other way around? E.g.:

var fileScope = Globby.
      reject(["/node_modules", "/bower_components"]).
      select(pattern).
      reject(I18nliner.ignore());

Another option would be for gglobby to do lazy evaluation in combination with a breadth-first search... But that would probably require a major overhaul to gglobby...

In the end, in i18nline, I just added an extra setting ignoreDirectories, which defaults to ['node_modules', 'dist', 'bower_components', '.git']. It's a bit of a cop-out but I just grab all the directories in the root project folder, filter out the ones matching ignoreDirectories and only then proceed to gglobying them... This works great! Speed goes from taking forever to 0.3 seconds ;) Only downside is that it won't find source files in the root folder this way.