Roadmap considerations

[oblador]

Posting this for discussion would love any type of input here, either do/don't/doesn't matter, do this instead, other considerations, what's most important to you, what you'd be able to help out with etc.

Move to jest

It's hard to write a great test runner with all types of different outputs (like JUnit, coverage etc). Currently we're using Listr to get a slightly nicer output of the progress, but it's flawed because it will swallow console.log output, is limited in how it handles concurrency also sometimes it will just output every frame instead of updating it. So my thought was to not try to innovate/solve all of these problems but use something existing that works well and my thought was to use jest.

Native concurrency

Running native tests are slow due to the screenshot process in the simulator itself being slow. That can be improved in multiple ways, for one we could take the screenshot from within the app itself and feed it over websockets (I've done this and it's fast) but it has the problem of not being accurate in some edge cases and requires integration of native libraries which makes it harder to get started. It should now be possible to run multiple simulators simultaneously but there's currently no way of sending commands to a specific connected device without doing alterations to the code itself. I've described it here: https://github.com/oblador/loki/issues/24 I guess my consideration is if I should give up on being a very light integration or trying to solve it in making storybook more flexible.

Visualisation/management

In my research I didn't find a good OSS library that just handled the diff management well (showing you what the differences were in the browser and enabling you to accept changes). There are some projects, but they either try to solve a bigger problem that are similar but not really the same or they just aren't solving it well enough. There's a few hosted services that one could use, which is probably the best solution for most cases. However there are OSS projects w/o money and sensitive projects where using hosted solutions is difficult for multiple reasons where a self hosted thing would be much easier. At some point I wanted to make a stand alone project (I named it Sigyn) to cater to these needs, but I'm not sure if it's better to focus on solving loki problems first. I'm afraid of that both will hinder loki adoption, either loki itself doesn't get enough attention to be good or the overall experience is not good to be used by devs.

Ping @nowells & @nuc

[nowells]

Move to jest

I like the idea of using jest to be runner, it seems like a great task runner with built in mechanics for concurrency, CLI outputting, and test reporting. I was looking over Listr and noticed some of these pain points where it was not really meant to solve the problems we are evolving into (but provided a nice system to start with for sure)

Native concurrency

I have not looked into running on native devices, so I will need to research/play around with the limitations there and the need to snapshot in app, but sounds interesting. I really like taking advantage of the willingness of storybook team to help improve their API to make testing easier, I am sure with thought we could provide some solid ideas (or solid PRs to them with features that would help)

Visualisation/management

I would love to add a visualization report. Before I came to loki I gave storybook-chrome-screenshot + reg-suit a shot. And I liked the simple html report that reg-suit provided which when run locally or on CI users could just open the file and see the output visually (new, deleted, changed) without needing to do something like ksdiff or opening the differences pngs directly.

I really love the interface that resemble.js provides in their demo https://huddleeng.github.io/Resemble.js/ click Use Example Images and could see making a UI that is similar to that or github/ksdiff swipers to visualize the difference.

But one of the things I think makes sense to keep loki focused on is ensuring it works in CI only automated validation mode first (producing static reports that can be opened and viewed in CI results) and then adding progressive enhancement for local development where that same static report can be served through a server where it could add in approvals and actions that could more closely match tools that focus on external approvals. Loki could span both use cases really well. I feel like focusing on making the image difference report (resemblejs/github/ksdiff) instead of just static images, interactive diffs would be a great space to dive into.

browsers

I noticed on the wish list browserstack/saucelabs integration. I think that is a great idea to open up ALL browsers for complete regression ability, however, one of the things I love about loki is that out of the box you get consistent image generation with docker image run locally and in CI. That is a HUGE selling point. Now just adding Firefox in Docker to round it out. I think we could definitely add a browserstack/saucelabs.

configuration

One thing that I know my team is facing is that in our organization, my team centrally manages general toolchain integration configuration, and we then allow our other internal teams to start with our standardized tools/configuration defaults and then extend. having configuration in package.json only makes that impossible to roll out sane defaults and evolve them, so looking into something like https://www.npmjs.com/package/rc-config-loader would make adoption for teams easier (centralizing their config they use in multiple repos). We do this with karma and eslint etc by having our own plugins/or base configurations that people import and then extend.

[nowells]

I am also curious what you think of resemblejs compared to GraphicsMagick. Resemble would not require native binaries to be installed seperately, and would give us access to do browser side report generation (as seen in https://huddleeng.github.io/Resemble.js/) which would be pretty slick for a more in-depth report to see what really changed.

[nuc]

jest

Moving to jest sounds a really nice idea. And it's also CI compatible out of the box.

In addition, we can have a look on how storyshots are integrated with jest. I don't know how much effort that might need, but it might make it possible to run loki under your usual yarn test --watch, and also give an interface for updating the reference images (similarly as -u updates the snapshots). That would be awesome 💯

native

Regarding native, I don't have much experience yet, but we are starting a react-native & storybook project these days, so I will probably have some feedback soon.

Visualisation/management

I like @nowells idea of a static reports, but I agree that it is really important to first make loki super stable. We do have a couple of pain points which slow development down for us.

browsers configuration

👍

[oblador]

Thank you for your input!

jest --watch would be pretty cool to have and I've actually thought about it myself before, I'm just a bit concerned that it would eat too much CPU to be feasible and not sure how much value it would actually provide as the pattern of development is slightly different. -u option (or just pressing u in watch mode) is def something we should support.

On the same topic, I'm thinking about making the whole thing a bit more modular, so that if you wanted to support another image differ or a storybook competitor, it would be possible to write it yourself. What do you think?

@nuc I'm strongly leaning towards that loki needs to be paired with a proper tool to review changes visually (and ideally be able to approve them). I have some ideas on how to do that, but it will be an optional and separate project in that case that should be possible to integrate with existing CIs. For that matter, I don't see why you couldn't use reg-suit with loki. Maybe we should start with setting up an example and documenting on how to do it yourself.

@nowells Regarding configuration, I agree that it seems to be a painpoint for users and I want to support that. Since it's dear to you, would you mind giving it a stab to solve that?

[nowells]

@nowells Regarding configuration, I agree that it seems to be a painpoint for users and I want to support that. Since it's dear to you, would you mind giving it a stab to solve that?

Absolutely! I will be on vacation this weekend, but will take a stab at it next week.

reg-suit overlaps with loki in the generation of differences. It does have an interesting plugin system to allow different phases to be configured (notification, publishing of assets, setup, etc.) which we could take conceptually and try to integrate (use that to allow us to add github status api plugin, or slack plugin, or....)

[nowells]

So I was playing around with the idea of using code generation to generate a jest test file to run our tests. I would use jest-worker inside each test to handle pooling and parallelization caps, and run test.concurrent all within a single file. You can play with the basic stub by running node setup.js && jest loki.test.js. So my thought being that we can have an adapter that generates the test expectations that are essentially calls to our worker to load a url and snapshot it. You can envision then that the calls to snapshot are totally separate from the construction of stories, and you could even use loki tools without storybook, as it would just be the construct of jest-worker API that take URLs and other metadata to screencap and compare as part of a test. Is this totally out of line of what you were thinking when talking about using jest?

setup.js

const puppeteer = require('puppeteer');
const fsExtra = require('fs-extra');
const stripIndent = require('strip-indent');

(async() => {
  const browser = await puppeteer.launch();

  try {
    const page = await browser.newPage();

    await page.goto('http://localhost:6006');

    const stories = await page.evaluate(() => {
      return window.frames[0].loki.getStorybook().map(component => ({
        kind: component.kind,
        stories: component.stories.map(story => story.name),
        skipped: component.skipped,
      }));
    });

    const setupCode = stripIndent(`
        const Worker = require('jest-worker').default;

        const worker = new Worker(require.resolve('./worker.js'));

        afterAll(() => {
          worker.end();
        });
    `);

    const testCode = stories.map(({kind, stories}) => {
      return stripIndent(`
        describe(${JSON.stringify(kind)}, () => {
          ${stories.map(story =>
          `test.concurrent(${JSON.stringify(story)}, async() => {
             const kind = ${JSON.stringify(kind)};
             const story = ${JSON.stringify(story)};

             const imageDifference = await worker.runTest(kind, story);
             expect(imageDifference).toEqual(0);
          }, 10000)
          `
          ).join('')}
        });
      `);
    }).join('');


    fsExtra.writeFile('loki.test.js', `${setupCode}${testCode}`);
  }
  finally {
    await browser.close();
  }

})().catch(err => {
  console.error(err);
  process.exit(1);
});

worker.js

const puppeteer = require('puppeteer');
const url = require('url');

module.exports.runTest = async function(kind, story) {
  const browser = await puppeteer.launch();

  try {
    const page = await browser.newPage();

    await page.goto(url.format({
      protocol: 'http',
      hostname: 'localhost',
      port: '6006',
      query: {
        selectedKind: kind,
        selectedStory: story,
        full: 1
      }
    }));
  }
  finally {
    await browser.close();
  }
}

[nowells]

I also envision cases where you want to be able to have each test be configured to have various tests run against it, so I could run my BrowserStack/SauceLabs with IE@11 screenshots only on a subset of tests, and have the rest of them run on chrome.docker because they are faster/cheaper. I really like breaking apart the various responsibilities of loki to make building blocks that can be reused and recombined in different ways.

[sibelius]

Detox provides screenshots https://github.com/wix/detox/releases/tag/8.0.0

[priomsrb]

Regarding visualisation: I recently created a PR for reg-cli that adds extra tools for comparing images: https://github.com/reg-viz/reg-cli/pull/199.

I've been using loki with reg-cli to generate reports and it works quite well for me. Hoping to see improvements to both in the future.

[oblador]

@nowells I didn't actually prototype or investigate exactly how yet, but my thought would be to have an API similar to this (very rough). My assumption is that you should be able to map the stories to tests in runtime without having to do code generation and writing to a file.

import loki from '@loki/jest'
import createChromeDockerTarget from '@loki/target-chrome-docker';
import diffingEngine from '@loki/diffing-looks-same';

describe('Chrome iPhone 7', () => {
  const selector = '.wrapper > *, #root > *';
  const target = createChromeDockerTarget({ width: 1366, height: 768, selector });

  // these could just as well be part of loki() but would allow for some granularity, 
  // such as doing some processing after the target has started etc
  beforeAll(target.start); 
  afterAll(target.stop);

  loki({ target, diffingEngine });
});

[oblador]

@sibelius Think detox is a slightly different use case than loki currently aims at, or did you mean to use it for parallelization? Detox is very invasive into the application and fairly painful to set up and want loki to be easy to get started as long as your project isn't super custom.

[sibelius]

the idea is to bring some of ideas of screenshot of detox to this project

not sure if this is possible or feasible

[oblador]

@sibelius By reading the code to me it seems like they are using the same approach as loki for taking screenshots.

[bahriddin]

Move to jest

I was researching on the internet to run loki with only changed files and found this issue. It is a great idea if it supports --watch mode as @nuc mentioned. Or at least options like --onlyChanged, --lastCommit and --changedSince would be great.

Thanks!

[shzhng]

Move to jest

@oblador First off, thank you for creating loki! It's been a massive boon to our testing practices.

What do you think is the likelihood of getting switched over to jest? We would love to use this for a bunch of features mentioned in @bahriddin's note above, as well as junit output for our CI set up.

Is there anything I can do to help with this? If you point me in the right direction, I might be able to dedicate some time to contributing.