E2E Tests with Protractor

[ltfschoen]

Progress

@diegonetto I want to create a pull request and update the Readme section on E2E Tests using Protractor. With the way I have set it up on the latest commit on my custom Generate Ionic app so far I have managed to get Protractor to work but only manually with the following command (and it performs similar to Capybara with Selenium Web Server in Rails with the feature tests automatically opening my browser, and entering inputs into form fields that I have predefined):

protractor protractor.conf.js

I have attempted to register the task in the Gruntfile.js (by adding an additional line "protractor:run" under "grunt.registerTask('test'...") but when I do and run the following command it just stops and doesn't run properly:

grunt test

Setup Methodology for Manual Protractor Testing

• Followed the Protractor Tutorial Here
• Installed Protractor and Web-Driver Manager (Selenium Server and Chromedriver)

npm install -g protractor

• Updated the Web-Driver Manager

webdriver-manager update

• Started the Selenium Server using a New Terminal Tab and the following command:

webdriver-manager start

• Created a File in the Root Directory of the App

touch protractor.conf.js

• Opened protractor.conf.js and added the following code (noting that the 'capabilities' section was added to overcome a browser error that appeared when trying to run the server (I used the Stackoverflow solution by sheffield Here

exports.config = {
  seleniumAddress: 'http://localhost:4444/wd/hub',
  capabilities: {
    'browserName': 'chrome',
    'chromeOptions': {
        args: ['--test-type']
    }
  },
  specs: ['./test/e2e/*_test.js'],
  baseUrl: 'http://localhost:9000' //default test port with Yeoman is 127.0.0.1 (localhost)
};

• Created an End-to-end Test (E2E) named 'scenarios_test.js' and included it in the /test/e2e/ directory (after creating the 'e2e' subfolder). Note that the code inside the 'it' block is just an example of the test I used

describe('YeomanIonic App', function() {
  describe('Sentences view', function() {
    beforeEach(function() {
      // match the address that opens with command 'grunt server'
      browser.get('/'); // opens /app/index.html (root)
    });
    // verify search box and repeater are correctly wired together
    it('should filter the sentences list as user types into the search box', function() {
      var sentenceList = element.all(by.repeater('sentence in sentences'));
      var query = element(by.model('query'));
      expect(sentenceList.count()).toBe(2);
      query.sendKeys('sentence');
      expect(sentenceList.count()).toBe(2);
      query.clear();
      query.sendKeys('1st');
      expect(sentenceList.count()).toBe(1);
    });
  });
});

• Run the Protractor server (where 'webdriver-manager start' command has already been executed in a separate terminal tab, as it must already be running first)

protractor protractor.conf.js

• The above all works great... but I wanted it automated with the 'grunt test' command, so I tried some more stuff below

Setup Methodology for Automated Protractor Testing

• Discovered and followed the steps of this awesome post by user2172816 on Stackoverflow Here
• Added to the Gruntfile.js after the Karma section

    protractor: {
      options: {
        keepAlive: true,
        configFile: "protractor.conf.js"
      },
      run: {}
    },

• Updated my package.json with the following after "devDependencies" to run the command after npm install to hopefully make it automated

 "scripts": {
    "install": "node node_modules/protractor/bin/webdriver-manager update",
    "test": "grunt test"
  }

• Added the E2E section in the Gruntfile.js as shown below (hoping this would watch and update the automated tests upon any changes in the e2e directory):

 grunt.registerTask('watch:karma', function () {
    var karma = {
      files: ['<%= yeoman.app %>/<%= yeoman.scripts %>/**/*.js', 'test/spec/**/*.js', 'test/e2e/**/*.js'],
.....

• And Finally...... Added the Protractor task 'protractor:run' under the registration of task 'test' in the Gruntfile.js (which doesn't seem to work unfortunately, see the earlier screenshot that displays error "Warning: Task "protractor:run" not found. Used --force, continuing. Done, but with warnings."). I also tried adding 'connect:test' and 'karma', which was shown in the Stackoverflow post, but doing so didn't help either):

 grunt.registerTask('test', [
    'clean:server',
    'concurrent:test',
    'autoprefixer',
    'karma:unit:start',
    'watch:karma',
    'protractor:run'
  ]);

Any ideas?

My system info

• from typing 'chrome://version/' in my browser: Google Chrome 37.0.2062.94 (Official Build 290621) OS Mac OS X JavaScript V8 3.27.34.14 Protractor Version 1.2.0

yo --version && echo $PATH $NODE_PATH && node -e 'console.log(process.platform, process.versions)'
1.2.1
/Users/Luke/.rvm/gems/ruby-2.1.2/bin:/Users/Luke/.rvm/gems/ruby-2.1.2@global/bin:/Users/Luke/.rvm/rubies/ruby-2.1.2/bin:/Users/Luke/Documents/google-cloud-sdk/bin:/Applications/Postgres93.app/Contents/MacOS/bin:/user/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Users/Luke/code/adt/adt-bundle-mac-x86_64-20140702/sdk/tools:/Users/Luke/code/adt/adt-bundle-mac-x86_64-20140702/sdk/tools:/Users/Luke/code/adt/adt-bundle-mac-x86_64-20140702/sdk/tools:/Users/Luke/.rvm/bin
darwin { http_parser: '1.0',
  node: '0.10.31',
  v8: '3.14.5.9',
  ares: '1.9.0-DEV',
  uv: '0.10.28',
  zlib: '1.2.3',
  modules: '11',
  openssl: '1.0.1i' }

i get the following info about my ChromeDriver version when i run the command 'webdriver-manager start'

Setting system property webdriver.chrome.driver to /usr/local/lib/node_modules/protractor/selenium/chromedriver
14:01:20.278 INFO - Java: Apple Inc. 20.65-b04-466.1
14:01:20.279 INFO - OS: Mac OS X 10.9.3 x86_64
14:01:20.407 INFO - v2.42.2, with Core v2.42.2. Built from revision 6a6995d

[ltfschoen]

I overcame the warning mentioned earlier (Warning: Task "protractor:run") by registering a 'protractor:run' task, as shown below, but it still does not execute 'protractor protractor.conf.js' when I run 'grunt test' at the command-line.

  grunt.registerTask('protractor:run', function () {
    var run = {
      files: ['protractor.conf.js'],
      tasks: ['protractor protractor:run']
    };
    grunt.config.set('protractor', run);
    return grunt.task.run(['protractor']);
  });

Also, as I do not have complex tests at this stage, I updated my 'protractor.conf.js' file to start selenium in standalone mode (i.e. so i don't have to run 'webdriver-manager start' in a separate terminal window first before running 'protractor protractor.conf.js'). Additionally, I added the multi capability so it simulates the feature tests in both Chrome and Firefox (however I get errors with Firefox). I ran the following two commands in terminal first:

npm install --save-dev protractor
npm install --save-dev chromedriver

exports.config = {
  chromeOnly: true,
  chromeDriver: './node_modules/chromedriver/bin/chromedriver',
  multiCapabilities: 
    [{
      'browserName': 'chrome',
      'chromeOptions': {
      args: ['--test-type']
      }
    },
    {
      'browserName': 'firefox'
    }
  ],
  onPrepare: function() {
     browser.driver.manage().window().setSize(800, 600);
  },
  specs: ['./test/e2e/*_test.js'],
  baseUrl: 'http://localhost:9000', //default test port with Yeoman is 127.0.0.1 (localhost)
  jasmineNodeOpts: {
    showColors: true,
    isVerbose: true,
    includeStackTrace: true
  }
};

[jimthedev]

@ltfschoen First off, great job. I am glad someone is working on a PR for this.

I have E2E/protractor tests implemented and I started off using this generator for my project so I am happy to give some input on this if it helps get that back into the original generator. My configs are weird because we use Saucelabs to run our tests on our CI server, but give devs the ability to run the tests locally if they prefer. At first I tried something like your method, using protractor directly, but then later decided to use https://github.com/teerapap/grunt-protractor-runner which some of the heavy lifting for you.

I ended up with two separate commands that I call depending on which testing environment I want to run my tests in. There certainly might be a better way to do this, but currently I haven't had a chance to investigate further.

Run e2e tests at saucelabs

grunt protractor:sauce

Run e2e tests locally

grunt protractor:local

To get it working, I had to make the following changes to my Gruntfile.js:

Inside of grunt.initConfig({ ... here.... }); i inserted:

protractor: {
      options: {
        configFile: 'node_modules/protractor/referenceConf.js', // Default config file
        keepAlive: false, // If false, the grunt process stops when the test fails.
        noColor: false, // If true, protractor will not use colors in its output.
        verbose: true,
        args: {
          // Arguments passed to the command
        }
      },
      sauce: {
        options: {
          verbose:true,
          keepAlive: false,
          configFile: 'protractor.sauce.conf.js',
          args: {}
        }
      },
      local: {
        options: {
          verbose: true,
          keepAlive:false,
          configFile: 'protractor.local.conf.js',
          args: {}
        }
      }
    }

At the end of my Gruntfile.js:

grunt.loadNpmTasks('grunt-protractor-runner');

protractor.local.conf.js looks like this:

/* The local protractor config */
exports.config = {
  baseUrl: 'http://localhost:9000/index.html',
  multiCapabilities: [
    {
      'browserName': 'safari'
    }, {
      'browserName': 'chrome'
    }
    // Commented out until protractor upgrades to webdriver v2.43
    // which supports Firefox 32
    //, {
    //  'browserName': 'firefox'
    // }
  ],
  specs: ['test/spec/e2e/**/*.js'],
  framework: 'jasmine',
  jasmineNodeOpts: {
    showColors: true
  }
};

My saucelabs bamboo config looks like this:

var path = require('path');
/* Transform the bambo sauce environment variable into a protractor variable */
var browsersTransformed = [];
var browsersFromEnvironment = JSON.parse(process.env.bamboo_SAUCE_ONDEMAND_BROWSERS);
browsersFromEnvironment.forEach(function(browser){
  browsersTransformed.push({
    'browserName': browser.browser,
    'name': 'Mobile E2E',
    'build': process.env.BAMBOO_BUILDNUMBER
  });
});

/* The protractor config */
exports.config = {
  seleniumPort: process.env.SELENIUM_PORT,
  baseUrl: 'http://localhost:9000/index.html',
  multiCapabilities: browsersTransformed,
  sauceUser: process.env.SAUCE_USER_NAME,
  sauceKey: process.env.SAUCE_API_KEY,
  specs: ['test/spec/e2e/**/*.js'],
  framework: 'jasmine',
  jasmineNodeOpts: {
    showColors: false
  }
};

Obviously to use grunt-protractor-runner you will need to add it via npm as well. One thing about going this route is that you should make sure to do is to ensure that you run node_modules/grunt-protractor-runner/scripts/webdriver-manager-update to make sure you are using the local version of webdriver inside the project instead of relying on a globally installed version.

[ltfschoen]

@jimthedev Thanks Jim for sharing some great tips, I'll be sure to test them out!

[kristianmandrup]

@jimthedev Awesome that you got Protractor running on SauceLabs with Bamboo integration. Exactly such a recipe I have been looking for all day. Any chance you could write a blog post on how this setup works in more detail? Which resources did you use to piece this together? How do I integrate the Bamboo and/or Sauce test reports with JIRA or similar external tools? Cheers!

[jimthedev]

@kristianmandrup I think that would be possible but it might be a couple weeks. I've got some hard deadlines the next two weeks, so if you have pressing questions I can answer them in brief or you can wait. Whatever you prefer.

[kristianmandrup]

Thanks ;) No rush. In a few weeks would be excellent. We will see how far we can get from tips you have provided so far...