playwright
playwright copied to clipboard
microsoft
[Feature] Custom test function locations
I'm building a wrapper to make the playwright test runner handles the Gherkin syntax.
The Playwright test runner shows test definition location in reports and its UI mode. I would like playwright to use the test definition locations from the feature file instead of the actual function definition ones that point within the wrapper files.
See full discussion about this feature here on the wrapper repo
Current behaviour
// example.ts
import { test } from "@playwright/test";
test.describe('title', () => {})
Playwright compute the location for this describe block as { file: 'c:\\...\\example.ts', line: 4, column: 1 }
New behaviour
// example.ts
import { test } from "@playwright/test";
test.describe('title', () => {
test('title', () => {}, { file: 'c:\\...\\example.feature', line: 7, column: 4 })
})
Playwright compute the location for the describe block as { file: 'c:\\...\\example.ts', line: 4, column: 1 } and use { file: 'c:\\...\\example.feature', line: 7, column: 4 } for the test block.
It should work this way for every test function registration methods (test/test.describe/test.step/test.skip/...)
Suggested implementation
I made this modifications in my node_modules and it works fine.
Update this function: https://github.com/microsoft/playwright/blob/2697e936639cab64faec9e4adfc4f2848e621cc4/packages/playwright-test/src/common/transform.ts#L229-L252
With this:
function wrapFunctionWithLocation<A extends any[], R>(func: (location: Location, ...args: A) => R): (...args: A) => R {
const isLocation(o: any): o is Location {
return o.line != undefined // this works but it doesn't feel very strong to me
}
return (...args) => {
if (isLocation(args[args.length - 1])) {
const location = args.pop()
return func(location, ...args)
}
const oldPrepareStackTrace = Error.prepareStackTrace;
Error.prepareStackTrace = (error, stackFrames) => {
const frame: NodeJS.CallSite = sourceMapSupport.wrapCallSite(stackFrames[1]);
const fileName = frame.getFileName();
// Node error stacks for modules use file:// urls instead of paths.
const file = (fileName && fileName.startsWith('file://')) ? url.fileURLToPath(fileName) : fileName;
return {
file,
line: frame.getLineNumber(),
column: frame.getColumnNumber(),
};
};
const oldStackTraceLimit = Error.stackTraceLimit;
Error.stackTraceLimit = 2;
const obj: { stack: Location } = {} as any;
Error.captureStackTrace(obj);
const location = obj.stack;
Error.stackTraceLimit = oldStackTraceLimit;
Error.prepareStackTrace = oldPrepareStackTrace;
return func(location, ...args);
};
}
@Niitch Playwright respects source maps if you generate them, so I'd recommend to generate a source map for the example.ts file that will correctly point to the example.feature. Let me know whether this helps.
@dgozman
For more context, here is how the wrapper is actually used:
// step-definitions/example.spec.ts
import { test as base } from "@playwright/test";
import GherkinWrapper from "gherkin-wrapper";
// Add fixtures
const test = base.extend<{value: string}>({
value: async ({}, use) => {
await use('go')
}
})
// Build the wrapper
const wrapper = new GherkinWrapper.forPlaywright(test)
// Register step functions
wrapper.given(/the Maker has started a game with the word "(.*)"/, async ({ page, value }, { match }) => {
// Do things ...
})
wrapper.when("the Maker starts a game", () => {...})
wrapper.then("the Maker waits for a Breaker to join", () => {...})
// Run tests
wrapper.test('./features/example.feature')
Behind the hood, the wrapper performs the actual calls to test.decribe/test/test.step/...
NB: The step function definitions show well in reporters and the UI mode
I am very new too source maps. It seems that they provide only 1 to 1 mappings between elements of the actual code and their representation.
The wrapper can't rely on this behaviour as it uses the same call definition for each test.decribe/test/test.step/... Every describe blocks are defined by the same code, and so it is for test, step and hooks blocks.
I rather need a 1 to many mapping but I can't find a way to do it with source maps.
@Niitch I see, you have a runtime wrapper, that's likely not going to work nicely with source maps as you described.
I am not really sure whether this is generated code, or not. If it is not generated, I don't think we have a good answer for you.
If the code is generated, I'd recommend to restructure it a bit and replace wrapper.test() with actual test() calls instead:
test('./features/example.feature', async ({ page }) => {
await wrapper.runTestFromFeature('./features/example.feature');
});
This way you should be able to leverage source maps, because you call test() in multiple places now. Let me know whether this helps.
@dgozman
It is indeed a runtime wrapper that does not generate code.
The playwright wrapper is defined here
The solution you suggest is not practicable. It would work, but I would need to restructure the code so that it works this way for describe and step blocks too. As a result, people would need to call the playwright test methods themselves for each line of the feature file, hence loosing all the wrapper benefits.
Are there any drawbacks to the solution I suggested in my first message ?
Are there any drawbacks to the solution I suggested in my first message ?
Well, it involves adding a new API. And since we haven't seen interest in such an API before this issue, it will not be prioritized. So figuring out a workaround would unblock you.
Hello @dgozman, actually this ticket is related to the most voted feature, so maybe you could prioritized it.-> https://github.com/microsoft/playwright/issues/11975
Many people expect some solution concerning the support of BDD from the Playwright runner (thanks @Niitch for developing a runtime wrapper) and having that new api will very much help with that.
cc: @aslushnikov , @pavelfeldman
@NikkTod I would encourage you to consider the sourceMap-based approach where you generate JS files that map back to the feature files. That way all the existing tooling, including step debugging in VS Code would be supported out of the box. The problem of using JS runtime to run non-JS (transpiled) code is not new and sticking with the established practices would make all the tool chains play nicely together.
Thanks @pavelfeldman, this was a very enlightening!
I wonder how the jumping between the .feature file and the step definitions in .ts / .tsx files would look. Could be awesome!
Side node: https://github.com/vitalets/playwright-bdd is already doing codegen, but no sourceMaps and a different API that is more similar to cucumber than to playwright.
I'm curious to hear what you think @Niitch
I knew about vitalets/playwright-bdd, but I wanted to try a non-codegen approach. I want a very playwright-native-like experience for the user.
I'll search about implementing codegen and sourcemaps.
This is how I'm mapping sources for test.step() calls in playwright-bdd.
Use private method testInfo._runAsStep where you can pass custom location:
const location = {
file: '/path/to/sample.feature',
line: 4,
column: 7,
};
// @ts-expect-error _runAsStep is private
await testInfo._runAsStep({ category: 'test.step', title: text, location }, () =>
stepDefinition.code.apply(world, parameters),
);
Then html report shows correct locations from feature file:
Hope this helps @Niitch in your work as well.
I would also really like this feature for a Playwright integration I'm writing. I just hacked together a quick proof-of-concept after going through this discussion, so I wanted to share what I ended up with.
I have no idea if any of this is a good idea, in fact I'm pretty sure it's not, but it works as of 1.34.3.
// example of how i'm integrating my service with playwright
const test = base.extend({
service: async ({}, use) => {
const service = new MyService();
await use(service);
}
})
class MyService {
constructor() {
this.doSomething = wrapAsStep(this.doSomething, this);
}
doSomething() {}
}
// wrapper for internal playwright testInfo._runAsStep so we can add steps with params
const runAsStep = async <T>(
location: any,
args: {
title: string;
params?: any;
},
fn: () => T
) => {
const info = test.info() as any;
return info._runAsStep(
{ category: 'test.step', ...args, location },
() => fn()
) as T;
};
// returns a bound version of the function that will run as a step
function wrapAsStep(fn: (...args: any[]) => any, context: any) {
return function (...args: any[]) {
const location = captureLocation(new Error());
return runAsStep(
location,
{
title: `service.${fn.name}`,
params: args
},
() => fn.apply(context, args)
);
};
}
// this may be fragile
function captureLocation(e: Error) {
const stack = e.stack;
if (!stack) return;
const lines = stack.split('\n');
const line = lines[2];
const match = line.match(/at (.*)$/);
if (!match) return;
const location = match[1];
return {
file: location.split(':')[0],
line: location.split(':')[1],
column: location.split(':')[2],
};
}
Ideally, I would just like public API for testInfo._runAsStep that accepts extra info to show in trace viewer, such as params, and of course location.
// this may be fragile
For capturing location of particular function in file you can have a look on Playwright's wrapFunctionWithLocation that gets file, line and column using stacktrace API.
@vitalets @mattjennings The soutions you suggested are good but only work test steps.
I have finally found a way to pass custom location objects for every functions of the test runner by calling methods of the internal TestTypeImpl class. You can access it like so.
@dgozman With this solution we no longer need to extends the playwright API.