robotframework
robotframework copied to clipboard
robotframework
Logging keyword instead of dot to console
I'm planning to add a few more commits to merge request:
- Show the keyword name and library name when it started too;
- Try to show only one line when the keyword is some final action;
- Change this change the original verbose.py to another name like normal.py and change it to default;
- Add some way to control the left marging in this to show information of one level with the same indentation.
What do you guys think about? Should I show a preview of the final verbose level to you?
Sounds cool. Do you plan to show all executed keywords? I'm afraid that would create too much output. Would it be enough to show just the top level kws? Or should we add both verbose and very-verbose modes?
@pekkaklarck
I was planning to do this two modes verbose and very-verbose, but I'm now working in a version with the verbose mode and a flag to say how deep to print the keywords. The normal version should be this flag set to zero and if the people need more verbosity change it. Something like:
------------------------------------------------------------------------------ with verbose_level = 3
Test1
resource_templates.Importar arquivo de regex "regex.json"
funcoes_comuns.Importar arquivo ${VG_PATH_REGEX}${p_regex...
SSHLibrary.Get File ${p_path_file_src} ${p_path_file... | PASS |
funcoes_comuns.Importar arquivo ${VG_PATH_REGEX}${p_regex... | PASS |
resource_templates.Importar arquivo de regex "regex.json" | PASS |
Test1 | PASS |
------------------------------------------------------------------------------ with verbose_level = 0
Test1 | PASS |
The verbose console mode will not change behaviour if not informed the two new flags: --verbositylevel and --labelmarker. The flag --verbositylevel will be used to control how deep the print should happen and it is set to zero by default to try not change behaviour with the previous version. Also, the way of print the keyword can be controlled by the other flag mentioned.
I updated the branch again because it was several commits... If someone like to see a example of how it works:
$ ./src/robot/run.py --dryrun --verbositylevel 1 ../robotFrameworkSmoke/markers.robot
[ ERROR ] Error in file '/home/rlucca/Git/robotFrameworkSmoke/markers.robot': Non-existing setting 'Non Existing'.
==============================================================================
Markers
==============================================================================
Suite Setup
BuiltIn.No Operation | PASS |
Suite Setup | PASS |
------------------------------------------------------------------------------
Few Pass Markers
BuiltIn.No Operation | PASS |
BuiltIn.No Operation | PASS |
BuiltIn.No Operation | PASS |
BuiltIn.No Operation | PASS |
BuiltIn.No Operation | PASS |
Few Pass Markers | PASS |
------------------------------------------------------------------------------
Few Pass And Fail Markers
BuiltIn.No Operation | PASS |
BuiltIn.No Operation | PASS |
BuiltIn.Run Keyword And Continue On Failure Fail | PASS |
BuiltIn.No Operation | PASS |
BuiltIn.Run Keyword And Continue On Failure Fail | PASS |
BuiltIn.No Operation | PASS |
BuiltIn.No Operation | PASS |
Few Pass And Fail Markers | PASS |
------------------------------------------------------------------------------
More Markers Than Fit Into Status Area During Very Deep Keyword
BuiltIn.No Operation | PASS |
KeywordLevel1 | PASS |
KeywordLevel11 | PASS |
KeywordLevel12 | PASS |
BuiltIn.No Operation | PASS |
More Markers Than Fit Into Status Area During Very Deep Keyword | PASS |
------------------------------------------------------------------------------
Warnings Are Shown Correctly
BuiltIn.No Operation | PASS |
BuiltIn.No Operation | PASS |
BuiltIn.No Operation | PASS |
BuiltIn.No Operation | PASS |
BuiltIn.Log Warning WARN | PASS |
BuiltIn.No Operation | PASS |
BuiltIn.No Operation | PASS |
Warnings Are Shown Correctly | PASS |
------------------------------------------------------------------------------
Suite Teardown
BuiltIn.No Operation | PASS |
Suite Teardown | PASS |
------------------------------------------------------------------------------
Markers | PASS |
4 critical tests, 4 passed, 0 failed
4 tests total, 4 passed, 0 failed
==============================================================================
Output: /home/rlucca/Git/robotframework/output.xml
Log: /home/rlucca/Git/robotframework/log.html
Report: /home/rlucca/Git/robotframework/report.html
$ ./src/robot/run.py --dryrun --verbositylevel 0 ../robotFrameworkSmoke/markers.robot
[ ERROR ] Error in file '/home/rlucca/Git/robotFrameworkSmoke/markers.robot': Non-existing setting 'Non Existing'.
==============================================================================
Markers
==============================================================================
Few Pass Markers | PASS |
------------------------------------------------------------------------------
Few Pass And Fail Markers | PASS |
------------------------------------------------------------------------------
More Markers Than Fit Into Status Area During Very Deep Keyword | PASS |
------------------------------------------------------------------------------
Warnings Are Shown Correctly | PASS |
------------------------------------------------------------------------------
Markers | PASS |
4 critical tests, 4 passed, 0 failed
4 tests total, 4 passed, 0 failed
==============================================================================
Output: /home/rlucca/Git/robotframework/output.xml
Log: /home/rlucca/Git/robotframework/log.html
Report: /home/rlucca/Git/robotframework/report.html
I don't have time to look at the actual PR now, but I have some comments about the overall design:
- I don't see much need to show other than the top-level keywords in the test on the console. Showing also the lower level keywords is likely to just make the output a mess.
- I want this to be as easy to configure as possible. Adding two new command line options is too much. I'd prefer just adding
--verbose, changing--console verboseto mean the same, and adding new--console normalthat is the default output.
What do you think about this? Let me know if you'd like to chat about the design on Slack.
I tested this PR locally and noticed problems that should be fixed.
I used this dummy data:
*** Settings ***
Suite Setup Nested keyword
Suite Teardown Nested keyword
Test Setup Nested keyword
Test Teardown Nested keyword
*** Test Cases ***
Passing test
Log Hello, world!
Nested keyword
Failing test
Log Hello, world!
Nested keyword pass=False
*** Keywords ***
Nested keyword
[Arguments] ${pass}=True
Nested keyword 2 ${pass}
Nested keyword 3 ${pass}
Nested keyword 2
[Arguments] ${pass}=True
Nested keyword 3 ${pass}
Nested keyword 3
[Arguments] ${pass}=True
Should be true ${pass}
With --console verbose option I got this output:
==============================================================================
Verbose
==============================================================================
Suite Setup
Nested keyword
Nested keyword 2 ${pass} | PASS |
Nested keyword 3 ${pass} | PASS |
Nested keyword | PASS |
Suite Setup | PASS |
------------------------------------------------------------------------------
Passing test
Nested keyword
Nested keyword 2 ${pass} | PASS |
Nested keyword 3 ${pass} | PASS |
Nested keyword | PASS |
BuiltIn.Log Hello, world! | PASS |
Nested keyword | PASS |
Nested keyword
Nested keyword 2 ${pass} | PASS |
Nested keyword 3 ${pass} | PASS |
Nested keyword | PASS |
Passing test | PASS |
------------------------------------------------------------------------------
Failing test
Nested keyword
Nested keyword 2 ${pass} | PASS |
Nested keyword 3 ${pass} | PASS |
Nested keyword | PASS |
BuiltIn.Log Hello, world! | PASS |
Nested keyword pass=False | FAIL |
Nested keyword
Nested keyword 2 ${pass} | PASS |
Nested keyword 3 ${pass} | PASS |
Nested keyword | PASS |
Failing test | FAIL |
'False' should be true.
------------------------------------------------------------------------------
Suite Teardown
Nested keyword
Nested keyword 2 ${pass} | PASS |
Nested keyword 3 ${pass} | PASS |
Nested keyword | PASS |
Suite Teardown | PASS |
------------------------------------------------------------------------------
Verbose | FAIL |
2 critical tests, 1 passed, 1 failed
2 tests total, 1 passed, 1 failed
==============================================================================
Problems:
-
Suite setup and teardown look like tests. They should should be marked somehow separately or omitted altogether.
-
Test setup and teardown look like normal keywords. Would be nice to separate them somehow.
-
Two levels of keywords are shown without a way to configure that. In my opinion just the top level keyword should be shown to view test structure but not more complicated lower level keywords. There could be a way to configure the verbosity level, though.
-
I don't think it's a good idea to show test case names twice.
In my opinion the next step should be agreeing on the design and only after that change the implementation. Most importantly we should agree how the output itself should look like. Notice that the old output doesn't need to be used as a base, the new verbose output can be designed from scratch (and also the current output can be redesigned).
Another thing to decide is how to handle verbosity levels. We could make it the level configurable like --verbose 2, but due to how command line options work, it's not possible/easy to have an option that has a default value and thus the common case --verbose wouldn't work. I'm starting to think having verbose mode (--verbose) that shows top level keywords and a separate "very verbose" mode (--veryverbose) that shows all keywords would be best.
What do others think about my comments and high level design ideas above? Is anyone interested to work with the design and propose how the output should look like? There isn't much time before RF 3.1 so RF 3.2 that's initially planned for H1/2019 could be a more realistic goal. Anyway, further discussion about the design and the timeline belongs to issue #1393 rather than into this PR.
@rlucca @pekkaklarck - any updates when this feature will be available?
We'd need someone to design how the output should look like in different cases first. I won't have time for that in the foreseeable future and obviously cannot make promises if/when someone else is interested on it.
It would be great to have this in and later design can be improved ...
@pekkaklarck Some ideas about the design.
Given this folder structure ...
├── tests
│ ├── results
│ ├── robot
│ │ ├── FHIR_PATIENT_TESTS
│ │ │ ├── A1.a_create_new_patient.robot
│ │ │ ├── A1.b_create_new_patient_with_given_id.robot
│ │ │ ├── A1.c_update_patient_with_given_id.robot
When execution is started from within tests folder
cd tests
robot -d results /robot
Then the actual output is
==============================================================================
Robot
==============================================================================
Robot.FHIR PATIENT TESTS :: MPI FHIR PATIENT TEST SUITE
==============================================================================
Robot.FHIR PATIENT TESTS.A1.a create new patient :: MPI FHIR Create Patient...
==============================================================================
Scenario: Create new patient with FHIR data set 1 (minimal) | PASS |
------------------------------------------------------------------------------
Scenario: Create new patient with FHIR data set 2 (maximal) | PASS |
------------------------------------------------------------------------------
Scenario: Create new patient with FHIR data set 3 (partial date) | PASS |
------------------------------------------------------------------------------
Robot.FHIR PATIENT TESTS.A1.a create new patient :: MPI FHIR Creat... | PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
But the "want to have" output is
==============================================================================
Robot
==============================================================================
FHIR PATIENT TESTS :: MPI FHIR PATIENT TEST SUITE
==============================================================================
A1.a create new patient :: MPI FHIR Create Patient Test Cases
==============================================================================
Scenario: Create new patient with FHIR data set 1 (minimal) | PASS |
------------------------------------------------------------------------------
Scenario: Create new patient with FHIR data set 2 (maximal) | PASS |
------------------------------------------------------------------------------
Scenario: Create new patient with FHIR data set 3 (partial date) | PASS |
------------------------------------------------------------------------------
Result | PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
Above I just removed some duplication which make output a bit hard to read when you execute a folder which contains multiple test suites like in given example.
And the "want to have" output with --console bdd or --console verbose is
==============================================================================
Robot
==============================================================================
FHIR PATIENT TESTS :: MPI FHIR PATIENT TEST SUITE
==============================================================================
A1.a create new patient :: MPI FHIR Create Patient Test Cases
==============================================================================
Scenario: Create new patient with FHIR data set 1 (minimal)
Given the server is set up to contain no patient records | PASS |
When a new patient is created with FHIR data set 1 | PASS |
Then patient creation via FHIR should be successful | PASS |
------------------------------------------------------------------------------
Scenario: Create new patient with FHIR data set 2 (maximal)
Given the server is set up to contain no patient records | PASS |
When a new patient is created with FHIR data set 1 | FAIL |
'False' should be true.
Then patient creation via FHIR should be successful | PASS |
Result | FAIL |
------------------------------------------------------------------------------
Scenario: Create new patient with FHIR data set 3 (partial date)
Given the server is set up to contain no patient records | PASS |
When a new patient is created with FHIR data set 3 | PASS |
Then patient creation via FHIR should be successful | PASS |
------------------------------------------------------------------------------
Overall Result | FAIL |
3 critical tests, 2 passed, 1 failed
3 tests total, 2 passed, 1 failed
Not sure, if someone is working on this, but I hacked this patch to get few more things:
- List of failed tests at the end of execution.
- It's hard when you have a big project and lot of tests all over in different modules. It would be helpful if all failed tests are shown at the last.
Failed tests:
- Suite: Dryrun
-- Passing keywords
-- Keywords with embedded arguments :
-- Library keyword with embedded arguments :
- Rather than showing all suite level, one full path would be nice and make o/p more readable.
======================================================================================================================
Cli.Console.Colors And Width :: On Windows colors are never actually written to output. Testing colors thus really ...
======================================================================================================================
Console Colors Auto
Run Tests With Colors --consolecolors auto | FAIL |
- width of verbose o/p is hard-coded and would be nice to allow expanded width, especially with formatting it will miss to show keyword.
@rlucca I want to show trace if step fails, can you please provide some pointer on how to achieve it?
This would help a lot.
We have some steps taking longer time than others, interacting with hardware. I would therefore want monitor the progress in realtime when developing testcases, even though I would disable this feature in the CI environment.
A feature like this can reduce the development cycle in our case with some minutes per run, which would add up quite quickly to hours.
Haven't tried this patch manually yet, but was about to write something similar myself if it wouldn't been available already.
You don't need a modification to the core to do this. You can use a listener to output any information you want.
On Fri, Sep 17, 2021 at 2:47 AM Max @.***> wrote:
This would help a lot.
We have some steps taking longer time than others, interacting with hardware. I would therefore want monitor the progress in realtime when developing testcases, even though I would disable this feature in the CI environment.
A feature like this can reduce the development cycle in our case with some minutes per run, which would add up quite quickly to hours.
Haven't tried this patch manually yet, but was about to write something similar myself if it wouldn't been available already.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/robotframework/robotframework/pull/2525#issuecomment-921582863, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAISMYUBY7OF5TZ7LBH3KULUCLW7VANCNFSM4C5D2OAA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You don't need a modification to the core to do this. You can use a listener to output any information you want.
@boakley How?