Add FE tests w/ Robot Framework

[msabramo]

After submitting GH-37 as a potential fix for GH-35, I wondered if there was a way to add FE tests that would detect that sort of issue. I found that Robot Framework is a popular tool for this, and it has a SeleniumLibrary that can be used to interact with web pages. This commit adds a simple test case that opens a browser, inputs text into the st_keyup input field, and waits for the page to contain some expected text. The test case is run by a shell script that uses the robot command to run the test case. The test case is run by a shell script that uses the robot command to run the test case.

This test fails with the code on main when using Streamlit 1.32.2, because of GH-35:

(.venv)
abramowi at Marcs-MBP-3 in ~/Code/OpenSource/streamlit-keyup (fe-tests-robot-framework)
$ pip freeze | grep '^streamlit'
streamlit==1.32.2

(.venv)
abramowi at Marcs-MBP-3 in ~/Code/OpenSource/streamlit-keyup (fe-tests-robot-framework)
$ ./run-robot-tests
==============================================================================
Robot
==============================================================================
Robot.Test Cases
==============================================================================
Interact with st_keyup                                                | FAIL |
Element 'css:iframe[title="st_keyup.st_keyup"]' not visible after 5 seconds.
------------------------------------------------------------------------------
Robot.Test Cases                                                      | FAIL |
1 test, 0 passed, 1 failed
==============================================================================
Robot                                                                 | FAIL |
1 test, 0 passed, 1 failed
==============================================================================
Output:  /Users/abramowi/Code/OpenSource/streamlit-keyup/tests/robot/output/output.xml
Log:     /Users/abramowi/Code/OpenSource/streamlit-keyup/tests/robot/output/log.html
Report:  /Users/abramowi/Code/OpenSource/streamlit-keyup/tests/robot/output/report.html

But it passes with the code in GH-37:

(.venv)
abramowi at Marcs-MBP-3 in ~/Code/OpenSource/streamlit-keyup (gh-35-not-rendering-with-streamlit-1.32.2●)
$ git branch
  fe-tests-robot-framework
* gh-35-not-rendering-with-streamlit-1.32.2
  main

(.venv)
abramowi at Marcs-MBP-3 in ~/Code/OpenSource/streamlit-keyup (gh-35-not-rendering-with-streamlit-1.32.2●)
$ git checkout -b gh-35-not-rendering-with-streamlit-1.32.2-with-fe-tests-robot-framework
Switched to a new branch 'gh-35-not-rendering-with-streamlit-1.32.2-with-fe-tests-robot-framework'

(.venv)
abramowi at Marcs-MBP-3 in ~/Code/OpenSource/streamlit-keyup (gh-35-not-rendering-with-streamlit-1.32.2-with-fe-tests-robot-framework●)
$ git merge fe-tests-robot-framework
Merge made by the 'ort' strategy.
 .gitignore                               |  5 ++++-
 run-robot-tests                          |  3 +++
 tests/robot/resources/app_keywords.robot | 16 ++++++++++++++++
 tests/robot/test_cases.robot             | 16 ++++++++++++++++
 4 files changed, 39 insertions(+), 1 deletion(-)
 create mode 100755 run-robot-tests
 create mode 100644 tests/robot/resources/app_keywords.robot
 create mode 100644 tests/robot/test_cases.robot

(.venv)
abramowi at Marcs-MBP-3 in ~/Code/OpenSource/streamlit-keyup (gh-35-not-rendering-with-streamlit-1.32.2-with-fe-tests-robot-framework)
$ ./run-robot-tests
==============================================================================
Robot
==============================================================================
Robot.Test Cases
==============================================================================
Interact with st_keyup                                                | PASS |
------------------------------------------------------------------------------
Robot.Test Cases                                                      | PASS |
1 test, 1 passed, 0 failed
==============================================================================
Robot                                                                 | PASS |
1 test, 1 passed, 0 failed
==============================================================================
Output:  /Users/abramowi/Code/OpenSource/streamlit-keyup/tests/robot/output/output.xml
Log:     /Users/abramowi/Code/OpenSource/streamlit-keyup/tests/robot/output/log.html
Report:  /Users/abramowi/Code/OpenSource/streamlit-keyup/tests/robot/output/report.html

Cc: @blackary, @nathancooperjones

[blackary]

I love this! I've wanted to play with Robot Framework, but hadn't gotten a chance yet. A couple things I would like to see before this gets merged, which I should be able to do eventually but you certainly can do if you would like to:

1. Add requirements-test.txt that includes robotframework and other other test dependencies
2. Add a github action to run this automatically on all PRs

[msabramo]

1. Add requirements-test.txt that includes robotframework and other other test dependencies

See e1b6d3f.

[msabramo]

2. Add a github action to run this automatically on all PRs

Yeah I agree that having CI to automate this will be great. Personally, I don't have a lot of experience with GitHub Actions. I recently set up a work project to run Robot Framework in CircleCi; it's probably fairly similar.

The bigger issue I see that I think deserves some thought is that running it on every commit to your repo is not quite enough. In fact, in the case of #35, AFAICT that was not caused by any changes in your repo; rather it seems to have been caused by a new version of Streamlit being released that violated implicit assumptions in streamlit-keyup. So while running Robot Framework tests on every commit is a good thing, I think ideally it would also be run automatically when new dependencies are released, or perhaps just on a schedule like nightly or weekly, which is probably easier.

Hmmm. I do wonder if this could be solved by having a set up with one of these bots like dependbot or such that notices new versions of dependencies and then creates PRs to upgrade to them.

It also seems like a good thing to ensure that streamlit-keyup works with a wide range of Streamlit versions; not just the latest. So maybe that implies doing some kind of matrix build thing where you run the tests against multiple Streamlit versions and multiple Python versions.

Not sure what the best way is yet, but I might be able to find a little time to send some PRs to experiment.