avocado icon indicating copy to clipboard operation
avocado copied to clipboard

Published 20 hours ago • verified publisher icon avocado-framework

nrunner doesn't seem to test anything when run over ssh

Open pmmccorm opened this issue 3 years ago • 10 comments

It seems to just "complete" but no tests seem to be ran:

$ ssh -t remote_node '.local/bin/avocado run --test-runner nrunner /bin/true' ; echo $?
JOB ID     : c60a4f33f0aff584e74dfd48a33c25ff9399a685
JOB LOG    : avocado/job-results/job-2021-08-13T18.52-c60a4f3/job.log
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 0.32 s
Connection to remote_node closed.
0

The job log (paths elided):

2021-08-13 18:52:56,839 job              L0287 INFO | Command line: .local/bin/avocado run --test-runner nrunner /bin/true
2021-08-13 18:52:56,840 job              L0288 INFO |
2021-08-13 18:52:56,840 job              L0281 INFO | Avocado version: 90.0
...
2021-08-13 18:52:56,844 job              L0304 INFO | Job ID: c60a4f33f0aff584e74dfd48a33c25ff9399a685
2021-08-13 18:52:56,844 job              L0307 INFO |
2021-08-13 18:52:56,852 selector_events  L0059 DEBUG| Using selector: EpollSelector
2021-08-13 18:52:56,903 job              L0640 INFO | Test results available in /job-2021-08-13T18.52-c60a4f3

Results.tap is empty. Same behavior with multiple tests given. Local runs complete fine.

pmmccorm avatar Aug 13 '21 18:08 pmmccorm

Hi @pmmccorm I can't reproduce this problem. Could you please share more information in how you're installing avocado in the remote node?

I installed a fresh VM and after installing avocado with python3 setup.py develop --user (make develop should work as well), it works here:



$ ssh -t [email protected] '.local/bin/avocado run --test-runner nrunner /bin/true' ; echo $?
JOB ID     : e5508397d15782b48fa1dbacd8303125b7f514a3
JOB LOG    : /home/ana/avocado/job-results/job-2021-08-16T12.58-e550839/job.log
 (1/1) /bin/true: STARTED
 (1/1) /bin/true: PASS (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/ana/avocado/job-results/job-2021-08-16T12.58-e550839/results.html
JOB TIME   : 0.75 s
Connection to 192.168.122.93 closed.
0

ana avatar Aug 16 '21 11:08 ana

Hello, thanks for responding.

I installed like:

$ pip3 install avocado-framework
Collecting avocado-framework
  Downloading https://files.pythonhosted.org/packages/f1/ec/938c7feacd8f2319f21a0b1f95c11bb31a8a5562ffddf75bbda691a76d90/avocado_framework-90.0-py3-none-any.whl (417kB)
    100% |████████████████████████████████| 419kB 4.5MB/s
Collecting setuptools (from avocado-framework)
  Downloading https://files.pythonhosted.org/packages/bd/25/5bdf7f1adeebd4e3fa76b2e2f045ae53ee208e40a4231ad0f0c3007e4353/setuptools-57.4.0-py3-none-any.whl (819kB)
    100% |████████████████████████████████| 819kB 2.3MB/s
Installing collected packages: setuptools, avocado-framework
Successfully installed avocado-framework-90.0 setuptools-57.4.0
$ pip3 freeze
apparmor==2.12
asn1crypto==0.24.0
avocado-framework==90.0
certifi==2018.1.18
chardet==3.0.4
cryptography==2.1.4
httplib2==0.9.2
idna==2.6
iotop==0.6
keyring==10.6.0
keyrings.alt==3.0
LibAppArmor==2.12
netifaces==0.10.4
ovs==2.14.0
pexpect==4.2.1
pycrypto==2.6.1
pygobject==3.26.1
python-apt==1.6.5+ubuntu0.6
python-debian==0.1.32
pyxdg==0.25
requests==2.18.4
requests-unixsocket==0.1.5
SecretStorage==2.3.1
six==1.11.0
sortedcontainers==1.5.7
sos==4.1
ssh-import-id==5.7
systemd-python==234
urllib3==1.22

Then from other host:


$ ssh -t target_host  '~/.local/bin/avocado run  --test-runner nrunner /bin/true /bin/true'
JOB ID     : 5e777ead1dec2ee9df80327c82edba836acaae5d
JOB LOG    : avocado/job-results/job-2021-08-18T00.31-5e777ea/job.log
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 0.35 s
Connection to target_host closed.

pmmccorm avatar Aug 18 '21 00:08 pmmccorm

Hi @pmmccorm , I couldn't reproduce it too. Could you please paste the --show=all output?

ssh -t target_host 'avocado --show=all run --test-runner='nrunner' /bin/true'

beraldoleal avatar Aug 18 '21 11:08 beraldoleal

Thanks @pmmccorm

I couldn't reproduce it on fedora34, but I have reproduced it on Debian 11 (bullesye).

$ ssh -t [email protected] '~/.local/bin/avocado --show=all  run  --test-runner nrunner /bin/true /bin/true'
[email protected]'s password: 
JOB ID     : d4a31adc583239b10cca10944ef69546a1054e06
JOB LOG    : /home/ana/avocado/job-results/job-2021-08-18T09.38-d4a31ad/job.log
avocado.sysinfo: Commands configured by file: /home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/commands
avocado.sysinfo: Commands configured by file: /home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/commands
avocado.sysinfo: Files configured by file: /home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/files
avocado.sysinfo: Files configured by file: /home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/files
avocado.sysinfo: File /home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/fail_commands does not exist.
avocado.sysinfo: File /home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/fail_commands does not exist.
avocado.sysinfo: File /home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/fail_files does not exist.
avocado.sysinfo: File /home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/fail_files does not exist.
avocado.sysinfo: Profilers configured by file: /home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/profilers
avocado.sysinfo: Profilers configured by file: /home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/profilers
avocado.sysinfo: Profiler disabled
avocado.sysinfo: Profiler disabled
avocado.sysinfo: Not logging /proc/slabinfo (lack of permissions)
avocado.sysinfo: Not logging /proc/slabinfo (lack of permissions)
avocado.sysinfo: Not logging /proc/pci (file does not exist)
avocado.sysinfo: Not logging /proc/pci (file does not exist)
avocado.sysinfo: Not logging /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor (file does not exist)
avocado.sysinfo: Not logging /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor (file does not exist)
avocado.sysinfo: Not logging /sys/kernel/debug/sched_features (file does not exist)
avocado.sysinfo: Not logging /sys/kernel/debug/sched_features (file does not exist)
avocado.test: Command line: /home/ana/.local/bin/avocado --show=all run --test-runner nrunner /bin/true /bin/true
avocado.test: 
avocado.test: Avocado version: 90.0
avocado.test: 
avocado.test: Avocado config:
avocado.test: 
avocado.test: {'assets.fetch.ignore_errors': False,
avocado.test:  'assets.fetch.references': [],
avocado.test:  'assets.list.days': None,
avocado.test:  'assets.list.overall_limit': None,
avocado.test:  'assets.list.size_filter': None,
avocado.test:  'assets.purge.days': None,
avocado.test:  'assets.purge.overall_limit': None,
avocado.test:  'assets.purge.size_filter': None,
avocado.test:  'assets.register.name': None,
avocado.test:  'assets.register.sha1_hash': None,
avocado.test:  'assets.register.url': None,
avocado.test:  'config': None,
avocado.test:  'config.datadir': False,
avocado.test:  'core.input_encoding': 'utf-8',
avocado.test:  'core.paginator': False,
avocado.test:  'core.show': {'early', 'test', 'debug', 'all', 'app'},
avocado.test:  'core.verbose': False,
avocado.test:  'datadir.paths.base_dir': '/home/ana/avocado',
avocado.test:  'datadir.paths.cache_dirs': ['/home/ana/avocado/data/cache'],
avocado.test:  'datadir.paths.data_dir': '/home/ana/avocado/data',
avocado.test:  'datadir.paths.logs_dir': '/home/ana/avocado/job-results',
avocado.test:  'datadir.paths.test_dir': '/usr/share/doc/avocado/tests',
avocado.test:  'diff.create_reports': False,
avocado.test:  'diff.filter': ['cmdline', 'time', 'variants', 'results', 'config', 'sysinfo'],
avocado.test:  'diff.html': None,
avocado.test:  'diff.jobids': [],
avocado.test:  'diff.open_browser': False,
avocado.test:  'diff.strip_id': False,
avocado.test:  'distro.distro_def_arch': '',
avocado.test:  'distro.distro_def_create': False,
avocado.test:  'distro.distro_def_name': '',
avocado.test:  'distro.distro_def_path': '',
avocado.test:  'distro.distro_def_release': '',
avocado.test:  'distro.distro_def_type': '',
avocado.test:  'distro.distro_def_version': '',
avocado.test:  'filter.by_tags.include_empty': False,
avocado.test:  'filter.by_tags.include_empty_key': False,
avocado.test:  'filter.by_tags.tags': [],
avocado.test:  'job.output.loglevel': 'DEBUG',
avocado.test:  'job.output.testlogs.logfiles': ['debug.log'],
avocado.test:  'job.output.testlogs.statuses': [],
avocado.test:  'job.replay.source_job_id': 'latest',
avocado.test:  'job.run.result.json.enabled': True,
avocado.test:  'job.run.result.json.output': None,
avocado.test:  'job.run.result.tap.enabled': True,
avocado.test:  'job.run.result.tap.include_logs': False,
avocado.test:  'job.run.result.tap.output': None,
avocado.test:  'job.run.result.xunit.enabled': True,
avocado.test:  'job.run.result.xunit.job_name': None,
avocado.test:  'job.run.result.xunit.max_test_log_chars': 100000,
avocado.test:  'job.run.result.xunit.output': None,
avocado.test:  'job.run.store_logging_stream': [],
avocado.test:  'job.run.timeout': 0,
avocado.test:  'jobs.get.output_files.destination': None,
avocado.test:  'jobs.get.output_files.job_id': None,
avocado.test:  'jobs.show.job_id': 'latest',
avocado.test:  'json.variants.load': None,
avocado.test:  'list.external_runner': None,
avocado.test:  'list.external_runner_chdir': None,
avocado.test:  'list.external_runner_testdir': None,
avocado.test:  'list.loaders': ['file', '@DEFAULT'],
avocado.test:  'list.recipes.write_to_directory': None,
avocado.test:  'list.references': [],
avocado.test:  'list.resolver': False,
avocado.test:  'list.write_to_json_file': None,
avocado.test:  'nrunner.max_parallel_tasks': 1,
avocado.test:  'nrunner.shuffle': False,
avocado.test:  'nrunner.spawner': 'process',
avocado.test:  'nrunner.status_server_buffer_size': 33554432,
avocado.test:  'nrunner.status_server_listen': '127.0.0.1:8888',
avocado.test:  'nrunner.status_server_uri': '127.0.0.1:8888',
avocado.test:  'plugins.cli.cmd.order': [],
avocado.test:  'plugins.cli.order': [],
avocado.test:  'plugins.disable': [],
avocado.test:  'plugins.init.order': [],
avocado.test:  'plugins.job.prepost.order': [],
avocado.test:  'plugins.jobscripts.post': '/etc/avocado/scripts/job/post.d/',
avocado.test:  'plugins.jobscripts.pre': '/etc/avocado/scripts/job/pre.d/',
avocado.test:  'plugins.jobscripts.warn_non_existing_dir': False,
avocado.test:  'plugins.jobscripts.warn_non_zero_status': True,
avocado.test:  'plugins.resolver.order': [],
avocado.test:  'plugins.result.order': [],
avocado.test:  'plugins.result_events.order': [],
avocado.test:  'plugins.runnable.runner.order': [],
avocado.test:  'plugins.runner.order': [],
avocado.test:  'plugins.skip_broken_plugin_notification': [],
avocado.test:  'plugins.spawner.order': [],
avocado.test:  'plugins.varianter.order': [],
avocado.test:  'run.dict_variants': [],
avocado.test:  'run.dry_run.enabled': False,
avocado.test:  'run.dry_run.no_cleanup': False,
avocado.test:  'run.execution_order': 'variants-per-test',
avocado.test:  'run.external_runner': None,
avocado.test:  'run.external_runner_chdir': None,
avocado.test:  'run.external_runner_testdir': None,
avocado.test:  'run.failfast': False,
avocado.test:  'run.ignore_missing_references': False,
avocado.test:  'run.job_category': None,
avocado.test:  'run.journal.enabled': False,
avocado.test:  'run.keep_tmp': False,
avocado.test:  'run.loaders': ['file', '@DEFAULT'],
avocado.test:  'run.log_test_data_directories': False,
avocado.test:  'run.output_check': True,
avocado.test:  'run.output_check_record': None,
avocado.test:  'run.references': ['/bin/true', '/bin/true'],
avocado.test:  'run.replay.ignore': [],
avocado.test:  'run.replay.job_id': None,
avocado.test:  'run.replay.resume': False,
avocado.test:  'run.replay.test_status': [],
avocado.test:  'run.results.archive': False,
avocado.test:  'run.results_dir': None,
avocado.test:  'run.test_parameters': [],
avocado.test:  'run.test_runner': 'nrunner',
avocado.test:  'run.unique_job_id': None,
avocado.test:  'run.wrapper.wrappers': [],
avocado.test:  'runner.exectest.exitcodes.skip': [],
avocado.test:  'runner.output.color': 'auto',
avocado.test:  'runner.output.colored': True,
avocado.test:  'runner.output.utf8': True,
avocado.test:  'runner.timeout.after_interrupted': 60,
avocado.test:  'runner.timeout.process_alive': 60,
avocado.test:  'runner.timeout.process_died': 10,
avocado.test:  'simpletests.status.failure_fields': ['status', 'stdout', 'stderr'],
avocado.test:  'simpletests.status.skip_location': 'all',
avocado.test:  'simpletests.status.skip_regex': '^SKIP$',
avocado.test:  'simpletests.status.warn_location': 'all',
avocado.test:  'simpletests.status.warn_regex': '^WARN$',
avocado.test:  'spawner.podman.bin': '/usr/bin/podman',
avocado.test:  'spawner.podman.image': 'fedora:31',
avocado.test:  'subcommand': 'run',
avocado.test:  'sysinfo.collect.commands_timeout': -1,
avocado.test:  'sysinfo.collect.enabled': True,
avocado.test:  'sysinfo.collect.installed_packages': False,
avocado.test:  'sysinfo.collect.locale': 'C',
avocado.test:  'sysinfo.collect.optimize': False,
avocado.test:  'sysinfo.collect.per_test': False,
avocado.test:  'sysinfo.collect.profiler': False,
avocado.test:  'sysinfo.collect.sysinfodir': None,
avocado.test:  'sysinfo.collectibles.commands': '/home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/commands',
avocado.test:  'sysinfo.collectibles.fail_commands': '/home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/fail_commands',
avocado.test:  'sysinfo.collectibles.fail_files': '/home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/fail_files',
avocado.test:  'sysinfo.collectibles.files': '/home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/files',
avocado.test:  'sysinfo.collectibles.profilers': '/home/ana/.local/lib/python3.9/site-packages/avocado/etc/avocado/sysinfo/profilers',
avocado.test:  'task.timeout.running': None,
avocado.test:  'variants.contents': False,
avocado.test:  'variants.debug': False,
avocado.test:  'variants.inherit': False,
avocado.test:  'variants.json_variants_dump': None,
avocado.test:  'variants.summary': 0,
avocado.test:  'variants.tree': False,
avocado.test:  'variants.variants': 1,
avocado.test:  'vmimage.get.arch': None,
avocado.test:  'vmimage.get.distro': None,
avocado.test:  'vmimage.get.version': None}
avocado.test: 
avocado.test: Avocado Data Directories:
avocado.test: 
avocado.test: base     /home/ana/avocado
avocado.test: tests    /home/ana/avocado/tests
avocado.test: data     /home/ana/avocado/data
avocado.test: logs     /home/ana/avocado/job-results/job-2021-08-18T09.38-d4a31ad
avocado.test: 
avocado.test: Temporary dir: /var/tmp/avocado_ptd2cvtw/avocado_job_miid4ahx
avocado.test: 
avocado.test: Job ID: d4a31adc583239b10cca10944ef69546a1054e06
avocado.test: 
asyncio: Using selector: EpollSelector
asyncio: Using selector: EpollSelector
avocado.test: Test results available in /home/ana/avocado/job-results/job-2021-08-18T09.38-d4a31ad
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
avocado.sysinfo: Not logging /proc/slabinfo (lack of permissions)
avocado.sysinfo: Not logging /proc/slabinfo (lack of permissions)
avocado.sysinfo: Not logging /proc/pci (file does not exist)
avocado.sysinfo: Not logging /proc/pci (file does not exist)
avocado.sysinfo: Not logging /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor (file does not exist)
avocado.sysinfo: Not logging /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor (file does not exist)
avocado.sysinfo: Not logging /sys/kernel/debug/sched_features (file does not exist)
avocado.sysinfo: Not logging /sys/kernel/debug/sched_features (file does not exist)
JOB TIME   : 0.24 s
Connection to 192.168.122.184 closed.

ana avatar Aug 18 '21 13:08 ana

Comparing the logs under job-results from a local and a remote run, I don't see anything different. Also, the problem is reproducible installing on debian 10 with both pip3 and python setup.py install

ana avatar Aug 18 '21 13:08 ana

@pmmccorm I was able to reproduce it. It seems Bebian systems will not put ~/.local/bin by default on users' PATH. Adding it is one possible workaround, but I'm investigating if this is also a regression.

Anyway, could you please verify if it works with ~/.local/bin added to the PATH env var?

clebergnu avatar Aug 19 '21 15:08 clebergnu

@pmmccorm could you check if #4882 at least clearer for you? Thanks.

clebergnu avatar Aug 19 '21 18:08 clebergnu

Sorry for late reply.

On Ubuntu Bionic. PATH before:

$ ssh -t remote  'echo $PATH'
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

Adding explicitly:

$ ssh -t remote  'PATH=$HOME/.local/bin ~/.local/bin/avocado run  --test-runner nrunner /bin/true /bin/true'

It works!

 (2/2) /bin/true: STARTED
 (1/2) /bin/true: STARTED
 (2/2) /bin/true: PASS (0.01 s)
 (1/2) /bin/true: PASS (0.01 s)

pmmccorm avatar Aug 23 '21 20:08 pmmccorm

I read #4882 : I think that will make what is going on more clear, or at least give some hints and error out properly (which one can then google). Thanks!

That being said it would be nice to not break like this w/ different runners, currently the following don't behave the same:

 ssh -t remote  '~/.local/bin/avocado run  --test-runner runner /bin/true
 ssh -t remote  '~/.local/bin/avocado run  --test-runner nrunner /bin/true

Which is unexpected. Maybe if runner is not in $PATH a couple hard-coded paths can be checked to work around this? Or I can close this issue if you want.

pmmccorm avatar Aug 23 '21 21:08 pmmccorm

I read #4882 : I think that will make what is going on more clear, or at least give some hints and error out properly (which one can then google). Thanks!

That being said it would be nice to not break like this w/ different runners, currently the following don't behave the same:

 ssh -t remote  '~/.local/bin/avocado run  --test-runner runner /bin/true
 ssh -t remote  '~/.local/bin/avocado run  --test-runner nrunner /bin/true

Which is unexpected. Maybe if runner is not in $PATH a couple hard-coded paths can be checked to work around this? Or I can close this issue if you want.

@pmmccorm thanks for the follow-up.

First, I believe the solved issue https://github.com/avocado-framework/avocado/issues/5088 is also related to this, and improves the situation as not requiring a $PATH to be set. Can you please check that?

With regards to the difference in behavior for runner and nrunner, we must take into account that:

  1. runner is being "dismantled" and will go away very soon (it's actually past due)
  2. nrunner is much more decoupled and uses external runners that can be standalone executables, so checking the availability of those external runners is nrunner specific.

Having said that, nrunner can do better here. One possibility is to introduce a UNKNOWN (or UNAVAILABLE, or MISSING) test result status (in addition to PASS, FAIL, ERROR, etc). The reason is that when Avocado starts a nrunner runner, it may terminate (because of an error) without providing a result.

In some cases Avocado will account for those as CANCEL, but that is not the ideal situation as CANCEL has a different meaning.

Let me know if #5088 works for you, and also your general opinion on the other points discussed here. Thanks!

clebergnu avatar Dec 08 '21 20:12 clebergnu

Based on the last comment (and the lack of response), I'm assuming this issue is no longer valid. Also, for remote test execution the remote spawner is a possibility now.

clebergnu avatar Feb 05 '24 12:02 clebergnu