Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testrunner raises exception when no tests are run #48263

Closed
gsnedders opened this issue Sep 19, 2024 · 3 comments · Fixed by #48265
Closed

testrunner raises exception when no tests are run #48263

gsnedders opened this issue Sep 19, 2024 · 3 comments · Fixed by #48265
Assignees
@jonathan-j-lee
Labels
infra wptrunner The automated test runner, commonly called through ./wpt run

Comments

@gsnedders
Copy link
Member

For example:

gsnedders@gsnedders-milk web-platform-tests % ./wpt run --log-mach=- --no-pause --no-fail-on-unexpected --this-chunk=1 --total-chunks=10 --chunk-type=hash --test-types=print-reftest safari
/Volumes/gsnedders/projects/wpt/web-platform-tests/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 3.3.6'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
 0:00.04 INFO Set tests_root to /Volumes/gsnedders/projects/wpt/web-platform-tests
 0:00.04 INFO Set metadata_root to /Volumes/gsnedders/projects/wpt/web-platform-tests
 0:00.04 INFO Set manifest_update to True
 0:00.04 INFO Set manifest_download to True
 0:00.06 INFO Interpreting channel 'nightly' as 'preview'
 0:26.30 wptserve INFO Starting http server on http://web-platform.test:8002
 0:26.30 wptserve INFO Starting http server on http://web-platform.test:8000
 0:26.30 wptserve INFO Starting http server on http://web-platform.test:8001
 0:26.30 wptserve INFO Starting http server on http://web-platform.test:8003
 0:26.30 wptserve INFO Starting https server on https://web-platform.test:8443
 0:26.32 wptserve INFO Starting https server on https://web-platform.test:8445
 0:26.35 wptserve INFO Starting https server on https://web-platform.test:8446
 0:26.38 wptserve INFO Starting https server on https://web-platform.test:8444
 0:26.39 wptserve INFO Create socket on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
 0:26.39 wptserve INFO Bind on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
 0:26.39 wptserve INFO Listen on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
 0:26.39 wptserve INFO Create socket on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
 0:26.39 wptserve INFO Bind on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
 0:26.39 wptserve INFO Listen on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
 0:26.41 wptserve INFO Starting http2 server on https://web-platform.test:9000
 0:26.96 SUITE_START: web-platform-test - running 41 tests
 0:26.96 WARNING Unsupported test type print-reftest for product safari
 0:26.96 INFO Using 1 child processes
 0:26.96 INFO No more tests
 0:26.96 SUITE_END

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 0 checks ()
Expected results: 0
Unexpected results: 0
OK
 0:26.96 INFO Got 0 unexpected results, with 0 unexpected passes
 0:26.96 INFO STDERR: Exception in thread 
 0:26.96 INFO STDERR: TestRunnerManager-0
 0:26.96 INFO STDERR: :
 0:26.96 INFO STDERR: Traceback (most recent call last):
 0:26.96 INFO STDERR:   File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner
 0:26.96 INFO STDERR:     
 0:26.96 INFO STDERR: self.run()
 0:26.96 INFO STDERR:   File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run
 0:26.96 INFO STDERR:     
 0:26.96 INFO STDERR: self._target(*self._args, **self._kwargs)
 0:26.96 INFO STDERR:   File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/wptrunner/wptrunner/testrunner.py", line 472, in run_loop
 0:26.96 INFO STDERR:     
 0:26.96 INFO STDERR: self._cleanup_run_loop()
 0:26.96 INFO STDERR:   File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/wptrunner/wptrunner/testrunner.py", line 494, in _cleanup_run_loop
 0:26.96 INFO STDERR:     
 0:26.96 INFO STDERR: self.stop_runner(force=force_stop)
 0:26.96 INFO STDERR:   File "/Volumes/gsnedders/projects/wpt/web-platform-tests/tools/wptrunner/wptrunner/testrunner.py", line 960, in stop_runner
 0:26.97 INFO STDERR:     
 0:26.97 INFO STDERR: self.browser.stop(force=True)
 0:26.97 INFO STDERR: AttributeError
 0:26.97 INFO STDERR: : 
 0:26.97 INFO STDERR: 'NoneType' object has no attribute 'stop'
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8001
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8000
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8002
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8003
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8443
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8444
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8445
 0:27.45 wptserve INFO Stopped http server on web-platform.test:8446
 0:27.46 wptserve INFO Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
 0:27.46 wptserve INFO Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
 0:27.46 wptserve INFO Stopped http server on web-platform.test:9000
 0:27.61 INFO Closing logging queue
 0:27.61 INFO queue closed
 0:27.61 CRITICAL No tests ran

This is almost certainly specific to running an unsupported test type; if you pass a path with no tests, then we simply get a critical error like "Unable to find any tests at the path(s)".
@gsnedders gsnedders added infra wptrunner The automated test runner, commonly called through ./wpt run labels Sep 19, 2024
@gsnedders
Copy link
Member Author

You can also hit this with overly many chunks: --this-chunk=1 --total-chunks=100000000 --chunk-type=hash --test-type=testharness hits this too.
@jonathan-j-lee
Copy link
Contributor

That's my bad: #48030 removes the early out in TestRunnerManager.stop_runner() if test_runner_proc is None. I'll forward-fix.

(Unrelated aside: #48238 adds print-reftest support for any browsers that speak WebDriver.)
@jonathan-j-lee jonathan-j-lee self-assigned this Sep 19, 2024
jonathan-j-lee added a commit to jonathan-j-lee/wpt that referenced this issue Sep 19, 2024
@jonathan-j-lee
[wptrunner] Guard against no browser during TestRunnerManager cleanup
f37160c 
Even with no browser, `stop_runner()` should still call `cleanup()` in
case there are other resources to clean up.

Fixes web-platform-tests#48263
@jonathan-j-lee jonathan-j-lee mentioned this issue Sep 19, 2024
Merged
@gsnedders
Copy link
Member Author

(Unrelated aside: #48238 adds print-reftest support for any browsers that speak WebDriver.)

(It doesn't entirely; e.g.

wpt/tools/wptrunner/wptrunner/browsers/safari.py

Lines 22 to 25 in 309f303

"executor": {"testharness": "WebDriverTestharnessExecutor",
"reftest": "WebDriverRefTestExecutor",
"wdspec": "WdspecExecutor",
"crashtest": "WebDriverCrashtestExecutor"},
is still required for each browser to specify what executor to use for each test type. And safaridriver doesn't support the underlying WebDriver command.)
jonathan-j-lee added a commit that referenced this issue Sep 19, 2024
@jonathan-j-lee
[wptrunner] Guard against no browser during TestRunnerManager clean…
b23d75f 
…up (#48265)

Even with no browser, `stop_runner()` should still call `cleanup()` in
case there are other resources to clean up.

Fixes #48263
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infra wptrunner The automated test runner, commonly called through ./wpt run
2 participants
@gsnedders @jonathan-j-lee