Add Playwright E2E tests for SSH Command integration#10
Merged
Conversation
Co-authored-by: gensyn <36128035+gensyn@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Add comprehensive Playwright end-to-end tests for SSH Command
Add Playwright E2E tests for SSH Command integration
Mar 19, 2026
Owner
|
@copilot Make sure the tests cover the following:
|
…put param, idempotent fixture Co-authored-by: gensyn <36128035+gensyn@users.noreply.github.com> Agent-Logs-Url: https://github.com/gensyn/ssh_command/sessions/432a420e-6206-475b-abd5-42d143e9aed1
Contributor
Author
Done in commit 51ac5e6. Here's what was added/changed:
|
…t-e2e-tests-ssh-command
Owner
|
@copilot The workspace now contains |
…2E tests Co-authored-by: gensyn <36128035+gensyn@users.noreply.github.com> Agent-Logs-Url: https://github.com/gensyn/ssh_command/sessions/5f3c11ad-d1e2-4195-b71d-0783d7cb69c5
Contributor
Author
Done in commit 4891ffd. The tests are now fully isolated in Docker with no host-side venv or browser required. What was added:
Running locally: # One-time build
docker compose build
# Run the full suite (starts HA + SSH servers automatically)
docker compose run --rm playwright-tests
# Clean up
docker compose down -v
|
Owner
|
@copilot The tests fail: |
…e_response wrapper Co-authored-by: gensyn <36128035+gensyn@users.noreply.github.com> Agent-Logs-Url: https://github.com/gensyn/ssh_command/sessions/d8a1d959-63ef-40a3-b624-25ed32ae872d
Contributor
Author
... Fixed in commit e479dbe. The tests now run and 35/55 pass. Here's a summary of root causes found and fixed: Fixed:
Remaining (20 tests still failing): Tests expecting HTTP 400 for validation errors (bad credentials, timeout, missing command, unreachable host) are getting HTTP 500 instead — the coordinator raises unhandled exceptions for these cases with asyncssh 2.21.0 that HA converts to 500 rather than the expected 400. This requires investigating the coordinator's exception handling with the installed asyncssh version. Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
Owner
|
@copilot The failing tests expect a status code of 400, but isn't 500 the default status code for Home Assistant when a server error occurs? |
…s code assertions Co-authored-by: gensyn <36128035+gensyn@users.noreply.github.com> Agent-Logs-Url: https://github.com/gensyn/ssh_command/sessions/56c56063-0bd7-4b04-9817-e044a91c8a22
Contributor
Author
You're right that All test assertions have been updated from Additionally, several other root causes were fixed in the same commit:
Result: 53/55 tests now pass. The 2 remaining failures are browser UI tests ( |
Copilot stopped work on behalf of
gensyn due to an error
March 25, 2026 12:49
Owner
|
@copilot This is the last failing Playwright test: |
Contributor
Author
... Fixed in commit 5b486d6. The root cause was that HA's Two changes:
|
Owner
|
@copilot Please extract the logic running playwright tests from |
…script Co-authored-by: gensyn <36128035+gensyn@users.noreply.github.com> Agent-Logs-Url: https://github.com/gensyn/ssh_command/sessions/e911b743-eec9-47c5-a25c-603956827904
Contributor
Author
Done in commit e365594. The Playwright logic is now in a standalone |
Owner
|
@copilot All tests are reasonably fast except for |
…ot just run() Co-authored-by: gensyn <36128035+gensyn@users.noreply.github.com> Agent-Logs-Url: https://github.com/gensyn/ssh_command/sessions/d1af4339-e502-42b8-bc64-4b7956ad8ef0
Contributor
Author
The timeout was only applied to Fixed in commit af1075d by adding |
Owner
|
@copilot The tests seem to cover neither authentication with key files, nor checking known hosts. Is it possible to add those? |
… infrastructure Co-authored-by: gensyn <36128035+gensyn@users.noreply.github.com> Agent-Logs-Url: https://github.com/gensyn/ssh_command/sessions/ec6cbad5-2c7a-40d9-96ef-cb2516aef395
Contributor
Author
Done in commit 5ace910. Three new tests were added to
Infrastructure changes to support these tests:
|
Owner
|
@copilot Change the playwright test worklflow to only run manually. |
Co-authored-by: gensyn <36128035+gensyn@users.noreply.github.com> Agent-Logs-Url: https://github.com/gensyn/ssh_command/sessions/a2be1b57-9972-461c-81ba-92ae95ae0ff7
Contributor
Author
This was referenced Mar 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
tests/playwright/suite covering the full SSH Command integration lifecycle against real SSH test servers, fully encapsulated in Docker for isolated, dependency-free execution — with a GitHub Actions workflow for manual CI runs.Structure
Running with Docker
No local Python environment or browser installation required:
Or use the dedicated script:
On the first run,
entrypoint.shautomatically completes the full HA onboarding flow (user creation, core config, analytics, and integration steps) via the API — no manual UI interaction needed.run_workflows_locally.shincludes the Playwright E2E tests by delegating torun_playwright_tests.sh.GitHub Actions
.github/workflows/playwright-tests.ymlis triggered manually only (workflow_dispatch): builds the images, executes the suite viadocker compose run, and uploadsplaywright-results/junit.xmlas a workflow artifact.Key fixture design (
conftest.py)ha_token— session-scoped; fetches a real HA access token via the login flow with retry logic;wait_for_hapolls/api/onboarding(no auth required, avoids HA's IP-ban mechanism) and waits until all onboarding steps are markeddonebefore proceedingcontext— authenticated Playwright browser context;storage_statepre-populateslocalStoragewithhassTokensbefore any navigation, preventing race conditions with HA's auth-check codeensure_integration— function-scoped; sets up the SSH Command config entry before each test and fully restores the pre-test state afterwards — removing entries added during the test and re-adding entries the test removed, so every test run starts from the same baselineha_api—requests.Sessionwith a long-lived token for direct REST API calls (faster than browser for non-UI assertions)ssh_server_1/ssh_server_2— two independent containers (ssh_docker_test_1,ssh_docker_test_2), each running sshd on the default port 22; driven bySSH_HOST_1,SSH_HOST_2,SSH_USER,SSH_PASSWORDenv varsInfrastructure notes
manifest.jsonpinsasyncssh==2.21.0to match the version pre-installed in the official HA Docker image (no PyPI access at runtime in the test environment)tests/playwright/pytest.inianchors pytest's rootdir totests/playwright/so pytest never traverses into the HA component package (/app/__init__.py), which requires HA-specific packages not present in the test runner image{"service_response": {...}}envelope that HA adds around all service call resultsentrypoint.shcompletes the HAintegrationonboarding step without an auth header and with the requiredclient_id+redirect_urifields; posting{}fails schema validation (422) and leaves the step incomplete, causing the HA SPA to redirect all browser sessions to/onboarding.htmlha-init-wrapper.sh, which pre-populates/etc/hostsusingnslookupbefore starting HA;dns_search: "."indocker-compose.yamlclears the Azure search domain that caused musl's resolver to failcoordinator.pypassesconnect_timeouttoasyncssh.connect()so unreachable-host tests complete in the user-specified timeout rather than the OS TCP default (60–130 s)Dockerfile.sshgenerates an ed25519 user auth key pair and populates the test user'sauthorized_keysat image build time;ssh-init-entrypoint.shwrites the private key and the server's host public key (as aknown_hostsline) to a named Docker volume (ssh_test_init) before starting sshd; the volume is mounted read-only into the HA container at/ssh-test-keys, making the credentials available to tests by pathCoverage highlights
TestIntegrationLifecycle.test_full_lifecycle) — single sequential test that:type == "create_entry"with exactly one entrytype == "abort"withreason == "single_instance_allowed", confirming the entry count remains at 1host,username,password,command,input(stdin),check_known_hosts,timeout, and both SSH serversssh_command.executeresponse shape:output,error,exit_statusinputparameter tested viacatcommand reading from stdintest_key_file_authentication) — connects usingkey_file(ed25519 private key) with no password; verifies the integration authenticates successfully via public-key authtest_check_known_hosts_true_valid) — connects withcheck_known_hosts=Trueand aknown_hostsfile containing the server's host key; verifies host verification passes and the command succeedstest_check_known_hosts_true_unknown_server) — connects withcheck_known_hosts=Truebut no validknown_hostssupplied; verifies the connection is rejected when the server's host key is not knownOriginal prompt
Goal
Add comprehensive Playwright end-to-end tests for the SSH Command Home Assistant custom component.
Background
SSH Command is a Home Assistant custom component that allows executing SSH commands on remote servers. This integration needs thorough E2E testing using Playwright to ensure all functionality works correctly with SSH test servers.
Test Environment
Tests should connect to SSH test servers provided by the docker-compose setup:
ssh_docker_test:2222(user: foo, password: pass)ssh_docker_test:2223(user: foo, password: pass)Requirements
Create Playwright tests in the
tests/playwrightdirectory that cover:Integration Setup Tests (
test_integration_setup.py):Command Execution Tests (
test_command_execution.py):echo,pwd)Service Tests (
test_services.py):ssh_command.sendserviceFrontend Tests (
test_frontend.py):Configuration Tests (
test_configuration.py):Security Tests (
test_security.py):Technical Details
conftest.pywith fixtures for:HOMEASSISTANT_URL(default: http://homeassistant:8123)SSH_HOST(default: ssh_docker_test)SSH_PORT_1(default: 2222)SSH_PORT_2(default: 2223)SSH_USER(default: foo)SSH_PASSWORD(default: pass)File Structure
Additional Files
Create:
tests/playwright/README.md- Documentation on how to run the teststests/playwright/requirements.txt- Python dependencies for testsNotes
The following is the prior conversation context from the user's chat exploration (may be truncated):
User: I have mutliple repositories containing Home Assistant custom components (@gensyn/task_tracker, @gensyn/ssh_command, @gensyn/ssh_docker) that I want to thorougly test using Playwright. For that I plan to spin up a Home Assistant instance in a Docker container solely for testing purposes, but I need a target for testing SSH servers and Docker hosts as well. For that I want to create a Docker container, that uses Docker-in-Docker to spin up two instances of @billchurch/ssh_test. This way, the outer Docker container can be the target of SSH Docker, and the contained SSH servers can be used for testing SSH Command. Task Tracker does not have outside dependencies.
Create a
Dockerfilefile and adocker-compose.yamlfile that serve my described purpose and provide them here.After that, create Playwright tests for each of my mentioned repositories. Put them in their respective repository in a new branch under "tests/playwright" and create a pull request for each repository.
Assistant: I'll help you set up the testing infrastructure and create Playwright tests for your Home Assistant custom components. Let me start by gathering information about your repositories and then provide the Docker configuration files and create th...
This pull request was created from Copilot chat.
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.