fleet
fleet copied to clipboard
Published
20 hours ago •
fleetdm
fleetdm
Release QA: 4.66.0
Goal: easy-to-follow test steps for checking a release manually
Important reference data
- fleetctl preview setup
- permissions documentation
- premium tests require license key (needs renewal)
fleetctl preview --license-key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCBJbmMuIiwiZXhwIjoxNjQwOTk1MjAwLCJzdWIiOiJkZXZlbG9wbWVudCIsImRldmljZXMiOjEwMCwibm90ZSI6ImZvciBkZXZlbG9wbWVudCBvbmx5IiwidGllciI6ImJhc2ljIiwiaWF0IjoxNjIyNDI2NTg2fQ.WmZ0kG4seW3IrNvULCHUPBSfFdqj38A_eiXdV_DFunMHechjHbkwtfkf1J6JQJoDyqn8raXpgbdhafDwv3rmDw - premium tests require license key (active - Expires Sunday, January 1, 2023 12:00:00 AM)
fleetctl preview --license-key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCBJbmMuIiwiZXhwIjoxNjcyNTMxMjAwLCJzdWIiOiJGbGVldCBEZXZpY2UgTWFuYWdlbWVudCIsImRldmljZXMiOjEwMCwibm90ZSI6ImZvciBkZXZlbG9wbWVudCBvbmx5IiwidGllciI6InByZW1pdW0iLCJpYXQiOjE2NDI1MjIxODF9.EGHQjIzM73YyMbnCruswzg360DEYCsDi9uz48YcDwQHq90BabGT5PIXRiculw79emGj5sk2aKgccTd2hU5J7Jw
Smoke Tests
Smoke tests are limited to core functionality and serve as a pre-release final review. If smoke tests are failing, a release cannot proceed.
Fleet core:
Fleet version (Head to the "My account" page in the Fleet UI or run fleetctl version):
Web browser (e.g. Chrome 88.0.4324):
Prerequisites
fleetctl previewis set up and running the desired test version using--tagparameters.- Unless you are explicitly testing older browser versions, browser is up to date.
- Certificate & flagfile are in place to create new host.
- In your browser, clear local storage using devtools.
Orchestration
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| Update flow |
| All previously created hosts/queries are verified to still exist | pass |
| Login flow |
|
| pass |
| Packs flow | Verify management, operation, and logging of ["2017 packs"](https://fleetdm.com/handbook/company/why-this-way#why-does-fleet-support-query-packs). |
| pass |
| Log destination flow | Verify log destination for software, query, policy, and packs. |
| pass |
| OS settings | Verify OS settings functionality |
| pass |
MDM
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| MDM enrollment flow | Verify MDM enrollments, run MDM commands |
| pass |
| MDM migration flow | Verify MDM migration for ADE and non-ADE hosts |
| pass |
| OS settings | Verify OS settings functionality |
| pass |
| Setup experience | Verify macOS Setup experience |
| pass |
| OS updates | Verify OS updates flow |
| pass |
| iOS/iPadOS | Verify enrollment, profiles, & software installs |
| pass |
| Certificates Upload | APNs cert and ABM token renewal workflow |
| pass |
Software
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass |
| Query flow | Create, edit, run, and delete queries. |
| pass |
| Host Flow | Verify a new host can be added and removed following modal instructions using your own device. |
| pass |
| My device page | Verify the end user's my device page loads successfully. |
| pass |
| Scripts | Verify script library and execution |
| pass |
| Software | Verify software library and install / download |
| pass |
| Migration Test | Verify Fleet can migrate to the next version with no issues. |
Using the github action https://github.com/fleetdm/fleet/actions/workflows/db-upgrade-test.yml
| pass |
All Product Groups
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| Release blockers | Verify there are no outstanding release blocking tickets. |
| pass/fail |
| Load tests - minor releases only unless otherwise specified | Verify all load test metrics are within acceptable range on final build of RC. |
| pass/fail |
Notes
Issues found new to this version:
Issues found that reproduce in last stable version:
What has not been tested:
Include any notes on whether issues should block release or not as needed:
fleetd agent:
Includes updates to:
- Orbit: True / False
- Desktop: True / False
- Chrome extension: True / False
List versions changes for any component updates below:
- Orbit
v1.xx.x>v1.xx.x - Desktop
v1.xx.x>v1.xx.x - Chrome extension
v1.xx.x>v1.xx.x
Testing gates for new fleetd release
Goal: Ensure new fleetd is tested and promoted from local > edge > stable channels
- Build a new
fleetdfrom the release candidate branch as needed for Orbit, Desktop, and Chrome Extension.
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| `fleetd` local testing |
1. Following [Testing TUF]([url](https://github.com/fleetdm/fleet/blob/main/tools/tuf/test/README.md)) instructions create binaries for Mac, Windows, and Ubuntu using your local TUF repository and install on macOS, Linux, and Windows hosts. |
1. Confirm the hosts install with the updated version and are working correctly. 2. Confirm any new features and/or bug fixes associated with this release are working as intended. |
pass/fail | `fleetd` auto-update tests |
1. Conduct the [`fleetd` auto-update n+1 test]([url](https://github.com/fleetdm/fleet/blob/main/tools/tuf/test/Fleetd-auto-update-test-guide.md)) 2. QA certifies new release by commenting in issue. |
1. Agent successfully auto-updates. 2. Issue is certified by QA. |
pass/fail | `fleetd` tests |
1. Set up a host in your instance to receive updates from the `edge` channels. 2. Work with engineer leading the release to push changes to the `edge` channel. |
1. Confirm the hosts running on the edge channel receive the update and are working correctly. 2. Confirm any new features and/or bug fixes associated with this release are working as intended. |
pass/fail |
New fleetd pushed to edge
Goal: Ensure fleetd version pushed to edge is working with the current released version of fleet.
- Fleet server is running the latest released version available on Fleet Releases page.
- Set Agent options to use edge in the Fleet server configuration. For example:
update_channels:
osqueryd: edge
orbit: edge
desktop: edge
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| Query flow | Run queries. | 1. Queries can be run manually | pass/fail |
| Host Flow | Verify a new host can be added using your own device. |
1. Hosts can enroll and report correct version of `fleetd` (orbit, osquery, desktop). 2. Refetching host vitals completes and returns updated information. | pass/fail |
| My device page | Verify the end user's my device page loads successfully. |
1. Clicking the Fleet desktop item, then "My device" successfully loads the my device page. 2. The "My device" page is populated correctly and as expected. 3. Styling and padding appears correct. | pass/fail |
| Scripts | Verify script execution |
1. Verify able to run a script on all host types from CLI. 2. From Host details (macOS, Windows, & Linux) run a script that should PASS, verify. 3. From Host details (macOS, Windows, & Linux) run a script that should FAIL, verify. 4. Verify script results display correctly in Activity feed. | pass/fail |
| Software | Verify software install / download |
1. From Host details (macOS, Windows, & Linux) run an install that should PASS, verify. 2. From My Device (macOS, Windows, & Linux) software tab should have self-service items available, verify. 3. Verify software installs display correctly in Activity feed. | pass/fail |
| OS settings | Verify OS settings functionality |
1. Verify able to configure Disk encryption (macOS, Windows, & Linux). 2. Verify host enrolled with Disk encryption enforced successfully encrypts. | pass/fail |
| Packs flow | Verify management, operation, and logging of ["2017 packs"](https://fleetdm.com/handbook/company/why-this-way#why-does-fleet-support-query-packs). |
1. Packs successfully run on host machines after migrations 2. New Packs can be created. 3. Packs can be edited and deleted 4. Packs results information is logged | pass/fail |
Notes
Issues found new to this version:
Issues found that reproduce in last stable version:
What has not been tested:
Include any notes on whether issues should block release or not as needed:
https://github.com/fleetdm/fleet/actions/runs/14254677273
FLEET_VERSION_A=v4.65.0 FLEET_VERSION_B=rc-minor-fleet-v4.66.0 go test -v
--- PASS: TestUpgradeAToB (57.53s)
PASS
ok github.com/fleetdm/fleet/v4/test/upgrade 57.549s
https://github.com/fleetdm/fleet/actions/runs/14254729734
FLEET_VERSION_A=v4.64.0 FLEET_VERSION_B=rc-minor-fleet-v4.66.0 go test -v
--- PASS: TestUpgradeAToB (51.60s)
PASS
ok github.com/fleetdm/fleet/v4/test/upgrade 51.620s
https://github.com/fleetdm/fleet/actions/runs/14254732625/job/39955185529
FLEET_VERSION_A=v4.63.0 FLEET_VERSION_B=rc-minor-fleet-v4.66.0 go test -v
--- PASS: TestUpgradeAToB (57.35s)
PASS
ok github.com/fleetdm/fleet/v4/test/upgrade 57.366s
Release QA shines bright, Guiding tests in cloud-cast light, Fleet's future takes flight.
