Skip to content

FIXED_TESTS.md

Latest commit

 

History

History
71 lines (58 loc) · 5.1 KB

FIXED_TESTS.md

File metadata and controls

71 lines (58 loc) · 5.1 KB
Raw

FIXED_TESTS.md: Salt Merge-Forward CI Regressions (3006.x -> 3007.x)

This document tracks the test regressions and CI failures resolved during the merge of Salt 3006.x into 3007.x (PR #68929).

1. Package Lifecycle Tests (Downgrade/Upgrade)

  • Files:
    • tests/pytests/pkg/downgrade/test_salt_downgrade.py
    • tests/pytests/pkg/upgrade/test_salt_upgrade.py
  • Symptom: AssertionError where 3007.13 was incorrectly evaluated as equal to 3007.13+187.g813a978cff due to .base_version usage.
  • Fix: Switched to full packaging.version.Version objects for comparison, correctly identifying that dev/git versions are "greater than" the base stable version. Also initialized original_py_version = None to resolve pylint warnings.

2. Salt-SSH Unit Tests

  • Files:
    • tests/pytests/unit/client/ssh/test_ssh.py
    • tests/pytests/unit/client/ssh/test_password.py
  • Symptom: ValueError (too many values to unpack) and AttributeError after refactoring.
  • Fix:
    • Refactored tests to match the renamed _handle_routine_thread method.
    • Updated mocks to handle the new 3-tuple return format (stdout, stderr, retcode).
    • Added robust retcode = None handling.
    • Switched to ANY for opts in display_output mocks to accommodate merge-added internal configuration keys.

3. Salt-Mine Integration & Runner Tests

  • Files:
    • tests/integration/modules/test_mine.py
    • tests/pytests/integration/runners/test_mine.py
  • Symptom: Flaky failures and race conditions where Mine data was not available immediately after being sent.
  • Fix: Ported 30-second polling logic and mine.update patterns from master to ensure data consistency before assertions.

4. Async Client Unit Tests

  • File: tests/pytests/unit/test_client.py
  • Symptom: RuntimeError: Event loop is closed and JID nesting errors in pub_async.
  • Fix: Ported the async def test pattern from master, ensuring Tornado/Asyncio loops are properly managed and that jid and timeout are correctly extracted from nested return structures.

5. Loader/Grains Cleanup Tests

  • File: tests/pytests/unit/loader/test_grains_cleanup.py
  • Symptom: Failures in grain provider cleanup due to stub module interference.
  • Fix: Aligned module filtering logic with master to correctly handle (and ignore) stub modules that were causing cleanup failures.

6. System Verification Unit Tests

  • File: tests/pytests/unit/utils/verify/test_verify.py
  • Symptom: Hard Crash/Hang of the unit test shard (specifically Unit 4 on Linux).
  • Fix: Patched resource.getrlimit and resource.setrlimit (and Windows equivalents) to prevent the test from actually lowering the process file descriptor limit to 256. Previously, hitting this limit caused Salt's logging and master processes to crash recursively without a summary.

7. Package Ownership Integration Tests

  • File: tests/pytests/pkg/integration/test_salt_user.py
  • Symptom: AssertionError: assert 'salt' == 'root' at various paths (e.g., /etc/salt/pki/minion/minion.pub, /var/cache/salt/master/proc).
  • Fix: Refactored test_pkg_paths to use a non-recursive, explicit path check for salt user ownership. This correctly aligns the test with Salt's 3006.x+ multi-user security model, where root-owned subdirectories often exist within salt-managed parent directories, and avoids the cascading failures caused by the previous recursive logic.

8. Integration Shard 1 (Widespread Collision)

  • Symptom: 169+ failures in Ubuntu 24.04 (and other Linux) integration shards.
  • Error: salt.loader.lazy: ERROR Module/package collision: '.../salt/utils/vault.py' and '.../salt/utils/vault'.
  • Fix: Deleted the redundant salt/utils/vault.py (which was accidentally restored from 3006.x) in favor of the salt/utils/vault/ directory structure required by 3007.x. Also removed redundant tests/pytests/unit/utils/test_vault.py.

9. GPG Key Download Failures

  • File: tests/support/pytest/helpers.py
  • Symptom: requests.exceptions.ConnectionError in restricted/air-gapped CI environments when downloading Broadcom GPG keys.
  • Fix: Added a local PGP public key fallback to the download_file helper, allowing tests to proceed even when the Broadcom artifactory is unreachable.

10. Systemd Masked Service Hangs

  • File: tests/pytests/pkg/upgrade/systemd/test_service_preservation.py
  • Symptom: 5-hour Hang in package upgrade tests.
  • Fix: Disabled automated service stopping for masked units during the install(upgrade=True) call. systemctl stop can block indefinitely on masked services in certain environments.

Core Supporting Fixes (Verified)

The following core changes were required to enable the test fixes above:

  • salt/client/ssh/__init__.py: Fixed SSH._expand_target to preserve user prefixes (e.g., user@host).
  • salt/pillar/__init__.py: Added deepcopy(opts) for Pillar renderer isolation.
  • pkg/windows/nsis/installer/Salt-Minion-Setup.nsi: Restored PR-original Windows MSI fix.