Conversation
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
July 27, 2026 13:40
56f4ce7 to
1e35420
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
July 27, 2026 14:57
1e35420 to
1eccbe5
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
July 28, 2026 06:08
1eccbe5 to
d87bf02
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
July 28, 2026 07:57
d87bf02 to
e65f267
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
July 28, 2026 08:27
e65f267 to
24a03e1
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
July 28, 2026 09:30
24a03e1 to
f510a20
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
August 19, 2026 06:41
f510a20 to
0a8d521
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
September 16, 2026 06:37
cdf1128 to
8caf8ed
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
September 16, 2026 07:37
8caf8ed to
fa03392
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
September 16, 2026 07:54
fa03392 to
55e6a03
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
September 16, 2026 07:55
55e6a03 to
fc7702c
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
September 16, 2026 07:56
fc7702c to
55ad969
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
September 16, 2026 08:22
55ad969 to
dd31dc8
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
September 17, 2026 06:38
58db33c to
c7e1e9a
Compare
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
September 17, 2026 06:46
c7e1e9a to
fbf6d79
Compare
gytsto
marked this pull request as ready for review
September 17, 2026 10:02
ping, the output notifier, the command grepper and the asyncio helpers are not libtelio-specific: they moved to natlab, which runs this suite as a downstream pipeline. Deleting them first keeps the next commit readable as "point everything at natlab" instead of burying it in a wall of deletions. Nothing imports these until that next commit lands.
Imports point at natlab, and the utilities that kept a local copy of a shared definition now take natlab's. TargetOS has to be the shared type: a second enum of the same shape silently fails every comparison against it, which sent every VM down the Linux branch. The logger keeps its console handler because natlab's is library-style, with no handlers attached. Container names resolve through COMPOSE_PROJECT_NAME so parallel labs on one host cannot address each other's containers. natlab.py is importable as `natlab` and shadows the installed package for anything resolving imports by path, so mypy skips that script and pylint does not verify the package's names. natlab is not a declared dependency: a checkout without it is unaffected, and run_local.py injects it when NATLAB_DIR is set.
.natlab.yml is the contract with the service: which tests to run, which build artifacts the lab must place, the files it must serve and the arguments this suite needs. CI hands natlab this checkout at the commit under test and it reports back through the trigger.
Every shard booted the whole lab, so tests that never touch a VM still waited for all of them. The shard plan declares classes that leave out what they do not need, and natlab assigns tests from the same plan, so a shard is never handed a test whose guest it never started.
The suite now imports the framework from natlab, which only the bridge installs, so the pipeline at v7.0.0 fails on import: it still runs the in-repo job, where `natlab` resolves to nat-lab/natlab.py. Temporary. Revert to a tag once libtelio-build has one containing the bridge.
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
September 18, 2026 12:42
24739ef to
9a198c7
Compare
natlab's CommandGrepper catches natlab's class while the connections here raise a local one of the same shape, so allow_process_failure never retried: a transient netsh failure on the Windows guest errored the test instead. Same fix as TargetOS - import the framework's class.
natlab exports CI_NODE_INDEX from its own NATLAB_SHARD_INDEX, so the only thing keeping the two splitters apart was NATLAB_SHARD_PLAN being absent from that job - and .natlab-tests still defines it. Both are trylast: one cuts items to its ninth, the other keeps a ninth of that, and the job passes having run about 1/81 of the suite. Also drop the blanket follow_imports = "skip" over natlab. It turned every natlab symbol into Any, which is what let a second ProcessExecError through in the first place. natlab.connection and natlab.process are followed now, so a local class of the same shape is a comparison-overlap error; run_local puts NATLAB_DIR/src on MYPYPATH so it resolves. Still skipped, with reasons in the config: `natlab` itself, which nat-lab/natlab.py shadows, and natlab's tools, which are typed against its Connection ABC that the connections here are duck-compatible with but not subclasses of.
It shadowed the installed natlab package for anything resolving imports by path, so `python -m pytest` and any PYTHONPATH containing nat-lab broke the whole suite at startup. The suite only worked because the pytest11 entry point imports natlab before pytest prepends nat-lab to sys.path. That removes the mypy exclusion, the follow_imports skip on `natlab` itself, and two pylint ignored-modules entries in one move. Also: the durations file is a balancing hint, so declare it optional rather than dying at prepare on all nine shards when the artifact lacks it; drop the NATLAB_LAB_DIR branch in openwrt_vm_util, which never ran (natlab does not put that variable in the pytest process's environment) and would have picked the same file anyway; and stop ignoring two paths twice while missing compiled_test_durations.json.
The shard plan places a test by its markers and starts a shard's services by the same keywords, so the two agree only as long as tests are marked. Nothing enforced it: a test parametrized with ConnectionTag.VM_MAC and no @pytest.mark.mac is eligible for a lite shard, which never started the mac guest, and fails there looking like anything but a missing marker. The tags come from the parametrized values each item was collected with, which is where the connections are actually built from. Tags named in a body are not counted - those are mocks and comparisons, and charging them to the test demands guests it never touches; scanning them turned 0 real findings into 193 false ones. 674 items pass today. Removing one `marks=pytest.mark.mac` fails collection naming that test.
gytsto
force-pushed
the
LLT-7477-natlab-service
branch
from
September 18, 2026 12:56
9a198c7 to
4cc9e71
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Run the nat-lab suite through the natlab service; the lab and the framework move there.
☑️ Definition of Done checklist