Translate the whole tyr benchmark dataset: 6 failure classes

[MBus123]

Swept the tyr dataset (planning-and-learning/planning-benchmarks-numeric, the suite tyr itself uses) with the translator, smallest instance per domain, 30 s budget each.

261 of 329 domains translated so far (sweep still running over ~470). One class already fixed in 11e5c26. The rest group into five.

What the dataset needs, and what we accept

The dataset ships repaired requirement declarations (strict_clean) plus committed requirements.*.json. Across all 468 domains exactly 11 requirements appear:

:strips :typing :negative-preconditions :action-costs :numeric-fluents
:equality :disjunctive-preconditions :existential-preconditions
:universal-preconditions :conditional-effects :derived-predicates

All 11 are in SUPPORTED_REQUIREMENTS. Nothing in the dataset is out of scope by declaration; :adl never appears because strict_clean expands it into its constituents.

1. Parameterised static cost functions — 19 domains

The largest class, and it is mainstream rather than exotic: elevators (4), agricola (2), data-network (2), cavediving, and more.

panicked at translate.rs:740:
cost expression PNE travel-slow(?f1, ?f2) of action (move-up-slow slow0-0 n2 n3)
has no numeric variable

The action is ground but its cost expression still carries ?f1, ?f2. The idiom is the standard one for IPC action costs:

(:functions (total-cost) - number (travel-slow ?f1 - count ?f2 - count) - number)
(:action move-up-slow :parameters (?lift ?f1 ?f2)
  :effect (and ... (increase (total-cost) (travel-slow ?f1 ?f2))))

travel-slow is static: assigned in :init, never modified. Action::instantiate does pass var_mapping to the cost, and instantiate_expression substitutes and folds a static PNE with a known init value into a constant. So the substitution is written but something on this path is not applying it. Worth checking whether the cost reaching sas_operator_cost is the lifted one.

2. Condition atom with no SAS variable — 11 domains

caldera (4), caldera-split, folding (2), others.

panicked at translate.rs:354:
condition atom Atom { predicate: "prop_host", args: ["id_birat", "id_zhost"] }
has no SAS variable; it is neither reachable nor statically decided

The assertion is right to fire; the question is why an atom survives Condition::instantiate_action without being resolved either way.

3. settlers — 2 domains

panicked at translate.rs:283. Not examined.

4. either types — 1 domain

storage: Expected atom, got list: List([Atom("either"), Atom("storearea"), Atom("crate")]). PDDL's disjunctive type (either t1 t2) in a typed list is unsupported by the type parser. Fast Downward supports it.

5. pipesworld-tankage, schedule — 2 domains

panicked at parsing_functions.rs:703. Not examined.

6. Grounding blow-up — 31 timeouts

All 16 mine-pddl (Minecraft) domains plus large classical instances. Not a support gap: mine-pddl/Bridge_Easy reaches 30.7 M relevant atoms, 70.7 M auxiliary, 101 M queue entries and is still going after two minutes. The repository already excludes two Minecraft fixtures as too slow for test builds. Worth a separate issue about grounding scale rather than lumping with the correctness bugs above.

Fixed already

normalize.rs:964 (2 Minecraft domains) was a regression from the silent-skip sweep: the new assertion rejected a conjunction nested in a conjunction, which is harmless. Fixed in 11e5c26.

Suggested test

A test that translates one instance per domain from this dataset, skipped when the checkout is absent. Every bug in this list was invisible to the repository's own fixtures.

[MBus123]

Progress: 24 of the 43 non-timeout failures now translate

Three fixes since the first sweep, each verified against the dataset:

Parameterised static cost functions (was 22 domains) — fixed in 405c489. Two bugs on the same idiom:

• Action::uniquify_variables renamed the parameters, precondition, effects and assignment effects, and forgot the cost. So parameters held the new names while cost kept the old ones, grounding bound the new names, substituting the cost changed nothing, and the cost arrived at SAS translation still spelled (travel-slow ?f1 ?f2).
• An action whose cost is a static function is only reachable where that function has a value. elevators defines travel-slow only for travellable floor pairs, so (move-up-slow ?lift ?f ?f) was generated with no cost at all. Reachability now carries the cost function's defined! guard, which is what Fast Downward does by handing the cost expression to condition_to_rule_body.

Worth noting what the second one would have looked like without the assertion that caught it. Refusing the task is loud; the quiet version is an operator whose cost is whatever the failed lookup returned, on a domain family where costs are the entire point.

elevators, agricola, data-network, cavediving and storage all translate now, pinned by action-costs/parameterised-cost-function.

Disjunctive types (1 domain) — fixed. storage declares (in ?x - (either storearea crate) ?p - place). Recorded under a joined name, which is enough for a declaration since declared argument types do not drive grounding here. Not enough on an action parameter, where it would need a disjunction of type atoms; no benchmark domain does that and the limitation is written at the site.

Nested conjunction (2 domains) — fixed in 11e5c26. A regression from the silent-skip sweep: the new assertion rejected a conjunction nested in a conjunction, which is harmless.

Still failing: 19

count	cause	domains
14	translate.rs:354 condition atom with no SAS variable	caldera (4), citycar, folding (2), psr-large, psr-middle (2), labyrinth (2), pipesworld-tankage (2)
2	translate.rs:283	settlers (2)
2	parsing_functions.rs:728	pipesworld-tankage, schedule
1	grounding timeout	mine-pddl/Move_to_Location

The 14-domain class is now the one to take next, and it has spread beyond ADL: psr, labyrinth and pipesworld-tankage are propositional. That makes it more likely to be one cause in instantiation rather than something ADL-specific.

[MBus123]

Refining the 14-domain class

The missing atoms are not confined to one kind of predicate, which rules out the two obvious explanations:

domain	missing atom	kind
psr-middle	fed(?l)	a :derived predicate ((:derived (fed ?l - LINE) ...))
citycar-opt14-adl	arrived(?c, ?x)	an ordinary predicate, and an action effect (car_arrived)

So it is not "derived predicates are missed" and not "ADL constructs are missed" — citycar has no :derived at all, and psr, labyrinth and pipesworld-tankage are propositional.

Both atoms must be reachable, because instantiation returns None for an atom that is neither in fluent_facts nor in init_facts, which drops the operator instead of reaching this assertion. And get_fluent_predicates does include every axiom head, so a derived predicate is fluent.

That narrows it to variable creation: an atom the reachability model proved reachable did not end up in any fact group, so it has no SAS variable. Look at fact_groups and whether every reachable atom is guaranteed a group, including a singleton one, rather than only those an invariant covers.

Confirming that would be quick: for one failing task, compare the reachable-atom set against the union of the fact groups and print the difference.

[MBus123]

The 14-domain class is about unreachable goal atoms

The assertion now reports how many atoms of the same predicate do have a variable. That was enough to diagnose it in one run.

citycar-opt14-adl, instance p2-2-2-1-2:

condition atom Atom { predicate: "arrived", args: ["car0", "junction1-1"] } has no
SAS variable ... 2 other atoms of that predicate do have one, for example
[arrived(car1, junction0-1), ...]

And the task's goal is:

(:goal (and (arrived car0 junction1-1) (arrived car1 junction1-0)))

So the atom with no variable is a goal atom. That fits the spread across psr, labyrinth and pipesworld-tankage: it is neither an ADL nor a derived-predicate problem.

There are two readings, and they need different fixes, so this needs deciding before coding:

(a) The relaxation genuinely cannot reach the goal atom. Then the task is provably unsolvable and the right output is a trivially unsolvable SAS task, which is what Fast Downward emits. Panicking is wrong but the analysis is right. Note the code already knows this shape: Bookkeeping::GoalReachable in instantiate.rs carries a comment saying the translation "already answers with a trivial task when the goal is out of reach".

(b) Reachability is under-approximating. These are solvable IPC instances that Fast Downward translates, and only 2 arrived atoms got variables where a 2-car instance should give more. If the relaxed model is missing reachable atoms, that is the more serious bug, and it would also silently weaken every heuristic that reasons over the grounded task rather than only breaking these 14 domains.

Distinguishing them is cheap: count the arrived atoms Fast Downward's own translator produces for the same instance and compare. Worth doing first, because (b) would change the priority of this issue entirely.

[MBus123]

41 of the 43 non-timeout failures now translate

The 14-domain class is fixed in 0c3ef41. A goal fact the relaxation cannot reach means the task has no plan, and the translation now emits a trivially unsolvable task instead of aborting.

The comparison against Fast Downward is what settled the diagnosis rather than a guess. On citycar-opt14-adl/p2-2-2-1-2 its translator reports 2 facts and 0 operators, having found the same goal out of reach. So the two readings resolved cleanly: our reachability analysis was right all along and only the response was wrong. The alternative — that we under-approximate and miss atoms FD finds — would have been much worse, weakening every heuristic rather than breaking 14 domains.

The check lives in the caller, not in translate_strips_conditions, because that function's invariant is sound for an operator and an axiom but not for the goal: instantiation strips the statically true and false atoms from a condition, and the goal is never instantiated. That is precisely why only goals ever hit it, and the assertion stays where it does hold.

adl/unreachable-goal pins it. The risk in this change is the opposite error, so I also confirmed the other 11 fixtures still reach their optima and delivery/pfile1 is unchanged at cost 22 / 35242 expansions.

Remaining

settlers-opt18-adl, settlers-sat18-adl — the only real failure left:

panicked at translate.rs:283:
PNE PrimitiveNumericExpression { symbol: "derived!difference_PNE ?sold()_PNE spl1()",
args: [], ntype: 'D' } not in numeric dictionary

A derived numeric expression, created by the pass that flattens compound arithmetic into binary axioms, does not get a numeric variable. Note the symbol still embeds ?sold — an unsubstituted variable inside a derived function's name — which looks like the same shape as the cost bug fixed in 405c489: a name minted before substitution.

38 timeouts, of which 16 are mine-pddl. Grounding scale, not support; Bridge_Easy reaches 30.7 M relevant atoms. Deserves its own issue.

A caution for anyone re-running the sweep: picking "the problem file" mechanically is error-prone in this dataset. sort puts pfile10 before pfile1, so a naive sweep times out on the largest instances, and several suites pair X-domain.pddl with X.pddl instead of sharing a domain.pddl. Two of my early "failures" were that, not the planner.

[MBus123]

Measured against tyr's own suite definitions, not just the dataset

Earlier numbers came from sweeping the benchmark dataset directory. That is not the same question as "does tyr's suite selection translate", so here it is properly.

planning-and-learning/tyr defines its suites in experiments/suite.py: 13 suites, 253 distinct domains.

suite	domains
SUITE_IPC_OPTIMAL_STRIPS	66
SUITE_IPC_SATISFICING_STRIPS	64
SUITE_AUTOSCALE_OPTIMAL_STRIPS	42
SUITE_AUTOSCALE_AGILE_STRIPS	42
SUITE_HTG	41
SUITE_IPC_OPTIMAL_ADL	24
SUITE_IPC2023_NUMERIC	20
SUITE_IPC_SATISFICING_ADL	19
SUITE_MINEPDDL	16
SUITE_IPC_LEARNING	10
SUITE_PUSHWORLD	4
SUITE_CNOT_SYNTHESIS	3
SUITE_BELUGA2025_SCALABILITY_DETERMINISTIC	1

Result over every one of those domains present in the local checkout:

• 196 translate
• 7 fail, all of them timeouts at 25 s: mine-pddl/Bridge, mine-pddl/Build_Cabin, three organic-synthesis variants, beluga2025/scalability-deterministic, organic-synthesis-sat18-strips
• 0 crashes, 0 unsupported requirements
• 46 not in this checkout: 30 htg-domains entries in tyr's flattened layout (htg-domains/flat/childsnack-contents-parsize1-cham3 and similar), which this checkout does not carry, and 16 autoscale entries

The 16 autoscale ones were a path bug in my sweep, not absences. Swept directly: 84 of 84 translate, covering both SUITE_AUTOSCALE_OPTIMAL_STRIPS and SUITE_AUTOSCALE_AGILE_STRIPS.

What "fixed PDDL" means here, from the source

src/pypddl_datasets/scripts/strict_clean.py states it repairs exactly two things and that "nothing else in the files is touched":

1. Requirements declarations — aggregates removed so they are declared atomically, unused tokens dropped, missing ones added. This is why :adl never appears in the dataset: it is expanded into its constituents.
2. For :action-costs tasks — the missing (:metric minimize (total-cost)) and (= (total-cost) 0) are added.

The second is worth noting against this repository's history. The dataset deliberately normalises tasks onto the total-cost idiom, and until e39c864 that idiom reported every plan at twice its cost here. So the repair the dataset applies is precisely the shape that was broken, and every :action-costs domain in these suites would have been silently mis-costed.

Sweeping this dataset: a warning

Domain files appear in four layouts, and guessing wrong makes a domain file look like a problem, which then reads as a parser bug. My first pass over tyr's suites reported ten parser failures in openstacks, organic-synthesis, trucks and others; all ten were this, and all ten translate fine. The layouts are:

• a shared domain.pddl
• <stem>-domain.pddl
• domain-<stem>.pddl
• domain_<stem>.pddl

Also sort puts pfile10 before pfile1, so a naive sweep measures the largest instance in each directory and reports timeouts that say nothing about support.

[MBus123]

Correction to the counts in the previous comment

I read the result file while the sweep was still running and reported the partial numbers as final. The completed run over tyr's 253 domains is:

• 200 translate (not 196)
• 21 fail (not 7) — all 21 are timeouts at 25 s, no crashes and no unsupported requirements
• 46 not in this checkout, as described above

The extra 14 are the rest of mine-pddl, which the sweep had not reached yet. The full timeout list is all 16 mine-pddl domains, three htg-domains/organic-synthesis-*, downward-benchmarks/organic-synthesis-sat18-strips, and beluga2025/scalability-deterministic.

The conclusion is unchanged: nothing in tyr's suite selection is unsupported, and every failure is grounding scale on the domains already known to blow up. Only the count was wrong.

[MBus123]

Measured on 7045f0d. Not fixed — and the root cause is different from what I had assumed.

Method: planforge --search 'astar(blind())' <domain> <problem>, 60–90 s limit, classified by log phase. Generated N rules followed by silence means grounding never finished; f = … lines mean grounding completed and search began.

mine-pddl — still broken, and it is memory, not time

Bridge_Easy prints Generated 3134 rules, then reaches 18.4 GB peak RSS and is killed with no search output. It never reaches the invariant budget at all.

organic-synthesis (opt18-strips) — now partly working

Instance	Result
p01	solved, cost 1, milliseconds
p10	solved, cost 2
p05	grounding, 16.6 GB peak, no search
p15	grounding, 14.5 GB peak, no search
p20	grounding, 16.6 GB peak, no search

Corrected diagnosis. The failure is memory exhaustion during instantiation — 14–18 GB — before search starts and before the invariant budget is consulted. The work landed on main (lazy cartesian products with early exit, CPU deadline threaded down into check_balance → is_solvable, propositional axioms grounded from the reachability model instead of the |objects|^arity type cross-product) targeted the invariant-finding phase. Those were real inefficiencies and are worth keeping, but these instances die earlier than that code runs.

Next steps

1. Instrument a counter for ground actions/atoms produced before OOM in instantiate.rs / build_model.rs.
2. Check whether reachable-tuple filtering happens before or after materialisation.
3. Run Fast Downward's own translator on the same instances. If FD also exhausts memory, these domains are simply beyond a grounding planner, and this issue should be reframed as "detect and report grounding blowup early" rather than treated as a bug to fix.

[MBus123]

Follow-up: I ran the reference implementations on the same instances. These domains defeat Fast Downward too, so this is not a PlanForge defect.

organic-synthesis-opt18-strips/p05

Translator	Time	Peak RSS	Result
PlanForge	<60 s	16.6 GB	killed, no search
Fast Downward (downward/src/translate/translate.py)	305 s	26.7 GB	TIMEOUT

mine-pddl/Bridge_Easy

Translator	Time	Peak RSS	Result
PlanForge	<90 s	18.4 GB	killed, no search
Numeric Fast Downward (numeric-fd/src/translate/translate.py)	241 s	16.0 GB	TIMEOUT

Both stall in the same phase — FD prints Computing model... and never returns, which is the Datalog model computation, i.e. exactly the grounding step where PlanForge exhausts memory.

Proposed reframing. "Translate the remaining 21 domains" is not achievable by fixing a bug, because the reference implementation — which PlanForge is a port of — cannot translate them either, and uses more memory doing it. What is achievable and worth doing:

1. Detect grounding blowup and fail cleanly. Track ground actions/atoms during instantiation against a configurable budget and report grounding exceeded N actions / M MB instead of being OOM-killed with no output. That is the CLAUDE.md rule — an unknown state should be made known, not guessed at — and it turns an unexplained death into a diagnosis.
2. Report the numbers on the way out so a user can tell "too big to ground" from "planner bug".

Suggest retitling this issue accordingly. The stage-2 grounding work (lazy cartesian products, deadline threading, axioms grounded from the reachability model) stays valuable on the domains that do translate — it just was never going to rescue these.

[MBus123]

Implemented on d54c983: grounding now enforces budgets and fails with a diagnosis instead of being OOM-killed.

Before — silence, then the OOM killer, 14–18 GB. After, on the same instances, in about 5 seconds:

INFO Generated 1346 rules.
INFO Grounding progress: 0 actions, 8879099 atoms, approximately 1.03 GiB held; building the grounding model.
Error: grounding exceeded the atom limit: 10,002,432 ground atoms (limit 10,000,000) while
building the grounding model; the task is likely too large to ground. Raise
--max-ground-atoms to continue, or use a smaller instance.

mine-pddl/Bridge_Easy behaves the same way (10,002,989 atoms). New flags: --max-ground-actions, --max-ground-atoms, --max-grounding-memory. Verified: no partial SAS task is emitted on failure (exit 1, zero files written), the 6-config expansion gate stays 6/6, and all 20 ipc2023 domains still translate.

The diagnosis gives a sharper localisation than we had. Both instances die at ~10 M ground atoms with 0 actions instantiated — so the explosion is in the reachable-atom set during model building, before action instantiation. That means the earlier stage-2 work (lazy cartesian products in is_solvable, deadline threading into check_balance) was aimed at the wrong phase: those run in invariant finding, which is reached later. They remain worthwhile for the domains that do translate; they were never going to help here.

If this is ever attacked directly, the target is the Datalog model computation / reachable-atom enumeration in build_model.rs, not instantiation and not invariant finding.