Skip to content

fix: resolve references against a base without a path as RFC 3986 does - #372

Merged
mielvds merged 24 commits into
digitalbazaar:masterfrom
jdsika:fix/iri-resolve-base-without-path
Nov 18, 2026
Merged

mielvds merged 24 commits into
digitalbazaar:masterfrom
jdsika:fix/iri-resolve-base-without-path

Conversation

@jdsika

@jdsika jdsika commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

iri_resolver.resolve() mishandles a base IRI with a host but no path, such as http://example.org. It always inserts a / before the reference, so /x becomes http://example.org//x. It also keeps dot segments, so ../x becomes http://example.org/../x and abc/../def even names a different resource, http://example.org/abc/def. jsonld.expand() and jsonld.to_rdf() produce the same wrong IRIs. This is rubensworks/relative-to-absolute-iri.js#24 in the library that resolve() is ported from. Separately, remove_dot_segments() drops the / that a final . or .. segment leaves before a query or fragment (/abc/def/./ghi/../..#abc gives /abc#abc), so a/.?x against http://example.org/ becomes http://example.org/a?x and http://abc/def/.?x becomes http://abc/def?x. And resolve() drops the dot of a first segment such as .gh (.gh against http://example.org/a becomes http://example.org/gh).

from pyld import jsonld
from pyld.iri_resolver import resolve

print(resolve("/x", "http://example.org"))
# master:  http://example.org//x
# this PR: http://example.org/x
print(resolve("../x", "http://example.org"))
# master:  http://example.org/../x
# this PR: http://example.org/x

doc = {"@id": "/x", "http://example.org/p": "x"}
print(jsonld.expand(doc, {"base": "http://example.org"})[0]["@id"])
# master:  http://example.org//x
# this PR: http://example.org/x

Cause: for a base without a path, resolve() returns base_iri + "/" + reference whatever the reference is. It removes dot segments only after a later / in the reference, and then also inside its query (abc?x=/../y became abc?x=/y). A base query stays in front of the path (x against http://example.org?q became http://example.org?q/x), and for any base it is taken as part of the base path when it contains a / (jjj against http://abc/def?ghi/jkl became http://abc/def?ghi/jjj). remove_dot_segments() adds the empty last segment after a final /. or /.. only at the very end of the string, not before a ? or #.

Fix: follow RFC 3986 for references without a scheme or host:

  • a base with a host and an empty path takes / as its path (§5.2.3: "If the base URI has a defined authority component and an empty path, then return a string consisting of "/" concatenated with the reference's path") and goes through the normal merge, so an absolute path replaces the base path instead of being appended to it. This does not apply to a base with an empty authority, no path and no query, such as file://, which is still resolved against its scheme only (a against file:// gives file:a);
  • the base query is dropped in one place, right after the fragment case, because it only applies to an empty reference or a fragment (§5.2.2). A base without a scheme whose only : is in its query, such as def?x:y, now raises a ValueError for a path reference, as a base without any : does;
  • remove_dot_segments() keeps the / of a final . or .. segment before a query or fragment too (§5.2.4). This applies to every reference, including absolute IRIs and network-path references;
  • a shortcut that dropped the leading . of a reference longer than two characters is removed. It existed for . followed by a query (.?a=b), which remove_dot_segments() now handles, and it also turned a segment like .gh into gh.

The remove_dot_segments() fix depends on the base-without-path fix: on master alone it would break /x/..//.?q against http://example.org, which master resolves correctly only because its extra / makes up for the dropped one.

relative-to-absolute-iri.js has the same code; its open issue rubensworks/relative-to-absolute-iri.js#24 reports the // case. The same fix for that library is rubensworks/relative-to-absolute-iri.js#33, so the port and the original stay in line. #371 builds on this PR, as rubensworks/relative-to-absolute-iri.js#34 builds on rubensworks/relative-to-absolute-iri.js#33.

Tests

tests/iri_resolver/test_resolve.py gets 24 cases and tests/iri_resolver/test_remove_dot_segments.py 4. The existing remove_dot_segments('/abc/def/./ghi/../..#abc') case now expects /abc/#abc instead of /abc#abc; no other existing case changes. Of these 29, 24 fail on master. Five pass on master as well: a plain query, and a . or .. segment before a query or fragment, which earlier versions of this PR broke. The full suite passes: 1950 passed, 27 skipped, 1 xfailed.

  • Against http://base.org: /, an absolute path, an absolute path with dot segments, ., ./, ./abc, ../abc, .., abc/../def, dot segments in a query, a plain query, abc/.?x and abc/def/..#x.
  • Bases with a query: http://base.org?q, http://base.org?a/b and http://abc/def?ghi/jkl, and .?y and .#y against http://abc/def/?x.
  • Bases with a path: abc/.?x against http://base.org/, .gh against http://example.org/a, and .x/a against http://ex.org/x/yy/zzz/f.ttl, the first case of Bugs to verify rubensworks/relative-to-absolute-iri.js#32.
  • A final dot segment before a query or fragment in an absolute IRI (http://abc/def/.?x) and a network-path reference (//abc/def/ghi/..#x).
  • remove_dot_segments(): /. and /.. before ? and before #.
  • A base without a scheme whose only : is in its query: abc against def?x:y raises a ValueError.

resolve() was compared with a strict RFC 3986 §5.2 resolver over 782,233 reference/base pairs: a hand-picked set of 4,730, the same references with /.?q, /..?q, /.#f and /..#f appended (13,588), 100,000 random pairs, and 663,915 pairs built from every path of up to three segments taken from '', ., .., a, .a and a:b, with or without a scheme or host, and with or without a query or fragment. Nothing that master or the previous version of this PR resolves correctly changes. Together with #371, PyLD gives the same results as relative-to-absolute-iri.js with rubensworks/relative-to-absolute-iri.js#33 and rubensworks/relative-to-absolute-iri.js#34 on all of them.

Found while cross-checking RDFLib's JSON-LD output with PyLD.

@anatoly-scherbakov

anatoly-scherbakov commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Thank you very much for the PR!

Would be nice to add a test for a reference that starts with . or ./ against a base that has a host and no path.

resolve('./abc', 'http://base.org') is http://base.org/./abc on master. After removing the dot segment, RFC 3986 gives http://base.org/abc.

resolve('.', 'http://base.org') is http://base.org/. on master. The result should be http://base.org/.

Comment thread CHANGELOG.md Outdated
- IRI validation now detects multiple # fragments so these IRI's are skipped during toRDF. Fixes [toRdf#te111](https://w3c.github.io/json-ld-api/tests/toRdf-manifest#te111) and [toRdf#te112](https://w3c.github.io/json-ld-api/tests/toRdf-manifest#te112).
- In `_create_node_map()`, ignored `@id: None` subjects are now dropped before they enter the node map to prevent `_graph_to_rdf()` from crashing when sorting mixed `None` and string keys. Fixes [toRdf#te122](https://w3c.github.io/json-ld-api/tests/toRdf-manifest#te122).
- Use a helper for generalized N-Quads normalization in the test runner, since rdflib cannot parse blank-node predicates. Fixes [toRdf#te075](https://w3c.github.io/json-ld-api/tests/toRdf-manifest#te075).
- `iri_resolver.resolve()` resolves a reference against a base with an authority and an empty path, such as `http://example.org`, as RFC 3986 does: `/x` became `http://example.org//x`, and dot segments were kept in a path and removed from a query. The query of the base IRI no longer stays in front of a path, or counts as part of the base path when it contains a `/`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rephrase this to be more concise and about the exact change 🙏

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewritten in acfba82 to state only the exact changes, without a general RFC claim:

iri_resolver.resolve() fixes references without a scheme or host against a base IRI with a host and no path (/x against http://example.org gave http://example.org//x) or with a query. For those references, it also no longer drops the / that a final . or .. segment leaves before a query or fragment (a/.?x), or the dot of a first segment such as .gh.

@jdsika

jdsika commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @anatoly-scherbakov, added in c034775: ., ./ and ./abc against http://base.org now resolve to http://base.org/, http://base.org/ and http://base.org/abc (on master: http://base.org/., http://base.org/./ and http://base.org/./abc).

Testing more dot-segment cases showed that the earlier version of this PR broke cases that master got right: a final . or .. segment lost the / it leaves before a query or fragment, e.g. abc/.?x against http://base.org gave http://base.org/abc?x, and .#y against http://abc/def/?x gave http://abc/def#y. 517dc79 now splits the reference at its first ? or # and removes dot segments from its path only, which also fixes the same problem against bases with a path (a/.?x against http://example.org/ gives http://example.org/a?x on master). 9558b5f removes a shortcut for . before a query that this made unnecessary and that turned .gh into gh. 70bd858 and 6187016 tidy this up without changing any result: the base query is dropped in one place, and the reference is split only where its path is resolved against the base. c585231, 3ffa7eb and 54de2ce add the tests. Against a strict RFC 3986 §5.2 resolver, over 64,628 reference/base pairs and a second random set of 100,000, nothing that matched the RFC on master changes. The same changes are on the JS PR, rubensworks/relative-to-absolute-iri.js#33.

jdsika added a commit to jdsika/pyld that referenced this pull request Nov 12, 2026
…to this branch

This branch no longer passes on a ':' that does not end a scheme as if
it did, so remove_dot_segments() now gets the whole path. It dropped
the '/' of a final dot segment before a query or fragment:
'//h/a:b/.?q' against 'http://x/' gave 'http://h/a:b?q' instead of
'http://h/a:b/?q'. The fix for remove_dot_segments() is in pyld#372,
and it needs the handling of a base IRI without a path from pyld#372
too, so this branch builds on it, as relative-to-absolute-iri.js#34
builds on relative-to-absolute-iri.js#33. The merge also brings in
upstream master; CHANGELOG.md keeps all entries.

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
@jdsika

jdsika commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Sorry, one more change after the approval. On the JS twin of this PR (rubensworks/relative-to-absolute-iri.js#33), rubensworks pointed out that splitting the query off in resolve() only worked around a bug in remove_dot_segments(): it drops the / of a final . or .. before a ? or #. The workaround only covered relative paths, so absolute IRIs and //host references still lost the slash (resolve('http://abc/def/.?x') gave http://abc/def?x). I should have fixed the function in the first place.

f86923c now fixes remove_dot_segments() itself, 65ca8c0 removes the split (no change in results), and 8024e31 adds a test for a base like def?x:y, which now raises a ValueError. d51a0b9 merges master to resolve the CHANGELOG conflict with #373, and d167f6e updates the entry. The description is updated, and PyLD stays in step with the JS PRs.

@mielvds

mielvds commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

Ah good you caught this 👍 If it possible to stack these PR's? This repo has had a rebase and merge policy from the start and if possible, we would like to keep that

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
…a path

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
… a path

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
…its query or fragment

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
… query tests

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
remove_dot_segments() added the empty last segment after a final '/.'
or '/..' only at the very end of the string, so it dropped the '/' when
a '?' or '#' followed: '/abc/def/./ghi/../..#abc' gave '/abc#abc'
instead of '/abc/#abc' (RFC 3986, section 5.2.4). resolve() worked
around this for relative paths only, so absolute IRIs and network-path
references still lost the '/' ('http://abc/def/.?x' gave
'http://abc/def?x').

Add the empty last segment whenever the dot segment is not followed by
another '/'. This is the same fix as relative-to-absolute-iri.js#33.

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
resolve() split the reference at its first '?' or '#' so that
remove_dot_segments() would not drop the '/' of a final dot segment.
remove_dot_segments() now keeps it, so the split is no longer needed.
This changes no result.

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Since the query of the base IRI is dropped before the scheme check, a
base IRI such as 'def?x:y' has no scheme left and raises a ValueError
for a path reference, as a base IRI without any ':' does.

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
The fix now lives in remove_dot_segments(), so it applies to every
reference, including absolute IRIs and network-path references.

Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
@jdsika
jdsika force-pushed the fix/iri-resolve-base-without-path branch from d167f6e to 0dcdfdc Compare November 13, 2026 08:52
@jdsika

jdsika commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

@mielvds Done. #372 is now rebased onto master without the merge commit (24 commits), and #371 is stacked on top of it, also without merges: its own 13 commits follow #372's. The code is unchanged, as both branch tips have the same tree as before, and the resolver tests pass at every commit.

As both PRs come from my fork, #371 cannot target #372's branch, so it lists #372's commits until #372 is merged. Once #372 is in, I'll rebase #371 onto master.

@jdsika

jdsika commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Do you find the number of commits annoying and should I squash them?

@mielvds

mielvds commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

Do you find the number of commits annoying and should I squash them?

No that's fine!

@mielvds
mielvds merged commit b87fa3e into digitalbazaar:master Nov 18, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants