Repository navigation
fix: resolve references against a base without a path as RFC 3986 does - #372
Conversation
|
Thank you very much for the PR! Would be nice to add a test for a reference that starts with
|
| - IRI validation now detects multiple # fragments so these IRI's are skipped during toRDF. Fixes [toRdf#te111](https://w3c.github.io/json-ld-api/tests/toRdf-manifest#te111) and [toRdf#te112](https://w3c.github.io/json-ld-api/tests/toRdf-manifest#te112). | ||
| - In `_create_node_map()`, ignored `@id: None` subjects are now dropped before they enter the node map to prevent `_graph_to_rdf()` from crashing when sorting mixed `None` and string keys. Fixes [toRdf#te122](https://w3c.github.io/json-ld-api/tests/toRdf-manifest#te122). | ||
| - Use a helper for generalized N-Quads normalization in the test runner, since rdflib cannot parse blank-node predicates. Fixes [toRdf#te075](https://w3c.github.io/json-ld-api/tests/toRdf-manifest#te075). | ||
| - `iri_resolver.resolve()` resolves a reference against a base with an authority and an empty path, such as `http://example.org`, as RFC 3986 does: `/x` became `http://example.org//x`, and dot segments were kept in a path and removed from a query. The query of the base IRI no longer stays in front of a path, or counts as part of the base path when it contains a `/`. |
There was a problem hiding this comment.
Please rephrase this to be more concise and about the exact change 🙏
There was a problem hiding this comment.
Rewritten in acfba82 to state only the exact changes, without a general RFC claim:
iri_resolver.resolve()fixes references without a scheme or host against a base IRI with a host and no path (/xagainsthttp://example.orggavehttp://example.org//x) or with a query. For those references, it also no longer drops the/that a final.or..segment leaves before a query or fragment (a/.?x), or the dot of a first segment such as.gh.
|
Thanks @anatoly-scherbakov, added in c034775: Testing more dot-segment cases showed that the earlier version of this PR broke cases that master got right: a final |
…to this branch This branch no longer passes on a ':' that does not end a scheme as if it did, so remove_dot_segments() now gets the whole path. It dropped the '/' of a final dot segment before a query or fragment: '//h/a:b/.?q' against 'http://x/' gave 'http://h/a:b?q' instead of 'http://h/a:b/?q'. The fix for remove_dot_segments() is in pyld#372, and it needs the handling of a base IRI without a path from pyld#372 too, so this branch builds on it, as relative-to-absolute-iri.js#34 builds on relative-to-absolute-iri.js#33. The merge also brings in upstream master; CHANGELOG.md keeps all entries. Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
|
Sorry, one more change after the approval. On the JS twin of this PR (rubensworks/relative-to-absolute-iri.js#33), rubensworks pointed out that splitting the query off in f86923c now fixes |
|
Ah good you caught this 👍 If it possible to stack these PR's? This repo has had a rebase and merge policy from the start and if possible, we would like to keep that |
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
…a path Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
… a path Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
…its query or fragment Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
… query tests Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
remove_dot_segments() added the empty last segment after a final '/.'
or '/..' only at the very end of the string, so it dropped the '/' when
a '?' or '#' followed: '/abc/def/./ghi/../..#abc' gave '/abc#abc'
instead of '/abc/#abc' (RFC 3986, section 5.2.4). resolve() worked
around this for relative paths only, so absolute IRIs and network-path
references still lost the '/' ('http://abc/def/.?x' gave
'http://abc/def?x').
Add the empty last segment whenever the dot segment is not followed by
another '/'. This is the same fix as relative-to-absolute-iri.js#33.
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
resolve() split the reference at its first '?' or '#' so that remove_dot_segments() would not drop the '/' of a final dot segment. remove_dot_segments() now keeps it, so the split is no longer needed. This changes no result. Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
Since the query of the base IRI is dropped before the scheme check, a base IRI such as 'def?x:y' has no scheme left and raises a ValueError for a path reference, as a base IRI without any ':' does. Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
The fix now lives in remove_dot_segments(), so it applies to every reference, including absolute IRIs and network-path references. Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
d167f6e to
0dcdfdc
Compare
|
@mielvds Done. #372 is now rebased onto master without the merge commit (24 commits), and #371 is stacked on top of it, also without merges: its own 13 commits follow #372's. The code is unchanged, as both branch tips have the same tree as before, and the resolver tests pass at every commit. As both PRs come from my fork, #371 cannot target #372's branch, so it lists #372's commits until #372 is merged. Once #372 is in, I'll rebase #371 onto master. |
|
Do you find the number of commits annoying and should I squash them? |
No that's fine! |
Summary
iri_resolver.resolve()mishandles a base IRI with a host but no path, such ashttp://example.org. It always inserts a/before the reference, so/xbecomeshttp://example.org//x. It also keeps dot segments, so../xbecomeshttp://example.org/../xandabc/../defeven names a different resource,http://example.org/abc/def.jsonld.expand()andjsonld.to_rdf()produce the same wrong IRIs. This is rubensworks/relative-to-absolute-iri.js#24 in the library thatresolve()is ported from. Separately,remove_dot_segments()drops the/that a final.or..segment leaves before a query or fragment (/abc/def/./ghi/../..#abcgives/abc#abc), soa/.?xagainsthttp://example.org/becomeshttp://example.org/a?xandhttp://abc/def/.?xbecomeshttp://abc/def?x. Andresolve()drops the dot of a first segment such as.gh(.ghagainsthttp://example.org/abecomeshttp://example.org/gh).Cause: for a base without a path,
resolve()returnsbase_iri + "/" + referencewhatever the reference is. It removes dot segments only after a later/in the reference, and then also inside its query (abc?x=/../ybecameabc?x=/y). A base query stays in front of the path (xagainsthttp://example.org?qbecamehttp://example.org?q/x), and for any base it is taken as part of the base path when it contains a/(jjjagainsthttp://abc/def?ghi/jklbecamehttp://abc/def?ghi/jjj).remove_dot_segments()adds the empty last segment after a final/.or/..only at the very end of the string, not before a?or#.Fix: follow RFC 3986 for references without a scheme or host:
/as its path (§5.2.3: "If the base URI has a defined authority component and an empty path, then return a string consisting of "/" concatenated with the reference's path") and goes through the normal merge, so an absolute path replaces the base path instead of being appended to it. This does not apply to a base with an empty authority, no path and no query, such asfile://, which is still resolved against its scheme only (aagainstfile://givesfile:a);:is in its query, such asdef?x:y, now raises aValueErrorfor a path reference, as a base without any:does;remove_dot_segments()keeps the/of a final.or..segment before a query or fragment too (§5.2.4). This applies to every reference, including absolute IRIs and network-path references;.of a reference longer than two characters is removed. It existed for.followed by a query (.?a=b), whichremove_dot_segments()now handles, and it also turned a segment like.ghintogh.The
remove_dot_segments()fix depends on the base-without-path fix: onmasteralone it would break/x/..//.?qagainsthttp://example.org, whichmasterresolves correctly only because its extra/makes up for the dropped one.relative-to-absolute-iri.js has the same code; its open issue rubensworks/relative-to-absolute-iri.js#24 reports the
//case. The same fix for that library is rubensworks/relative-to-absolute-iri.js#33, so the port and the original stay in line. #371 builds on this PR, as rubensworks/relative-to-absolute-iri.js#34 builds on rubensworks/relative-to-absolute-iri.js#33.Tests
tests/iri_resolver/test_resolve.pygets 24 cases andtests/iri_resolver/test_remove_dot_segments.py4. The existingremove_dot_segments('/abc/def/./ghi/../..#abc')case now expects/abc/#abcinstead of/abc#abc; no other existing case changes. Of these 29, 24 fail onmaster. Five pass onmasteras well: a plain query, and a.or..segment before a query or fragment, which earlier versions of this PR broke. The full suite passes: 1950 passed, 27 skipped, 1 xfailed.http://base.org:/, an absolute path, an absolute path with dot segments,.,./,./abc,../abc,..,abc/../def, dot segments in a query, a plain query,abc/.?xandabc/def/..#x.http://base.org?q,http://base.org?a/bandhttp://abc/def?ghi/jkl, and.?yand.#yagainsthttp://abc/def/?x.abc/.?xagainsthttp://base.org/,.ghagainsthttp://example.org/a, and.x/aagainsthttp://ex.org/x/yy/zzz/f.ttl, the first case of Bugs to verify rubensworks/relative-to-absolute-iri.js#32.http://abc/def/.?x) and a network-path reference (//abc/def/ghi/..#x).remove_dot_segments():/.and/..before?and before#.:is in its query:abcagainstdef?x:yraises aValueError.resolve()was compared with a strict RFC 3986 §5.2 resolver over 782,233 reference/base pairs: a hand-picked set of 4,730, the same references with/.?q,/..?q,/.#fand/..#fappended (13,588), 100,000 random pairs, and 663,915 pairs built from every path of up to three segments taken from'',.,..,a,.aanda:b, with or without a scheme or host, and with or without a query or fragment. Nothing thatmasteror the previous version of this PR resolves correctly changes. Together with #371, PyLD gives the same results as relative-to-absolute-iri.js with rubensworks/relative-to-absolute-iri.js#33 and rubensworks/relative-to-absolute-iri.js#34 on all of them.Found while cross-checking RDFLib's JSON-LD output with PyLD.