Repository navigation
WIP: implement Jelly-RDF 1.2 - #386
Closed
Ostrzyciel wants to merge 45 commits into
Closed
Ostrzyciel wants to merge 45 commits into
Ostrzyciel wants to merge 45 commits into
Conversation
- Use `is` in term equality checks as a fast path. - Walk graphs in spog order, instead of the current "random" hash order. This improves compression ratios massively. - Add a cursor API to the serializer, so that we can only indicate changed terms, not entire statements. This skips equality checks entirely.
The upb protobuf backend copies a message by serializing and parsing it again. The writer created every row on its own and then copied it twice: the statement into its row, and every row into the frame. Now rows are added straight into the frame being built. Lookup entry rows are filled in place, and each statement is encoded into a reusable message per stream and copied once into its row. FrameFlow holds the frame and is no longer a UserList, and the encoder methods add their rows to the stream's flow instead of returning them. Output is byte-identical. Instructions per write: -30% (geomean of the three Jelly writers on dbpedia-live and nanopubs), cycles -29%.
Most IRIs repeat, but every use split the IRI into prefix and name (making new strings), hashed them and looked them up twice in each lookup table. TermEncoder now caches, per IRI, its prefix and name and the indices of their lookup entries. An index is still valid if the lookup holds the same string object at it, so a cache hit only marks the entries as used, exactly as a lookup would: eviction and the output stay the same. Reference indices are computed from the entry index instead of looking the string up again. The cache is cleared when it holds 4x the name table size. Output is byte-identical. Instructions per write: -8% (geomean), generic writer -10% to -17%, rdflib writers -4% to -7%.
The repeated-term check compared every term with the previous one using ==, which for rdflib terms runs Identifier.__eq__, plain Python code. For two terms of the same exact class URIRef or BNode, that only compares their strings. The check is now TermEncoder.same_term(), and RDFLibTermEncoder does the string comparison itself for those classes, compiled to a plain C string comparison. Literals and other terms still use ==. Output is byte-identical. Instructions per write of the rdflib writer fed from a list: -7% to -10%; the other writers are unchanged.
URIRef is a str subclass, so a dict lookup that finds an equal but different URIRef object calls rdflib's Identifier.__eq__, plain Python code. That was 5% of the rdflib writer. The rdflib encoder now passes str(iri) to the encoder, which copies the string but compares it in C. Output is byte-identical. Instructions per write: rdflib writers -2% to -6%.
Every row is added through FrameFlow.new_row(), and every statement checks frame_from_bounds(), both called from compiled code into plain Python. flows.py is now compiled too. For mypyc: the class-level default logical type is a ClassVar, default_logical_type; FrameFlow.__init__ no longer takes extra keyword arguments; BoundedFrameFlow is no longer a dataclass, and its __init__ has no @OverRide (mypyc generated broken C for it). Output is byte-identical. Instructions per write: -5% (geomean).
A statement was encoded into a reusable message and copied into its row, because the rows for its new lookup entries have to come before it. Now it is written in two steps. First the terms that changed are encoded into EncodedTerm objects (IRI indices, literal parts, blank node label, quoted triple or default graph), which adds the lookup entry rows. Then the statement's row is added and the terms are written straight into it. A statement whose terms all repeat is marked with SetInParent(). The integration encoders now encode a term into an EncodedTerm (encode_spo(term, out), encode_graph(term, out)) instead of writing into a protobuf message. Output is byte-identical. Instructions per write: -11% (geomean).
Writing an rdflib Graph or Dataset ran its loops over subjects, predicates and objects, and the grouping of each graph's triples, as plain Python. They are now in walk.py (write_tree(), write_trees()), which is compiled. Streams get one emit() for the cursor API, replacing emit_triple() and emit_quad(), so one loop serves both. walk.memory_fast_path switches the in-memory store's fast path off for tests, as monkeypatching a function no longer reaches calls inside the compiled module. Output is byte-identical. Instructions per write of the Graph writer: -7% (dbpedia-live), -13% (nanopubs).
Two rdflib Literals are never equal if their lexical forms differ, and terms of different classes among URIRef, BNode and Literal are never equal. Both cases are now decided with plain string comparisons; only literals with the same lexical form still go to rdflib's __eq__. Output is byte-identical. Instructions per write of the rdflib writer fed from a list: -2% (nanopubs) to -5.5% (dbpedia-live).
Writing a stream of rdflib statements looped over them in plain Python, and flat_stream_to_frames() wrapped the input in one more generator. The loops are now TripleStream.triples() and QuadStream.quads(), which are compiled, and the input is passed on as it is. Output is byte-identical. Instructions per write of the rdflib writer fed from a list: -2%.
On the DGX Spark the two clusters of performance cores (5-9, 15-19) differ by about 3%, so a case should not move between them across runs. Without --cpus, `run` now pins to one cluster of the fastest cores, found from sysfs (highest maximum frequency, shared last-level cache), or keeps the cores it is already pinned to. With several --cpus sets, each case always runs on the same set. The README no longer says the two clusters give the same results.
rdflib.Literal normalizes by default: it parses the lexical form into a Python value and writes it back. That is slow and changes the data, e.g. "2013-12-31T22:50:02Z" became "2013-12-31T22:50:02+00:00". The rdflib reader now creates literals with normalize=False. Instructions per read on dbpedia-live: -12% (Graph), -13% (statements).
Like Jelly-JVM, the decoder keeps the last IRI term made for each name table entry, with the name and prefix strings it was made from. If the same strings come up again, the term is reused instead of building the IRI string and asking the adapter for a new term. Adapters' IRI terms must therefore be immutable (rdflib's and the generic ones are). Instructions per read: -12% (geometric mean), -24.5% for rdflib statements on nanopubs.
The decoder keeps up to 16384 literal terms, keyed by lexical form, language and datatype, and empties the cache when it is full. Literals repeat often (26% to 94% hits on the RiverBench datasets we checked), and rdflib's Literal is slow to make, as it parses the value. Adapters' literal terms must therefore be immutable. Instructions per read: -5% to -11% for rdflib, about the same for the generic reader.
Dataset.add made a new context Graph for every quad, even when given one. When the Graph or Dataset class doesn't override add(), the parser now keeps one context Graph per graph name and gives statements to the store's addN in batches of 1000, which also helps stores with a bulk insert. Otherwise it calls add() for every statement, as before. Instructions per read: -45% for a Dataset on nanopubs, -10% for a Graph on dbpedia-live.
The decoder asked each statement for each of its oneofs with WhichOneof() and then read the field by name, and found the term handler in a dict by the field's type. It now gets all set fields with one ListFields() call and works out each term's position and kind from the field number. upb looks up any attribute of a message as a field name first, even a method's name, so WhichOneof() and ListFields() are now called through the message classes. Instructions per read: -3% to -7%.
The decoder read each row's field by name, then found its handler in a dict by type, and checked the row's type again with isinstance() to know whether to yield the result. It now branches on the field name that WhichOneof() gives, most common rows first. Instructions per read: -8% to -14%.
Triple(*terms) and Quad(*terms) ran a __new__ written in Python (for the generic namedtuples, one generated from a string). The adapters now call tuple.__new__ directly. Instructions per read: -2% to -5%.
The adapters moved from generic/parse.py, which tests monkeypatch and so stays interpreted, to a new generic/decode.py compiled with mypyc. The decoder now calls them without going through the interpreter. Instructions per read for the generic reader: -7% (dbpedia-live), -2.4% (nanopubs).
The adapters moved from rdflib/parse.py to a new rdflib/decode.py, compiled with mypyc. Triple, Quad and Prefix moved to a new rdflib/statements.py, which stays interpreted, as mypyc can't compile subclasses of tuple. The unused _adapter_missing() was dropped. Instructions per read for rdflib: -1% to -2.5%.
Indexing a deque walks through its blocks of 64 entries, so reading an entry from a 4000-name table took tens of steps. Instructions per read: -2% to -3%.
- Drop code the branch left unused: the HasGraph, T and Terms aliases, and LookupEncoder.encode_prefix_term_index() and encode_name_term_index(), replaced by *_index_to_write(). Their tests now test the methods in use, and no longer need patching, so they also run against the compiled build. - Parse CPU lists in one place in the benchmarks. - Simplify decode_literal() and stream_for_type(), fix docstrings that described the old code, and remove comments that only repeat the code.
Write Jelly-RDF 1.2 (the column layout) by default, and read all versions of Jelly-RDF. Setting the protocol version to 1 or 2 still writes the row layout of Jelly-RDF 1.0 and 1.1. The compatibility choices follow Jelly-JVM: GRAPHS streams are not written, the 1.1-only options are left out of 1.2 streams, and streams of graphs or datasets are written as streams of RDF Messages, one message per graph or dataset. - New compiled modules build (serialize/columns.py) and read (parse/columns.py) column batches. Each frame is one epoch of the lookups, with a budget, so that all its entries fit in them. - The reader checks the stream options of each layout, accepts restated options, and groups the statements of a stream of RDF Messages by message. An empty stream is an empty graph. - The generic API supports RDF 1.2 triple terms and base directions. - The conformance tests run both the 1.1 and 1.2 suites, with jelly-cli 0.11. On RiverBench data, 1.2 is 1.7-2.2x faster to write, 1.5-2.3x faster to read, and 25-53% smaller than 1.1.
The term types of a column whose values are of more than one type were first unpacked into a list, counted four times, and then merged with a next() call per value. The kinds field also came from protobuf untyped, so mypyc indexed it as any object. Now one pass over the kinds, typed as bytes, takes each value from its list by index. Jelly-RDF 1.2 read: -11.7% (generic) and -9.0% (rdflib) instructions per statement, geomean of three RiverBench datasets.
When reading into rdflib, URIRef() checks every new IRI with a regular expression, and only logs a warning for an invalid one. The check is now skipped unless validate_iris=True is passed to the parse functions or to Graph.parse(). The IRI terms are the same either way. Jelly-RDF 1.2 read with rdflib: -9.3% instructions per statement, geomean of three RiverBench datasets.
A column of mixed term types kept one int per value, which were packed at the end of the frame with bytearray operations that mypyc compiles as generic Python ones. The column now keeps the packed bytes, ORing each value's two bits into the last byte, as Jelly-JVM does. Same output. Jelly-RDF 1.2 write: -4.9% (generic) and -2.9% (rdflib) instructions per statement, geomean of three RiverBench datasets.
A BNode went through isinstance() checks against URIRef and Literal before its own. For rdflib's terms, which derive from abstract base classes, such a check runs ABCMeta.__instancecheck__ in Python. Exact type checks now come first, for subjects, objects and graph names. Jelly-RDF 1.2 write with rdflib: -5.0% instructions per statement (-11.4% on the dataset with the most blank nodes), geomean of three RiverBench datasets.
Each value that takes one cell was a run of its own, added with two list appends and read back by loading the run. A stretch of such values is now one run that next() walks through by index, so a column without layout tokens is a single run. The layout tokens are copied into a typed list first, so that mypyc indexes them natively. Jelly-RDF 1.2 read: -11.8% (generic) and -9.8% (rdflib) instructions per statement, geomean of three RiverBench datasets.
Each IRI of a column was resolved with Decoder.iri_term(), which looked up the name and the prefix with LookupDecoder.at(), and the ids came from the protobuf containers untyped. The ids are now copied into typed lists, and the IRIs resolved against the lookup lists and the IRI cache in the loop; only a new IRI calls out. Jelly-RDF 1.2 read: -9.6% (generic) and -8.1% (rdflib) instructions per statement, geomean of three RiverBench datasets.
list() on a protobuf repeated field goes through the sequence iterator, one call per element; a slice of it is copied by upb in one call. Jelly-RDF 1.2 read: -5.7% (generic) and -4.6% (rdflib) instructions per statement, geomean of three RiverBench datasets.
Each statement of a Jelly-RDF 1.2 frame went through six nested generators before reaching the caller. Decoder.iter_rows(), the frame and batch setup, and flat_rows() now return iterators instead of yielding (itertools.chain in C for flat_rows), and one generator yields a batch's statements and the ends of its RDF Messages. A frame's options and lookup entries are now read when iter_rows() is called, not on the first row. Jelly-RDF 1.2 read: -3.8% (generic) and -2.9% (rdflib) instructions per statement, geomean of three RiverBench datasets.
The literal cache was keyed by (lexical form, language, datatype), a new tuple for every literal. It is now one dict of lexical forms per language tag and datatype. A column whose literals are all of one kind looks them up in that dict directly, and a column of mixed kinds resolves each kind once. Jelly-RDF 1.2 read: -1.3% (generic) and -0.8% (rdflib) instructions per statement, geomean of three RiverBench datasets; -3.2% and -1.5% on the dataset with the most literals.
split_iri() called rpartition() up to twice and joined the prefix with its separator, making several strings for each new IRI. One or two rfind() calls and two slices give the same split. Jelly-RDF 1.2 write: -3.6% (generic) and -2.8% (rdflib) instructions per statement, geomean of three RiverBench datasets; -6.3% and -4.9% on the dataset with the most new IRIs.
Literal.language and Literal.datatype are Python properties, so each literal written made two Python calls. The slots behind them, _language and _datatype, have been there since rdflib 4. Jelly-RDF 1.2 write with rdflib: -3.3% instructions per statement, geomean of three RiverBench datasets; -5.7% on the dataset with the most literals.
Given a label, BNode.__new__ only passes it on to str.__new__, after a few Python statements. The reader now makes the same object with str.__new__(BNode, label). Jelly-RDF 1.2 read with rdflib: -1.7% instructions per statement, geomean of three RiverBench datasets; -4.7% on the dataset with the most blank nodes.
A column of literals of mixed kinds called Decoder.literal_term() for each literal, which looked up the cache of the kind with a new key tuple. Each kind is now resolved once into an object that holds its parts and its cache, and is used for columns of one kind too. Jelly-RDF 1.2 read: -3.7% (generic) and -2.3% (rdflib) instructions per statement, geomean of three RiverBench datasets.
At the end of each frame, the prefix ids and literal kinds of every column were compressed in another pass, into new lists. The column now stores the prefix ids as written (0 for the same as the previous one) and notes whether any of them, or any literal kind, differs, so the lists are written as they are, or as their one value. Jelly-RDF 1.2 write: -3.5% (generic) and -2.7% (rdflib) instructions per statement, geomean of three RiverBench datasets.
mypyc compiles tuple.__new__(Triple, terms) as a method call that looks up __new__ on tuple for every statement. The adapters now call the function bound once at module level, for statements and for rdflib's IRIs and blank nodes. Jelly-RDF 1.2 read: -3.4% (generic) and -3.8% (rdflib) instructions per statement, geomean of three RiverBench datasets.
The column writer unpacked each statement as any iterable, through the iterator protocol. Statements are almost always tuples, which are now indexed directly; anything else is still unpacked. Jelly-RDF 1.2 write: -2.1% (generic) and -1.7% (rdflib) instructions per statement, geomean of three RiverBench datasets.
Ostrzyciel
marked this pull request as draft
October 26, 2026 07:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.