Skip to content

fix: prepare earcut payload inside the queued worker task - #230

Merged
kylebarron merged 2 commits into
geoarrow:mainfrom
jsve:fix/earcut-prepare-in-task
Sep 25, 2026
Merged

kylebarron merged 2 commits into
geoarrow:mainfrom
jsve:fix/earcut-prepare-in-task

Conversation

@jsve

@jsve jsve commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Closes #225

Background

_earcutPolygonData() and _earcutMultiPolygonData() cloned the geometry with preparePostMessage(..., true) (and interleaved separated coordinates) before calling pool.queue(), then waited on pool.completed().

When many record batches create polygon layers faster than earcut can keep up, for example with progressive rendering:

  • every layer waiting for a worker held its own copy of its geometry, no matter how small the pool
  • each layer waited for the whole shared pool to drain instead of its own task, and because completed() rejects on any task failure, one layer's error could fail another layer's triangulation

Change

Interleaving and preparePostMessage now run inside the queued task, and each layer awaits the task returned by pool.queue(). Copies are limited by the number of active pool tasks, which is the pool size at the default concurrency of one. Each layer resolves or fails with its own triangulation only. metrics timing is unchanged.

Tests

Added tests/solid-polygon-layer-earcut.test.ts. It uses a real threads Pool with an in-process worker that clones, rehydrates, and triangulates the payload the same way the earcut worker does. For Polygon and MultiPolygon data, with both interleaved and separated coordinates:

  • while the only worker is busy and the layer's task is queued, neither getInterleavedPolygon nor preparePostMessage has run
  • after the worker is released, preparePostMessage runs once, getInterleavedPolygon runs only for separated coordinates, and the result matches main-thread earcut
  • a layer resolves with its own result while another pool task is still running

All eight tests from 8558c54 fail on main (the last one by timing out) and pass after applying de1478a. pnpm typecheck, pnpm test, and biome ci . pass.

_earcutPolygonData() and _earcutMultiPolygonData() interleaved and cloned
the geometry with preparePostMessage(..., true) before queueing the task,
then waited on pool.completed(). Every layer waiting for a worker therefore
held its own copy of the geometry, and each layer waited for, and could be
rejected by, unrelated tasks in a shared pool.

Do the interleaving and cloning inside the queued task, and await that task
directly. Copies are now limited by the number of active pool tasks, and
each layer resolves with its own triangulation.

Closes geoarrow#225
Use a real threads Pool with an in-process worker that clones, rehydrates,
and triangulates the payload like the earcut worker. For Polygon and
MultiPolygon data with interleaved and separated coordinates, check that:

- while the only worker is busy, neither getInterleavedPolygon nor
  preparePostMessage has run
- once the worker is released, each runs as expected and the layer gets the
  correct triangulation
- a layer resolves with its own result while another pool task is running

Pools are drained after each test.
@jsve

jsve commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

This overlaps with #232, which cancels queued triangulation tasks when a layer is "finalized". As mentioned in that PR body, they both complement eachother. There is a plan for how to handle if #230 merges first, and this would probably be my recommended order.

If PR #232 is already merged, rebase #230 on top of main and keep geometry preparation inside the callback, but pass that callback through _queueEarcutTask() instead of calling pool.queue() directly. Keep the cancellation handling and try/finally metrics cleanup from #232.

Don't trust git conflict markers, check the surrounding code as well. Git can retain a direct pool.queue() call outside the conflict markers.

If maintainers think both issues these PRs address should be fixed, and this merge-order makes reviewing difficult, I'm happy to combine the PRs into one. When i started out, i thought there would be a cleaner separation between them, but implementations were slightly more entangled than i thought.

@kylebarron
kylebarron merged commit 7eaad3b into geoarrow:main Sep 25, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prepare geometry inside the queued earcut task

2 participants