Upgrade the WebGL viewer to modern Three.js + WebGL2: scope, inventory, and staged plan

[mvdoc]

Note from Matteo: the following is a possible plan by Fable on how to upgrade the WebGL viewer with modern JS/WebGL. I don't think we should do it yet, but to me it looks doable with current agentic tools. We should probably tackle this (if we want to) once we have implemented @kroq-gar78's visual tests (#672).

---

Summary

The WebGL viewer bundles Three.js r69 (September 2014; current is r180+). This issue scopes what a modernization to current Three.js + WebGL2 would take, based on a full inventory of the viewer's r69 API surface, and proposes a staged plan. It supersedes the long-dormant #46.

Verdict up front: a straight version bump is impossible, but the coupling is concentrated and well-understood. Roughly 2,600 tightly-coupled lines across 9 files; of the 57 distinct THREE.* symbols used, ~75% port mechanically. The cost lives in five interlocked mechanisms (below). The strongest technical argument for doing it: under WebGL2, two of the hairiest subsystems (16-bit index chunking, float-encoding picking) become deletable rather than portable — the end state is less code than today.

Why bother

• Every shader-adjacent change (e.g. the lighting refactor NF: refactor webgl lighting #679, the displacement refactor Refactor cortical sheet displacement into reusable shader utility #708) is written against r69 conventions that no current documentation describes.
• Open rendering bugs on Firefox (Viewer fails to load surface shader for cortex.Vertex object in Firefox #375) and Safari (Pycortex does not work on Safari #118) may well be r69 artifacts.
• WebGL2 32-bit indices eliminate the 65535-vertex draw-chunk machinery; float render targets eliminate the RGBA float-encoding picking hack.
• Reopens access to the modern Three.js ecosystem (loaders, controls, and eventually the WebGPU renderer).

Why it keeps not happening (the five hard parts)

1. geometry.offsets / 16-bit index chunking — r69 split the ~300k-vertex meshes into 65535-vertex chunks with rebased indices, and pycortex threads offsets through its own geometry math as a first-class argument: culling, normals, areas, smoothing in mriview_utils.js (_cull_flatmap_vertices, computeNormal, computeAreas, smoothVertexData, explicit rebasing at mriview_utils.js:578), with ~a dozen consumers in mriview_surface.js including live index-buffer hot-swap on flatmap cull (mriview_surface.js:516-524). Modern Three.js removed the mechanism entirely. Under WebGL2 this whole scheme is deleted, not ported.
2. r69 Phong ShaderChunk/UniformsLib lighting — shaderlib.js splices THREE.ShaderChunk["lights_phong_*"] at 9 points and merges UniformsLib["lights"] (mriview_surface.js, sliceplane.js). The modern lighting chunks have a completely different contract (BSDF/struct-based), so this must be rewritten — most likely as self-contained GLSL with our own light uniforms rather than chasing Three's internal chunk API again.
3. ShaderMaterial attributes: option (removed in r72) — used for the hand-rolled mixSurfs*/mixNorms* inflate/flatten blend and the data attributes (dataset.js:255, facepick.js, svgoverlay.js, descriptor tables in shaderlib.js:665-681, 900-911). The same attributes already exist on the BufferGeometry, so this is mostly coordinated deletion across five files. Note the viewer does not use Three's morph targets, and the surface path is already BufferGeometry — the blockers named in Upgrade to latest three.js #46 back in 2013 no longer exist.
4. Private-API picking — facepick.js:192-197 and mriview_utils.js:96-101 read pixels via the removed __webglFramebuffer internal. Modern replacement (readRenderTargetPixels, float render targets under WebGL2) is simpler than the current code.
5. LuminanceFormat → RedFormat for data textures (dataset.js:352, 402; LuminanceFormat removed in r137) — a small API change that forces a swizzle audit through every data-sampling site in shaderlib.js.

Plus a long mechanical tail: addAttribute→setAttribute (22 sites), enableScissorTest→setScissorTest, PointCloud→Points, LinePieces→LineSegments, ImageUtils.loadTexture→TextureLoader, the 3-arg renderer.render(scene, camera, target) signature (removed r103), the EventDispatcher.prototype.apply mixin idiom (12 sites), geometry.dynamic, THREE.Face3 in CTMLoader's legacy branch, and a sliceplane.js Geometry→BufferGeometry rewrite (the one file still on legacy geometry in a hot path). shaderlib.js's 1,075 lines of GLSL ES 1.00 need translation for WebGL2 (attribute/varying/gl_FragColor; the #extension GL_OES_standard_derivatives pragma becomes a compile error in GLSL ES 3.00).

Prior art

• Upgrade to latest three.js #46 — "Upgrade to latest three.js", open since 2013. The blockers named there (morph targets, index reordering) are either gone or deletable now.
• Branch origin/threejs-r84, commit 5dcc9db5 — "Tried (and failed) to upgrade to threejs r84" (2017). Its diff concentrates exactly where this inventory predicts (shaderlib.js, mriview_utils.js, CTMLoader.js, facepick.js). Read it before starting; it is a free map of the minefield.
• Two assets the 2017 attempt did not have: the headless Playwright suite (cortex/tests/test_webgl_headless.py parametrizes data types and view presets; cortex/tests/test_webgl_data.py encodes the premultiply/alpha contract) and the draft visual-regression harness in CI add visual regression tests #672.

Proposed plan

Stage 0 — dead-code cull (independent PR, zero risk, do regardless). surfgeometry.js, surfload.js, LandscapeControls.js, shadowtex.js, svgroi.js, and the broken simple.html (references a non-existent Three.js) are loaded by no live template. Removing them drops ~1,100 lines and several of the hairiest couplings (raw-VBO BufferGeometry subclass, THREE.Loader, 3 dead THREE symbols) before porting starts.

Stage 1 — regression net. Land or extend the visual-regression tests (#672) so the headless suite produces pixel-comparison baselines for: each dataview type, inflate/flatten animation endpoints, ROI/sulci overlays, picking, and the alpha/premultiply cases from test_webgl_data.py.

Stage 2 — let the shader layer settle. #708 (displacement refactor) touches the same shaderlib.js seams; land it first so the port has a stable base.

Stage 3 — the port, on a long-lived branch, one subsystem at a time, validating against the Stage 1 baselines at each step. Suggested order (safest → riskiest):

1. Mechanical renames + renderer.render/setRenderTarget + EventDispatcher mixins.
2. Picking: replace __webglFramebuffer reads with readRenderTargetPixels; move to float render targets and delete the encode_float_vec2 GLSL tricks.
3. Geometry: delete the offsets chunking (32-bit indices), switch attributes.index → setIndex, addAttribute → setAttribute, drop the ShaderMaterial attributes: option, rewrite sliceplane.js on BufferGeometry, port CTMLoader (keep its file-chunk offsets — unrelated to geometry offsets, easy to conflate).
4. Data textures: LuminanceFormat → RedFormat + swizzle audit; re-verify the premultiply contract.
5. Lighting: replace the r69 Phong chunk splices with self-contained GLSL; translate shaderlib.js to GLSL ES 3.00.
6. Color management: modern Three defaults to sRGB output (outputColorSpace) — must be configured for parity so colormapped data values survive round-trip exactly.

Explicit non-goal: an intermediate hop (r84/r1xx). It pays the migration cost twice and keeps the offsets scheme.

Risks

• Visual regressions are subtle and costly in a scientific tool — the alpha/premultiply pipeline and exact colormap value round-trip especially. Stage 1 is the mitigation; nothing merges without pixel parity.
• The lighting rewrite (Stage 3.5) is the highest-uncertainty piece and interacts with the recently refactored lighting options (uniform_illumination, topleft_lighting, specularity, bumpy_flatmap).
• static.html/htmlembed.py single-file export and the JSProxy websocket RPC must keep working; they are covered by the headless suite but worth explicit checks.

Effort estimate

A few focused weeks for someone familiar with the shader pipeline, dominated by Stages 3.3 (geometry) and 3.5 (lighting). Stages 0–1 are small, independent, and worth doing even if the port never happens.