Examples: Add UltraHDRLoader

[mjurczyk]

Related issue: N/A

Description

Adds UltraHDRLoader to examples, dropping the external dependencies / WASM modules.

👉 Demo & comparison with gainmap-js and RGBELoader

Inspired by MONOGRID/gainmap-js, read through UltraHDR docs and the format turns out to be a bit too trivial to justify 3MB dependency currently required to load the files in three.js.

This implementation follows the specs and is compatible with gainmap-creator HDRs, without any external dependencies (based almost entirely on this method from the original Cpp implementation.)

• Most of the original implementation is not really needed on web, loader outlines what's supported and what's dropped.
• No additional PMREMGenerators or WebGLRenderers are created, loader follows the style of RGBELoader, with a SRGB_TO_LINEAR in-memory table added for a 4x speed-up when loading 4K textures (3x for 2K, 2x for 1K.)

Speed comparison

Image Resolution	gainmap-js	UltraHDRLoader	RGBELoader
2K	~100ms	~110ms	~160ms
4K	~190ms	~430ms	~620ms

[Mugen87]

To sum up, the idea of this PR is to change the way the engine loads Ultra HDR images. Right now, the example webgl_loader_texture_hdrjpg relies on the third-party loader HDRJPGLoader which is part of the @monogrid/gainmap-js package. With the suggested UltraHDRLoader, the engine would provide a more optimized loader without additional dependencies, right?

In this case, I would probably remove webgl_loader_texture_hdrjpg as an example since we really want to redirect devs to UltraHDRLoader if they want to load UltraHDR images.

[mjurczyk]

With the suggested UltraHDRLoader, the engine would provide a more optimized loader without additional dependencies, right?

Yes, the only regression compared to gainmap-js should be lack of WEBP+JSON support, which was a custom out-of-spec implementation.
Can add it to this loader later too, just wanted to keep the initial PR close to spec and compact - is easier to review.

[...] I would probably remove webgl_loader_texture_hdrjpg [...]

Done. Gainmap converter is linked in UltraHDRLoader example, credit to MONOGRID also given 🙇‍♂️

[Mugen87]

Unfortunately, I see a performance regression with the new loader (tested with Chrome 126.0.6478.127 and MacOS 14.5). I use the following to measure times:

const loader = new HDRJPGLoader( renderer );

console.time( 'hdr' );
const hdrJpg = await loader.loadAsync( 'textures/equirectangular/spruit_sunrise_4k.hdr.jpg' );
console.timeEnd( 'hdr' );

And in your example:

loader = new UltraHDRLoader();
loader.setDataType( THREE.FloatType );

console.time( 'hdr' );
const texture = await loader.loadAsync( 'textures/equirectangular/spruit_sunrise_4k.hdr.jpg' );
console.timeEnd( 'hdr' );

With HDRJPGLoader I see around 140ms in the browser console.

With UltraHDRLoader, it is around 620ms. The performance degradation is even noticeable without measuring since the example lags a bit at the beginning.

[mjurczyk]

True - was unrealistic on my side. Gotten way too overoptimistic and measured quickly and incorrectly - esp. that gainmap-js runs on GPU with WebGLRenderer and shaders to calculate the HDR image.

Trimmed Performance where it was viable:

Image Resolution	gainmap-js	UltraHDRLoader (before)	UltraHDRLoader (now)	RGBELoader
2K	~100ms	~400ms	~110ms	~160ms
4K	~190ms	~800ms	~430ms	~620ms

Could you confirm the numbers from third column in a spare moment? Considering UltraHDRLoader is CPU-only like RGBELoader, the results seem more fair now.

Since it's a loader, not a real-time feature, before I'd dive too deep into overoptimisations:

• Code right now's still quite readable and clean - would the 10-20ms gains from hacky browser-dependent optimisations be worth the loss to readability?
• It's viable to follow the way gainmap-js does the SDR-to-HDR conversion, creating a new WebGLRenderer and rendering to a quad. But it feels like we'd be trading 200ms of loading 4K images (2K images now load similarly quick) for a sizeable maintenance burden? Using a WebGLRenderer & custom ShaderMaterial makes the conversion dependent on GLSL and three-js core (ex. how Texture handles colorSpaces, value defaults, future WebGPU compatibility.)

(Updated PR description ✅)

[Mugen87]

It is important that certain components are developed in third-party repositories and not everything in this one. Otherwise the maintenance burdens is too high and we focus too less on the core features of the engine.

There must be a good reason for adding a component and tbh I don't see it in this case. I would rather prefer continue using a third-party loader as adding one to the repository. How the third-party loader is then developed and how complex it is internally does not really matter to use as long as the API is consistent to other loaders.

Looking at the numbers, UltraHDRLoader is still significantly slower than HDRJPGLoader with 430ms compared to 140ms on my machine with a 4K texture.

TBH, not much is speaking for an addition of this loader. Even if the performance would be on pair I see no reason for moving Ultra HDR loader to the main repository as long as https://github.com/MONOGRID/gainmap-js is actively maintained.

The final decision is up to @mrdoob though so I defer to him at this point.

[mjurczyk]

Understandable 🫡

Just to step back to my main motivation behind including a simpler version of gainmap-js UltraHDRLoader in examples/ - it'll also allow community-driven helpers like Environment from drei to default to UltraHDRs out-of-the-box, without asking people to install additional 3MB of gainmap-js peer dependencies. Giving UltraHDR the same exposure as RGBELoader now gives to HDR / EXR. It'd may be a good idea to promote UltraHDR as the default IBL format on web / mobile, due to it's kinda-insane filesize reduction.

Waiting for @mrdoob 🙌

[mrdoob]

100% agree with this!

I remember trying to push for a simpler cpu-only version but I can't find the conversation now. Maybe it was a private conversation with @elalish...

[mrdoob]

While you're at it, would you like to give it a go at doing a UltraHDRExporter too?

[Mugen87]

It'd may be a good idea to promote UltraHDR as the default IBL format on web / mobile,

Okay, that is a good point. If UltraHDRLoader should be the upcoming standard for loading HDR maps than I support the addition as well.

Hopefully the parsing performance can be improved in the future so the delay gets a bit less noticeable when loading a 4K texture.

[mrdoob]

I was just trying to replace RGBELoader with UltraHDRLoader in webgl_tonemapping and I'm seeing some data loss...
Notice how the orange tones around the sun are gone... 🤔

HDR	gainmap-js	UltraHDR

I used gainmap-creator to convert venice_sunset_1k.hdr to UltraHDR (0.95 quality).

[mjurczyk]

@mrdoob - I'll take a look and see why that may be happening - could be something with the recent optimisations or the SRB-to-Linear table. Will make a separate PR.

@kristiker - I'll check how much that'll speed up, Performance tab didn't highlight it 👀

[...] doing a UltraHDRExporter too [...]

Yes, I had code for it before doing UltraHDRLoader, but want to make sure the loader works well first.

[...] Hopefully the parsing performance can be improved in the future so the delay gets a bit less noticeable when loading a 4K texture. [...]

To match gainmap-js the loader probably need to use GPU - which is possible, for both RGBELoader and UltraHDRLoader, but probably best to wait for TSL to be ready for that. Otherwise it'll require writing GLSL by hand.

[donmccurdy]

I was just trying to replace RGBELoader with UltraHDRLoader in webgl_tonemapping and I'm seeing some data loss...

I'm wondering if the sRGB-to-Linear conversion should be moved to affect only the SDR value on line 539? Have not tested this yet but will take a look.

[mjurczyk]

The color grading issue was with the channel index not being applied to the gainmap image (gainmap loop used only the red channel because of that 🙇‍♂️ )

As for SRGB-to-Linear - according to spec, that's indeed correct. Only the SDR image should be converted and gainmap image is assumed to be linear right away. The reason for moving it to the last step is in Canvas API (here) - currently it allows only rendering colors in SRGB & Display-P3, which I think accidentally converts the gainmap from LinearSRGB to SRGB (either this step or the loading the image via <img>) 👀

[donmccurdy]

Note that the gainmap is neither sRGB nor Linear-sRGB, it is non-color data. When the gainmap is drawn to the Canvas API, I'd expect the Canvas API to either (a) observe no color space and draw the gainmap without conversion, or (b) assume the gainmap is already sRGB and (again) draw the gainmap without conversion. In either case, calling getImageData with colorSpace: 'srgb' is the right choice, as it again avoids any conversions, which would inherently be unwanted on non-color data.

Which is not to say that something in the loading process couldn't be applying an unwanted conversion ... the web platform does not really understand either Linear-sRGB or non-color identifiers well ... but I think we can avoid that if so.

[elalish]

Very glad to see UltraHDR coming into the fold! I still think the GPU is the right way to go, but that may entail some refactoring of how loading is done. Still, especially with WebGPU coming along with compute shaders, I think we'd do well to have a renderer available to offload parallel compute to in any function, not just the actual renderer. PMREMGenerator is another example of this pattern - maybe it's time we made GPU compute a first-class citizen.

[elalish]

And I'm all for reducing external dependencies, but I'm not sure where that 3MB came from - when I added gainmap-js to model-viewer, it only added a few kb.

[mrdoob]

And I'm all for reducing external dependencies, but I'm not sure where that 3MB came from - when I added gainmap-js to model-viewer, it only added a few kb.

I think they mean the size of @monogrid/gainmap-js when importing via npm on their project?

[elalish]

I believe the encoder is 99% of that size; the decoder is quite small.

[mrdoob]

Yes, that's what I meant.
It's about dev dependencies size.

[mjurczyk]

I believe the encoder is 99% of that size; the decoder is quite small.

Yes, the decoder is small but they come in a single package (incl. that 3MB of WASM.) It's not a problem for final builds, but to add UltraHDR to drei for ex., both the decoder+WASM encoder would need to be added as a single peerDependency.

[...] PMREMGenerator is another example of this pattern [...]

GPU is 100% the way to go, and can be added to the current UltraHDRLoader quite easily - but, unless I'm mistaken, the amount of webgl contexts per tab is limited right now? Using one for PMREMGenerator in core already reduces that amount by 1, and then adding new WebGLRenderers depending on which / how many Loaders user imports from examples may cause sudden Context lost errors?

[elalish]

Yeah, ideally I think we should always have a single context, or at least make that the golden path. The question is how to architect for that? One option is to pass a context to the loader, but that is perhaps less clean. Another option, like PMREM, is to do all the processing lazily and internally, so the renderer will just build the final texture when needed.