Repository navigation
Conversation
This is using the recently added GaussianSplat support for three.js and integrates PLY, SPZ and GLTF loaders. Longer term we probably won't need SPZ, but for now this will be used to flesh out the glTF GS support and SPZ will be useful as a compressed quality comparison point. Unlike the other demos, we use WebGPU; WebGL has more limitations wrt GS rendering in three.js.
This is a single-header MIT/public domain library that can load PLY files; gltfpack will use it to load Gaussian Splat PLYs. Source: https://github.com/maluoi/header-libs
If given a PLY file as an input, gltfpack will now parse the file assuming a Gaussian Splat input. The result is a mesh that is using KHR_gs conventions for the attribute names, and preserves the SH data. Note that by itself this does not lead to a coherent GS output; we are not using the extension, and the streams mostly use _custom attribute that gltfpack encoding assumes to be an integer. This is also not fully tested because we can't produce a full scene output yet.
With this change we fully support the (uncompressed) output of the gaussian splats parsed from the PLY; each attribute writes a set of floats according to the attribute type, and the mesh also gets the required KHR_gaussian_splatting extension. For now this only works when -noq is specified; also, PLY parsing requires fixup for coordinate system and non-linear data transformations to actually render properly, which was impossible to test before but now is obviously wrong and will be fixed shortly. The extra faux node was not used; it's probably simpler to just keep meshes unattached for now.
This fixes a few bugs and actually converts the GS data. Almost every attribute is stored differently in PLY vs glTF: - Quaternions are stored as wxyz in PLY and xyzw in glTF - Scales are stored as log-scale in PLY and linear in glTF - Opacity is stored as pre-sigmoid-activation in PLY... - SH coefficients are stored in channel-major order (all R, G, B) - PLY files usually use a different (non-standard) coordinate space Miraculously, SH0 (DC) is stored as is... not that this is a good thing, since renderers usually transform it to a much easier to quantize color space, but we're limited by glTF spec in how the data is represented.
Since we can't use cgltf_attribute_type to represent the kind of data a stream stores, we add int splats to store the enum (for now defined in splats.cpp to contain the logic). This mostly replaces the string comparisons, but more importantly allows us to reuse the existing per-attribute statistics. As a side-effect, this also splits the different attribute types from each other as far as buffer views are concerned.
- Add omitted byte counts for splat buffers to stats. - Fix accounting for SH1-3 costs; these require multiple vectors per degree, and it's more intuitive to specify the cost per degree - Fix emission of KHR_mesh_quantization; also covers scenes with no meshes. - Automatically enable pos_float when processing PLY files; for now this affects the entire scene, but it's easier than supporting a mesh mix. - Reduce the use of STL when parsing PLYs which makes debug builds faster.
We are currently limited to the formats that glTF supports; this significantly limits quantization options, as only rotation and opacity can be quantized using normalized integers in practice. Technically, position and scale permit the use of quantization using the node matrix to dequantize; however, they use the same scale which usually does not work well as the range of scales and positions is completely different. For now, we leave opacity unquantized, and use exponential filters for all attribute types other than rotation, which is using normalized integers and quaternion encoding. The bit counts are using existing -vX settings, although this is not very intuitive. Using -cc also reduces bit counts for successive SH dimensions to reduce the cost.
Instead of relying on the input order, we now sort splats by their position using Morton order. This often does not change the results as input splats *are* often spatially local, but this is a good safeguard for input data, and this pipeline can filter splats in the future too.
When using -cc, we now encode SH1-3 using normalized bytes; each VEC3 is stored as 4 bytes (3 bytes rounded to a 4b boundary). This is technically outside of glTF specification which mandates that VEC3 for SH data is always floats. However, SH1-3 range is exactly [-1,1], so this is really a significant spec omission that unnecessarily inflates memory requirements and file sizes. If there is a separate extension at some point, like KHR_gaussian_splatting_quantization, we will enable that. To wit, three.js loader just works with these files, as it uses generic infrastructure to decode input attributes that works for int8 inputs. It does re-quantize with incorrect scale and rounding but that is something we could fix separately. As part of this we need to encode SNORM values with an arbitrary bit count, which encodeSnorm was previously unprepared for; we could zero out the low bits but that introduces extra bias; we could round the 8-bit value to the nearest representable 4/5-bit value, but the double rounding introduces extra error. Scaling by (range-1)/(range-step) means that we get properly rounded values in one go.
Instead of always retaining the input SH dimensions, we can use -gsd to specify the highest dimension we want. The extra dimensions will be truncated; so -gsd 0 returns a SH0-only 3DGS file. For now we simply remove the extras during processing; it should be possible to fuse them into earlier SH bands instead, but that will be a future enhancement.
We now default to SH dim 2 (-gsd 3 can be used to get the 3rd degree back). There are some inefficiencies at encoding higher SH dimensions, and the more practical default is likely 1 or 2, so we'll pick 2 and allow to go up or down as needed. At -cc, we don't really need 9-bit quaternion components; 8 bit is fine with quaternion filter. Also simplify and re-align configurations for SH bands; we drop an extra bit for SH1/2, and change -c to use one more bit for each SH band vs -cc. This can be adjusted further since the base (-vc) is configurable separately.
When -cc is specified, we additionally filter splats that are close to the quantization threshold; this is a little bit more aggressive than the quantization technically allows, but the filtering is imperceptible in practice. This has no effect on splat files that were already processed, but can remove many redundant splats from trained files.
SH data is quite noisy, and channels often flip between positive and negative values between the neighboring values. We are using Exp filter to store an unusually low width mantissa. The interaction between these is suboptimal due to sign extension. Specifically, flipping signs between noisy low-bit-width data results in the sign extension propagating into the top bytes of the stored 24-bit mantissa. These become 0 or 0xff depending on the original sign; the resulting deltas (+-1) need to be encoded, which takes space. Instead, for this data it's better to left-align the mantissa: eg for 8-bit mantissas, multiply them by 65536, subtract 16 from the exponent. This means that the low two bytes of the word are always zero, which is free from the encoding perspective. Additionally, for SH0 some files have well-behaved ranges, but some had outliers that pushed the shared component range too high. This resulted in uneven SH0 quality, which is a problem because SH0 is the most significant channel from the quality perspective. Ordinarily we'd work around this using ClampedZero mode... but SH0 data typically has a useful range of [-2, 2], and ClampedZero results in much worse compression as the exponent flips between 0 and 1. For now we special case a custom exponent encoding for SH to work around these issues: we still use ClampedZero as the underlying encoder, but adjust the range for SH0 to [-2, 2] (divide by 2 before encode, and shift exponent by 1 after encode); and, for all SH components, we left-align the mantissa by whole bytes to ensure the least significant bytes are zero. The left alignment by itself decodes exactly the same as right-aligned data, just compresses better; the SH0 adjustment changes the precision in practice, so this required adjusting the SH0 bit count to match.
Quaternion encoding at 8 bits, which is the default for -cc, runs into a similar problem to exponent filter: high byte may oscillate between positive and negative sign extension, and require extra bits to encode. This is less severe vs the exponent case, but still ends up losing a few bits. For now, we patch this by left-aligning the components and leaving the low byte at zero, but this uses a slightly out-of-spec encoding for .w, which instead of encoding 1.0 at K=16, encodes 1.0 at K=8 shifted by 8. We can't simply encode 1.0 at K=16 because this increases the dequantization error too much. In the future this could be solved with a more first class support for this during quat encoding using an adjusted quantizer, but this problem is a bit academic as in practice .w is used to recover the scaling factor as is.
We use the unicorn stuffy asset from Babylon assets; to avoid having to check out the entire repository, we use a sparse checkout limited to just this one file.
Add documentation for Gaussian Splatting, including the options we provide and current limitations.
- Fix interaction with -cf flag; we use filters so -cf can't work - Fix interaction with out of range bit counts - Fix help text around file extensions and document -gsd - Reserve space for writeSplatStreamExpSH to reduce copying - Minor assert/conditional cleanup
The output is *not* final; in particular it's quite likely that the quality tuning will need to be reworked with more explicit controls, and output in -cc mode may require extra extensions in the future.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Given the input .ply file with 3DGS data, gltfpack will now produce an output glTF file that uses
KHR_gaussian_splattingextension.This will load the PLY using excellent micro_ply.h and convert the data according to the extension spec; notably, input coordinate space is assumed to be COLMAP (-X right -Y up -Z forward), which is converted to glTF coordinate space; input SH data is assumed to be trained against sRGB.
As it stands,
KHR_gaussian_splattingis limited in terms of compression options that it supports; by default the output GLB file will be large, on par with the input PLY binary if all SH degrees are preserved. However:-gsd Kwhere K is the last SH degree that should be kept. It defaults to 2.-c/-cc.-ccis used; this mostly affects trained PLY inputsThe bulk of this work is compression exploration. This is not using any not-yet-specified 3DGS extensions, and I do not plan on supporting them. Instead, this relies on existing standardized
KHR_meshopt_compressionextension, which is orthogonal to 3DGS, and compresses most streams with the vertex codec plus various filters. In other words, this PR is doing what it can do without using new compression extensions.When
-cis used, you get a fully spec-compliant file, and per-component precision that is fairly high so that the output is generally perceptually lossless if all SHs are preserved; so-c -gsd 3gives an almost perfect output at a moderate size.-calone drops SH3 which carries perceptual loss on scenes I tested but is still quite usable, and-gsd 2felt like a better default to avoid large output files. This default might change, especially if glTF allowsint8SHn...When
-ccis used, component precision is tightened a bit further to approximately match SPZ settings. In addition, the output files in this mode currently violate the spec in relatively benign ways, by encoding SHn data as normalized integers. three.js implementation of glTF GS just works with this with no further changes; other renderers may or may not support it. This is experimental, and I'm going to try to figure out how to best make this mode spec-compliant later; files in this mode might start using something likeEXT_gaussian_splatting_quantizationin the future.I've mostly tuned this using three.js renderer that applies additional quantization at load time, so I don't know how representative these are across renderers. As noted, glTF base specification has many restrictions around GS data that make it difficult to achieve a perfect balance between size & quality; I've done what I can for now. Here's size vs image difference charts on a couple assets, although the comparison against Bonsai is a bit biased because SPZ conversion does not optimize the scene well; the GLB sizes here are after external compression like gzip (so this measures
.glb.gzfile size, vs.spzwhich is internally compressed):Some compression tweaks required working around limitations with existing vertex filter encoders; in the future native meshopt encoders might be extended with some of this, I haven't decided to what extent these tweaks are GS-specific. No decoder changes are necessary to decode these files though, this just works with existing
meshopt_decoder.js.Future work:
-cis used as SH3 stream is too large when decompressed in this mode.Ecosystem work:
-cc(KHR_gaussian_splatting: SH1-3 should have retained the normalized byte/short formats KhronosGroup/glTF#2674)Different compression levels (
-cbasically identical,-ccloses some shading details):Different SH degrees below 3: