NA-MIC Project Weeks

Back to Projects List

Cinematic rendering in Slicer leveraging VTK Physically Based Rendering (PBR)

Key Investigators

• Shreeraj Jadhav, Kitware, USA
• Jiayi Xu, Kitware, USA
• Jean-Christophe Fillion-Robin Kitware, USA
• Andras Lasso (PerkLab, Queen’s University, Canada)

Project Description

• The goal of this project is to identify and evaluate Kitware’s previous efforts on cinematic rendering and how these can be leveraged in Slicer.

• Slicer internally uses VTK as its rendering engine to display the 2D and 3D viewports in its interface. Since release 9.0, VTK has more enhanced rendering capabilities such as the Physically Based Rendering (PBR) lighting model and the integration of ray tracing backends such as the Nvidia OptiX and intel’s OSPRay engine. These capabilities are already being leveraged in Paraview as discussed in the following blog posts here and here.

• Primary focus will be on VTK’s PBR capabilities and ray tracing backends (OptiX, OSPRay). Since Slicer already uses VTK rendering pipeline, and it will be significantly easier to incorporate/leverage these capabilities from within Slicer.

Objective

Approach and Plan

1. Review accessible offerings on cinematic rendering (VTK PBR, VTK backends: OSPRay and OptiX, omniverse, etc) that can be utilized inside Slicer.
2. Develop prototypes showing how these can be enabled and used in Slicer.
   • Start with surface rendering (PBR, ambient occlusion, global ilumination)
   • Investigate what capabilities can be enabled for volume rendering (perhaps OSPRay).
   • Tradeoffs: Performance vs Image Quality without degrading user experience.
   • Changes to user interface, parameters tuning, simplification.
3. Review existing Slicer modules (such as Light Module) that enhance Slicer’s rendering capabilities and evaluate how these can be included in the current effort.
4. Evaluate how integrating these in Slicer will affect other modules such as LookingGlass, OpenXR, etc.

Progress and Next Steps

Use of vtkSSAOPass class to generate ambient occlusion (AO) for volumes:

• Volume mapper cannot directly work when AO pass is enabled, need further investigations to understand how this could be done.
• Initial attempt encountered OpenGL State errors (see error dump below) trigger by this.
• To reproduce this error: use github vtk-examples branch and build/run the SSAOVolume example. ``` Generic Warning: In C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkOpenGLState.cxx, line 1069 Error glEnable/Disable1 OpenGL errors detected 0 : (1282) Invalid operation

with stack trace of at vtksys::SystemInformationImplementation::GetProgramStack in C:\D\slicer-d1\VTK\Utilities\KWSys\vtksys\SystemInformation.cxx line 3979 at vtksys::SystemInformation::GetProgramStack in C:\D\slicer-d1\VTK\Utilities\KWSys\vtksys\SystemInformation.cxx line 829 at anonymous namespace'::reportOpenGLErrors in C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkOpenGLState.cxx line 294 at vtkOpenGLState::SetEnumState in C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkOpenGLState.cxx line 1069 at vtkOpenGLState::ScopedglEnableDisable::~ScopedglEnableDisable in C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkOpenGLState.h line 299 at vtkOpenGLState::vtkglBlitFramebuffer in C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkOpenGLState.cxx line 1758 at vtkOpenGLGPUVolumeRayCastMapper::vtkInternal::CaptureDepthTexture in C:\D\slicer-d1\VTK\Rendering\VolumeOpenGL2\vtkOpenGLGPUVolumeRayCastMapper.cxx line 817 at vtkOpenGLGPUVolumeRayCastMapper::GPURender in C:\D\slicer-d1\VTK\Rendering\VolumeOpenGL2\vtkOpenGLGPUVolumeRayCastMapper.cxx line 3102 at vtkGPUVolumeRayCastMapper::Render in C:\D\slicer-d1\VTK\Rendering\Volume\vtkGPUVolumeRayCastMapper.cxx line 171 at vtkVolume::RenderVolumetricGeometry in C:\D\slicer-d1\VTK\Rendering\Core\vtkVolume.cxx line 380 at vtkProp::RenderFilteredVolumetricGeometry in C:\D\slicer-d1\VTK\Rendering\Core\vtkProp.cxx line 324 at vtkDefaultPass::RenderFilteredVolumetricGeometry in C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkDefaultPass.cxx line 167 at vtkVolumetricPass::Render in C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkVolumetricPass.cxx line 44 at vtkSequencePass::Render in C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkSequencePass.cxx line 71 at vtkCameraPass::Render in C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkCameraPass.cxx line 145 at vtkRenderStepsPass::Render in C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkRenderStepsPass.cxx line 207 at vtkSSAOPass::RenderDelegate in C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkSSAOPass.cxx line 240 at vtkSSAOPass::Render in C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkSSAOPass.cxx line 499 at vtkOpenGLRenderer::DeviceRender in C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkOpenGLRenderer.cxx line 285 at vtkRenderer::Render in C:\D\slicer-d1\VTK\Rendering\Core\vtkRenderer.cxx line 385 at vtkRendererCollection::Render in C:\D\slicer-d1\VTK\Rendering\Core\vtkRendererCollection.cxx line 53 at vtkRenderWindow::DoStereoRender in C:\D\slicer-d1\VTK\Rendering\Core\vtkRenderWindow.cxx line 347 at vtkRenderWindow::Render in C:\D\slicer-d1\VTK\Rendering\Core\vtkRenderWindow.cxx line 306 at vtkOpenGLRenderWindow::Render in C:\D\slicer-d1\VTK\Rendering\OpenGL2\vtkOpenGLRenderWindow.cxx line 2345 at main in C:\D\Shadows\Shadows.cxx line 132 ``

Adapt vtkSSAOPass to create local ambient occlusion (LAO) implementation for volumes:

• ComputeKernel() in vtkSSAOPass can be modified to adapted for LAO of volumes [Jiayi]
• Possibly, only compute an occlusion volume once (pre-compute), use this for shading in raycaster

Through discussions with the VTK team (Timothee Chabat, Mathieu Westphal), we identified another feature which could help improve shading in volume rendering:

• https://gitlab.kitware.com/vtk/vtk/-/merge_requests/9231
• an accurate ambient occlusion effect can be achieved with the current VTK master, setting the GlobalIllumationReach to 0 and setting VolumetricScatteringBlending to something >= 1.0

Image Courtesy: Gaspard Thevenon

Illustrations

Background and References

Related modules:

• Models: In recent Slicer versions, PBR interpolation can be selected and PBR material properties can be edited.
• Lights (in Sandbox extension): it can configure lighting, PBR, image based lighting, ambient shading (SSAO)

Slicer Discourse References:

1. https://discourse.slicer.org/t/how-to-perform-3d-cinematic-rendering/7313
2. https://discourse.slicer.org/t/is-there-interest-in-higher-quality-rendering-for-slicer/6862/5
3. https://discourse.slicer.org/t/2021-01-19-hangout/15585/2

VTK References:

1. VTK PBR https://www.kitware.com/vtk-pbr/
2. PBR integration in Paraview https://www.kitware.com/physically-based-rendering-improvements-in-paraview/
3. Related merge request for VTK https://gitlab.kitware.com/vtk/vtk/-/merge_requests/5584