Edit this page

NA-MIC Project Weeks

Back to Projects List

PolySeg in the browser

Key Investigators

Project Description

Various data structures are available to represent segmentation results. Unfortunately none of them are optimal for storage, analysis, and real-time visualization at the same time, so a trade-off is typically made to choose one that is most suitable for the main purpose of the application.

PolySeg is a library for polymorph segmentation representation for medical image computing. PolySeg is integrated within the 3D Slicer open-source medical image analysis and visualization platform. Automatic conversion between various different segment representations (Contours, Ribbon, Surface, Binary Labelmap, Fractional Labelmap) is driven by a conversion graph.

As segmentation tools become available for the web browser through toolkits such as Cornerstone Tools and VTK.js, demand is increasing for the same type of polymorphic segmentation representation in web applications. The goal of this project is to understand how the functionality in PolySeg can be most easily exploited inside the browser.

Objective

  1. Identify whether or not it is possible to cross-compile PolySeg (and its VTK dependencies) to WebAssembly.
  2. Support automatic conversion between two segment representations (e.g. contour and binary labelmap) in the browser as a proof-of-concept.
  3. Create examples using Cornerstone Tools and VTK.js which show how to create, convert, and display segments between their representations.

Approach and Plan

  1. Csaba will explain how PolySeg works and what its dependencies are.
  2. Investigators will discuss what the minimum viable product is for using PolySeg in the browser. e.g. Most end users will only need to convert between contours and binary labelmaps.
  3. We will investigate how PolySeg & VTK could be cross-compiled for use inside the browser. We will also explore whether or not the functionality can be obtained through a port of PolySeg to JavaScript which depends on VTK.js.

Progress and Next Steps

Progress

  1. Learned how to use itk-js to build combined ITK/VTK pipelines which can run in the browser.
  2. Added PolySeg as a build step in a fork of the itk-js Docker image which is used for cross-compilation (via dockcross).
  3. Used combined Docker image to build PolySeg tests to JavaScript
  4. Built PolySeg example conversion pipeline from Binary Labelmap to Closed Surface which runs in Node.js and in the Browser (WASM bundle around 6.2 MB, non-WASM JS bundle 10.2 MB)
  5. Built example page which runs the conversion pipeline on an input labelmap file and displays it with VTK.js

Pull Request to PolySeg with the code: https://github.com/PerkLab/PolySeg/pull/4

Next Steps

  1. Deal with issues around VTK’s multithreading (pthread_attr_setscope) breaking cross-compilation procedures in Emscripten. There is already an open ticket for enabling multithreading at itk-js so for now we just disabled it everywhere. The next step is to settle on the proper approach to disable pthread usage without having a fork of VTK. The built-in environment variable (CMAKE_USE_PTHREADS_INIT) did not do the trick on third-party extensions such as vtkhdf5.
  2. Deal with issues around library linking between PolySeg and VTK. If PolySeg uses BUILD_SHARED_LIBS:BOOL=ON, then everything works normally but duplicate links show up when building the pipeline (symbol multiply defined!). If BUILD_SHARED_LIBS:BOOL=OFF, certain VTK functions are not accessible within PolySeg for some reason (e.g. vtkImageStencil, vtkImageAccumulate). This is the key remaining blocker before this project could be considered a success.
  3. Once things are working properly, we should contribute back some examples to itk-js. Much of the knowledge needed for this project was present in itk-js already, but it was inside things like tests or source code.

Illustrations

Background and References