Goals of JuliaImages: Image types

[Tokazama]

There are a lot of different types of "images" that aren't necessarily going to be represented by an array. I'm specifically thinking of meshes and quaternions right now. There are other packages that support these, but would it be within the scope of JuliaImages to support something similar to ImageMeta with a mesh or quaternion type in the data field?

I understand including these could open up a lot of other image types and has the potential to create a considerable amount of work in the future if included. With this in mind it may be good to more clearly define the goals of JuliaImages.

[timholy]

Fundamentally this should be a collection of algorithms & traits, so there is nothing stopping you from adding a package that implements a compatible API, including extending methods in JuliaImages to work on them. Assuming there really is a tight connection, it would also be great to host such packages at JuliaImages.

I can imagine a mesh being "array-like" so that at least a subset of operations work. How is a quaternion related to an image? It's just a single number.

[Tokazama]

Thanks for the clarification. I may look into creating a mesh type in the near future, as it may be useful in an upcoming project.

I mention quaternions because the NIfTI standard supports a type where each voxel is a quaternion. I suppose that would be better implemented in a way similar to color channels though. I honestly haven't come across an actual case of it yet though.

[timholy]

"each voxel is a quaternion" means you have an array of quaternions, ergo, an "image." I believe many operations are specialized for AbstractArray{<:NumberLike} where NumberLike = Union{Number, AbstractGray}. That should include quaternions.

[Tokazama]

Perhaps this is worth a completely different issue, but it may be helpful to have a package dedicated to officially supported types. A similar idea was brought up here JuliaNeuroscience/NIfTI.jl#16 (comment).

I thought it may be good to bring up in light of #772, the idea being that one route to supporting GPU computing would be using AFArray from the FireArray package as the data field of an ImageMeta type. It would also give us the opportunity to officially pin down what dependencies we would like to have. Array like packages like FireArray and StaticArrays may be naturally supported but recent adventures have brought me to the realization that there's a Point type in GeometryTypes, ImageDraw, and hidden within the FixedSizeArraysWillBeRemoved module of StaticArrays. There are other less obvious instances where it seems like it would be best to find a more parsimonious solution (eg., ColorizedArray and SegmentedImage could theoretically build on the same type so that output from both are fully supported by the rest of JuliaImages API)

I've started something here which is just a few scripts I have for conveniently passing types to functions and constructing them. Perhaps it could be the modest beginnings of such a package.

[timholy]

Wow, that's a lot of types! Can I ask, specifically what is the problem you're trying to solve? Defining a bunch of types usually means you're trying to dispatch/infer something, but what specifically are you hoping to achieve?

Unrelated, but does ArrayFire bring major advantages over CuArrays? It seems nice to support native solutions that are open-source and ones we can improve as needed. (e.g., at last check ArrayFire only supported a tiny handful of kernels for image filtering)

[Tokazama]

I'm currently using it for loading images straight into certain types. I probably only need to use the ImageMetaAxis variants but started from the ground up as I was getting use to how to construct them. I'm refining a long pipeline that has steps where images are resized, normalized, and reoriented at various stages. Each stage is better suited for each Array type I've specified but has quality control steps in between, so they have to be saved and reloaded. To be honest I haven't tested to see the performance difference entailed by loading and then converting images, but I'm trying to process ~10,000 MRI images so every bit counts. It's also more convenient than typing out the full set of traits for me.

This by no means needs to be a template though. It's just a small bit of what I've been using and illustrates some of what I'm talking about. For example, the Path types are for diffusion weighted imaging tractography, but it may be general enough to use other places. Or there may even be a similar type I've been unable to track down yet within JuliaImages. Either way it would be nice to have a standard package/documentation dedicated for image types.

My understanding is that ArrayFire is open-source and is suppose to be faster than CUDA but that's pretty much the limit of my understanding. If it's going to be difficult to incorporate and is low yield then it would probably be best left out. I just thought it was a good example of how being explicit about support for certain packages and types is relevant to other active areas of interest.

[Tokazama]

I'm closing this for now since it seems that most of what I've brought up has been addressed or seems to be progressing in the linked threads.