raster image

[plops]

Hi, I would like plotting of raster images in IJulia. Gadfly supports histogram2d, which nearly does what I want.

In our research group, we currently use dipimage/Matlab http://www.diplib.org/ , which comes with a simple but useful image viewer (see attached screenshot) and it would be amazing to have similar features in IJulia.

Some of the dipimage viewer's features are (sorted by importance):

• zoom in and out with keys i/o
• the position of the mouse cursor is displayed together with the value of the pixel below (can be any datatype, even complex)
• display 3d volumes, step through them with n/p
• display either xy or xz slices of the volume
• link two displays, so that n/p steps through them simultaneously
• display 4d volumes, step through them with n/p and b/f

I'm not sure how one could implement the value display in the browser. Perhaps one can send raw binary data to javascript nowadays (network latency isn't an issue for me, lossy compression is)?

[dcjones]

I'd really like to implement at least some of this. I recently added the low level code needed to embed rasterized images in the SVG output, so combined with Images.jl, it's within reach.

Some of the fancier features you mention, like stepping through slices, might be best implemented as a layer on top of Gadfly. We have a student this summer who is going to be working on bidirectional communication between IJulia and julia, and that will hopefully enable this sort of thing: e.g. upon pressing n, IJulia will be able to tell Gadfly to draw a new slice.

[timholy]

While it doesn't leverage IJulia, you can also look at ImageView.jl, which does most or all of the things you're asking for.

[dcjones]

Yes, I forgot about ImageView! You should definitely try that out, since it's designed for the sort of things you want to do.

[plops]

I'm using ImageView. Unfortunately, most of my colleges use Windows and I would like to avoid the dependency on Tk.

[timholy]

I know what you mean. If Gtk is any easier, there is a gtk branch of ImageView. It hasn't been updated in a while, though, so it likely requires an older version of Gtk.jl.

[mokasin]

See also dcjones/Gadfly.jl/issues/133

[axsk]

I wonder what the current progress on this is, as I need to plot information related to an image (from Images.jl) on top of that image. Is this already possible with Gadfly now? If so, I can't figure out how :(

[timholy]

If your drawing needs are very simple, see the annotation framework in ImageView.

[axsk]

@timholy That would indeed be a possibility, though I couldn't view the result in IJulia anymore. Can I export the annotated picture back to an Image to finally write it to a file?

I wonder how one should implement this feature to Gadfly. What would be the GoG way of plotting/describing a raster-graphic (/image/picture/bitmap?)? I think of a "Geom.image" geometry, taking an image object from Images.jl as attribute. But what about scale, position, alpha, ...?

The following code looks interesting to me: https://github.com/dcjones/Compose.jl/blob/1ec971eb419ae4ec5c09a6f00337e64412a2f05c/src/container.jl#L464 If the context should be rastered and the backend is SVG, then it gets compiled to a PNG (draw), embedded into a Bitmap object (is object the right word here?) which then will be inlined into the SVG. (https://github.com/dcjones/Compose.jl/blob/1ec971eb419ae4ec5c09a6f00337e64412a2f05c/src/svg.jl#L909).

So technically it should not be too hard to extend Compose to plot raster-graphics given already given in the PNG format. (While writing this I realize that this probably is what dcjones meant in his first answer :)) Can I access the PNG data directly via Images.jl?

If so I think I could try adding this feature and would open a PR.

[dcjones]

So technically it should not be too hard to extend Compose to plot raster-graphics given already given in the PNG format.

Right, the code more embedding PNG data in SVG images is already there in Compose. You should be able to do this sort of thing:

using Compose

draw(SVG("output.svg", 3inch, 3inch),
     compose(context(), bitmap("image/png", readall("some-image.png"), 0, 0, 1, 1))

The other significant todo is to implement this on the cairo backend so it works for PNG, PDF, etc. Currently you'll get an error message if you try drawing that to any of those backends. Cairo does have the ability to do this though (see cairo_image_surface_create_from_png).

[axsk]

This realy works when plotting to SVG like you wrote, but I get the following error when trying that from IJulia:

Embedding bitmaps in Cairo backends (i.e. PNG, PDF, PS) is not supported.

 in draw at C:\cygwin64\home\Alex\.julia\v0.4\Compose\src\cairo_backends.jl:694
 in drawpart at C:\cygwin64\home\Alex\.julia\v0.4\Compose\src\container.jl:510
 in writemime at C:\cygwin64\home\Alex\.julia\v0.4\Compose\src\Compose.jl:191
 in base64encode at base64.jl:156

Now I am confused as I thought IJulia is using the SVGJS backend, so shouldn't it still work?

[axsk]

I tried writing the cairo backend part, but didn't get far. Unfortunately cairo_image_surface_create_from_png just takes a path to a png file, which seems to restrictive.

I then found two methods in the Cairo backend, both having their pros and cons:

internal storage	pixel array (cairo_format_t)	PNG
compression	no	yes
easy matrix plot	yes	no (via Images.jl?)
cairo method	cairo_image_surface_create_for_data	cairo_image_surface_create_from_png_stream
SVG method	? (convert to PNG via Images.jl?)	<image> element
alpha blending	Cairo: ARGB, SVG: ?	Cairo: ?, SVG: opacity tag

I think in the end both have their uses and I don't know which is superior. It would be nice to have the alpha property on both backends. I believe in the end it comes down to whether one wants to plot matrices or images. How much work would it be to add corresponding image and matrix geometries to Gadfly?

What do you think about this?

[axsk]

Assuming it was possible to embed the bitmaps into a Compose context, how would Compose/Gadfly decide which layer (i.e. the bitmap or some other plot) should be on top over the other?

[lobingera]

at all: I got this notified in my github notifications, but i don't see any update? What happened.

[grinsted]

sorry . I deleted the comment... :-)

My comment was a use case which could be taken into consideration for design choices concerning raster images. I pass it here by mail instead, but i thought it was better not to clog up the github issue with my personal wishes.

Anyway here's my use case. This is a figure i made in matlab, and now i am investigating how feasible it is to do something similar using gadfly/compose. The figure has 3 panels (and a colorbar), each panel has the following elements:

• a background image in black and white (the continents)
• a line plot of the coast line
• a textured surface with alpha blending (set fully transparent over continents) of the data in the plot.
• some text on top (the a,b,c labels).

[image: Inline image 1]

In other figures i rotate the alpha blended texture map (because the image background and the data overlay use different map-projections). Example [image: Inline image 2]

Best Aslak

On Thu, Dec 10, 2015 at 11:09 AM, Andreas Lobinger <[email protected]

wrote:

at all: I got this notified in my github notifications, but i don't see any update? What happened.

— Reply to this email directly or view it on GitHub https://github.com/dcjones/Gadfly.jl/issues/288#issuecomment-163563817.

Dr. Aslak Grinsted

Centre for Ice and Climate http://www.iceandclimate.nbi.ku.dk/, Niels Bohr Institute, University of Copenhagen Juliane Maries Vej 30 (Room 243), 2100 Copenhagen Ø, Denmark http://www.glaciology.net | twitter: @agrinsted http://twitter.com/agrinsted | phone: +45 353-20510

[lobingera]

@grinsted Unfortunatly i can't see your inline images on github.

[grinsted]

ok,... here's the first image: http://tinypic.com/view.php?pic=154wnti&s=9

On Fri, Dec 11, 2015 at 11:37 AM, Andreas Lobinger <[email protected]

wrote:

@grinsted https://github.com/grinsted Unfortunatly i can't see your inline images on github.

— Reply to this email directly or view it on GitHub https://github.com/dcjones/Gadfly.jl/issues/288#issuecomment-163905297.

Dr. Aslak Grinsted

Centre for Ice and Climate http://www.iceandclimate.nbi.ku.dk/, Niels Bohr Institute, University of Copenhagen Juliane Maries Vej 30 (Room 243), 2100 Copenhagen Ø, Denmark http://www.glaciology.net | twitter: @agrinsted http://twitter.com/agrinsted | phone: +45 353-20510

[jingpengw]

draw(SVG("output.svg", 3inch, 3inch), compose(context(), bitmap("image/png", readall("libraries.png"), 0, 0, 1, 1)))

ERROR: MethodError: `bitmap` has no method matching bitmap(::ASCIIString, ::UTF8String, ::Int64, ::Int64, ::Int64, ::Int64)
Closest candidates are:
  bitmap(::AbstractString, ::Array{UInt8,1}, ::Any, ::Any, ::Any, ::Any)
  bitmap(::AbstractString, ::Array{UInt8,1}, ::Any, ::Any, ::Any, ::Any, ::Any)

[lobingera]

@jingpengwu I think the output of readall changed in the meanwhile, so you need to convert from UTF8String to the Uint8 array.

[jingpengw]

Thanks a lot. It works draw(SVG("output.svg", 3inch, 3inch), compose(context(), bitmap("image/png", Array{UInt8}(readall("price.png")), 0, 0, 1, 1)))

[jingpengw]

I am also struggling to make a web based tool to visualize and manipulate image stacks. The Escher and Compose looks promising, but I also got stucked by the images.

As mentioned above, I can read from png image, but can not create a compose from an array!

[lobingera]

with https://github.com/dcjones/Compose.jl/issues/141?

[jingpengw]

Yep, that is exactly what I am looking for! Thanks a lot for your effort! It seems to be a building problem to merge to master?

[lobingera]

@jingpengwu, look's like it builds on release but not on nightly.

[RobBlackwell]

As of Feb 2019, I was able to display an image from Images and its histogram using the code below. Does anybody know if there is a better way these days please?

using Compose
using Gadfly
using Images
using TestImages

set_default_plot_size(8inch,4inch)

img = testimage("lighthouse")
img1 = Gray.(img)

edges, counts = imhist(img1, 1/255:1/255:1)

p1 = plot(x=0:255,y=counts,
          Guide.xticks(ticks=[0,50,100,150,200,250]),
          Guide.yticks(ticks=[0, 5000, 10000,15000,20000]),
          Guide.xlabel("Intensity"),
          Guide.ylabel("Frequency"), Geom.line);

p2 = compose(context(),
             bitmap("image/png",
                    repr("image/png",img1), 0,0,1,1));

hstack(p1, p2)

[Mattriks]

The histogram could be done like this:

p1 = plot(x=gray.(img1), Geom.histogram(density=true))

What plot layout are you trying to achieve? (see also #1169)

[RobBlackwell]

Thanks.

I have a number of graphs pertaining to an image and I’d like to display them alongside the image.

The key requirement is for a nice way to display images, ideally with axes showing pixel spatial dimensions, with ability to vary the size and aspect ratio. Something like a Gadfly version of “imshow”

I’d like to be able to line up the axes so that a pixel in the image could be cross referenced with a point on a graph.

[tlnagy]

I just ran into this problem as well. In my case, I would like to plot the centroid of a cell above it as it is moving. I defined the following custom Guide type (inspired by Rob's code):

struct BackgroundImage <: Gadfly.GuideElement
    ctx::Context
end

function BackgroundImage(img::AbstractArray{<: Colorant, 2})
    yrange, xrange = Base.axes(img)
    BackgroundImage(compose(context(),
              bitmap("image/png",
                     repr("image/png",img), first(xrange),first(yrange),last(xrange),last(yrange))))
end

function render(guide::BackgroundImage, theme::Gadfly.Theme,
                aes::Gadfly.Aesthetics)
    ctx = compose(context(), svgclass("geometry"), guide.ctx)
    return [PositionedGuide([ctx], 0, under_guide_position)]
end

I get this via SVG:

plot(x=xs, y=ys, Geom.path, Guide.BackgroundImage(view(cell1, :, :, 300)), coord)

Which is what I want, but saving to PNG or other formats doesn't work via Cairo:

julia> draw(PNG("test.png", dpi=300), p)
Embedding bitmaps in Cairo backends (i.e. PNG, PDF, PS) is not supported.

Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] draw(::Compose.Image{Compose.PNGBackend}, ::Compose.BitmapPrimitive{Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}},Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}}) at /home/tamas/.julia/packages/Compose/Opbga/src/cairo_backends.jl:773
 [3] draw(::Compose.Image{Compose.PNGBackend}, ::Compose.Form{Compose.BitmapPrimitive{Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}},Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}}}) at /home/tamas/.julia/packages/Compose/Opbga/src/cairo_backends.jl:638
 [4] drawpart(::Compose.Image{Compose.PNGBackend}, ::Compose.Context, ::Compose.IdentityTransform, ::Compose.UnitBox{Float64,Float64,Float64,Float64}, ::Measures.BoundingBox{Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}},Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}}}) at /home/tamas/.julia/packages/Compose/Opbga/src/container.jl:477
 [5] drawpart(::Compose.Image{Compose.PNGBackend}, ::Compose.Context, ::Compose.IdentityTransform, ::Compose.UnitBox{Float64,Float64,Float64,Float64}, ::Measures.BoundingBox{Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}},Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}}}) at /home/tamas/.julia/packages/Compose/Opbga/src/container.jl:509
 [6] drawpart(::Compose.Image{Compose.PNGBackend}, ::Compose.Context, ::Compose.IdentityTransform, ::Compose.UnitBox{Float64,Float64,Float64,Float64}, ::Measures.BoundingBox{Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}},Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}}}) at /home/tamas/.julia/packages/Compose/Opbga/src/container.jl:503 (repeats 2 times)
 [7] drawpart(::Compose.Image{Compose.PNGBackend}, ::Compose.Context, ::Compose.IdentityTransform, ::Compose.UnitBox{Float64,Float64,Float64,Float64}, ::Measures.BoundingBox{Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}},Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}}}) at /home/tamas/.julia/packages/Compose/Opbga/src/container.jl:509
 [8] drawpart(::Compose.Image{Compose.PNGBackend}, ::Compose.Table, ::Compose.IdentityTransform, ::Compose.UnitBox{Float64,Float64,Float64,Float64}, ::Measures.BoundingBox{Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}},Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}}}) at /home/tamas/.julia/packages/Compose/Opbga/src/container.jl:390
 [9] drawpart(::Compose.Image{Compose.PNGBackend}, ::Compose.Context, ::Compose.IdentityTransform, ::Compose.UnitBox{Float64,Float64,Float64,Float64}, ::Measures.BoundingBox{Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}},Tuple{Measures.Length{:mm,Float64},Measures.Length{:mm,Float64}}}) at /home/tamas/.julia/packages/Compose/Opbga/src/container.jl:509 (repeats 2 times)
 [10] draw(::Compose.Image{Compose.PNGBackend}, ::Compose.Context) at /home/tamas/.julia/packages/Compose/Opbga/src/container.jl:356
 [11] draw(::Compose.Image{Compose.PNGBackend}, ::Plot) at /home/tamas/.julia/dev/Gadfly/src/Gadfly.jl:864
 [12] top-level scope at In[55]:2

[Mattriks]

Yes bitmaps in Cairo backends not supported in Compose, but see GiovineItalia/Compose.jl#140, GiovineItalia/Compose.jl#141

[tlnagy]

I had even commented on the 2nd PR haha. Thanks for finding that @Mattriks. I'll see if I can get around to updating that code and see if all the issues have been resolved later this week.

An interim solution was to save as a SVG and then use ImageMagick's convert to switch to PNG, worked in a pitch.

[tbenst]

@tlnagy thanks for your code, super helpful! How did you define coords? not sure how to align

using Compose, Gadfly, Images, TestImages
 
struct BackgroundImage <: Gadfly.GuideElement
    ctx::Context
end

function BackgroundImage(img::AbstractArray{<: Colorant, 2})
    yrange, xrange = Base.axes(img)
    BackgroundImage(compose(context(),
              bitmap("image/png",
                     repr("image/png",img), first(xrange),first(yrange),last(xrange),last(yrange))))
end

function Gadfly.render(guide::BackgroundImage, theme::Gadfly.Theme,
                aes::Gadfly.Aesthetics)
    ctx = compose(context(), svgclass("geometry"), guide.ctx)
    return [Guide.PositionedGuide([ctx], 0, Guide.under_guide_position)]
end

function cartIdx2Array(x::Vector{<:CartesianIndex})
    # make tall matrix
    permutedims(reduce(hcat, collect.(Tuple.(x))), (2,1))
end

img = testimage("lighthouse")
img1 = Gray.(img)
points = cartIdx2Array(findall(img1 .> 0.95))
coords = Coord.cartesian(xmin=0, xmax=size(img1,2), ymin=-size(img,1),ymax=0)

plot(x=points[:,2], y=-points[:,1], Geom.point,BackgroundImage(img), coords)

P.S. Imagemagick failed in converting the SVG. Inkscape succeeds, but with weird artifacts inkscape -w 1024 -h 1024 input.svg --export-filename output.png

[tlnagy]

I can try and find the rest of the code used to generate that plot, but my guess is that I did Coord(fixed = true) and I set the boundaries (xmin, xmax, ymin, ymax) to match the extrema of the image. I believe I also flipped the y axis with yflip=true. Does that help?

P.S. Imagemagick failed in converting the SVG.

What errors did you get? Did you save the SVG from julia with draw(SVG("name of file.png"), p)?