Meshroom
Meshroom copied to clipboard
alicevision
Image masking
I'd like to suggest image masking as a feature for a future version of Meshroom. In case this would be implemented, masked areas should not be used for keypoint detection nor for depth maps and texture generation. A simple import of masks in the form of binary images would be enough, a build in editor is not necessary.
Thank you for developing this great photogrammetry software!
@robertguetzkow Can I ask how you would create your binary masks? I don't know how I would do that right now so I would want a built in editor.
You could just paint in some rough mask per image in GIMP or photoshop and save it as a PNG using the image name it applies to with a predetermined extension or suffix.
@finnbear exactly as @octupe described. Sorry been on vacation, that's why I didn't respond sooner. GIMP, Krita, Affinity Photo, Photoshop or any other image editing software that allows to use layers.
I agree that this would be a good feature.
In the meantime, a simple workaround is to apply your mask on the input pixel (RGB=0). There is no drawback in doing that. You just need to be careful to keep your image metadata (if your image editing software doesn't preserve metadata, you can transfer metadata from one image to another with exiftool).
@fabiencastan wouldn't that workaround select edges in the RGB=0 area as features, therefore possibly mismatching images when masking multiple objects/areas?
Yes, metadata needs to be preserved for focal length calculation. Exiftool is very useful if you need to manually add, edit or copy exif data. I definitely recommend it.
Yes sure but in practice it works quite fine. I have always used soft-edges on my masks.
If there was to be built-in masking, a hierarchical agglomerative clustering procedure that expands the background and foreground regions from the initial strokes seems to be the best solution.
Sorry Im a bit new. Does this entail pulling each photo into an editor and painting out what isnt necessary for the scan? I agree a bounding volume you could define once the point cloud was generated would save a lot of time .
@zfarm yes that was my idea. It has the benefit that there are already plenty image editors out there that are stable and it would greatly reduce the necessary work to integrate this feature. Integrating a mask editor into Meshroom seems overkill to me.
@robertguetzkow Of course you can apply masks to a few images or paint out unwanted areas by hand - but what about datasets with 100+ images? GMIC can extract the foreground and create reject masks - but not in bulk. I was thinking more of Image Segmentation like this to automatically identify and select the object in multiple images and create masks.
@zfarm Yes, a bounding volume/Mask (as a tool in the 3D Viewer?) for the point cloud would be nice. Maybe PCL (->github) could be useful here.
@natowi sure I can think of several more automated and sophisticated approaches, but somebody has to implement it as well. I was just being pragmatic. Having any masking at all would be an improvement. In case somebody wants to try your segmentation approach, I'd highly suggest to not rely on automated processing alone. There should always be a way to make manual adjustments to the masks.
I'd highly suggest to not rely on automated processing alone
@robertguetzkow I agree When I have some time, I´ll look for a suitable library or tool we could use for this task and think about a way to integrate it in the Meshroom workflow. (This is a good starting point)
Have a look to 3df Zephyr "Masquerade" plugin, which you can start from inside the program right after importing images at a click o f abutton: it makes you draw on first image which parts to take into account and which to ignore, and then you can propagate same mask to all images with one click. Very useful if you have fixed obstructed area on your images.
I was talking about the idea: an "image mask editor" embedded in Meshroom rather than an external program.
@jumpjack Yes, I know. That´s why I am looking for an open-source project with a compatible license and good usability/results which could be merged with Meshroom, as the main devs have enough to do (I think) with working on Meshroom core features.
I'd be interested in something that could accept an alpha channel from a png or tiff as a form of mask input.
I would be happy with importing masks from an external program (e.g. GIMP, Photoshop) as black and white images or as alpha channel from a png, tiff, tga, etc..
Example image: https://i.imgur.com/ZNrztz1g.jpg Example mask that could be a stand-alone image or an alpha channel: https://i.imgur.com/UDHvlKVg.jpg
An embedded editor is not strictly necessary (although I wouldn't object to having both features).
I wrote a simple script which masks the filtered depth maps. It performs the final masking step, using masks generated externally (using Imagemagick at the moment).
At the moment, this is just something which I run from the commandline, but I would be willing to convert it to a meshroom node if someone could give me some pointers how to do this.
I've looked into the node definition files, but some kind of short introduction or description of the process would be really helpful, otherwise it's trial and error for me.
Never mind, I figured it out from the sources.
I now have a node-based workflow for masking, see #566
Amazing! this will be helpful. Do you know if its going to be in the next build or.. is there a way to implement it?
On Tue, Sep 3, 2019 at 8:47 AM ALfuhrmann [email protected] wrote:
Never mind, I figured it out from the sources. See #566 https://github.com/alicevision/meshroom/issues/566
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/alicevision/meshroom/issues/188?email_source=notifications&email_token=ACERY54HVV4NEH3VYPKYWBDQHZMHBA5CNFSM4FPLZRDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5YCHLA#issuecomment-527442860, or mute the thread https://github.com/notifications/unsubscribe-auth/ACERY54RQ6Y3Q6DL7KPZZ7DQHZMHBANCNFSM4FPLZRDA .
-- Sam Gebhardt
I posted this on the other thread:
Here is everything you should need for testing:
Copy the "lib" folder from the ZIP in your meshroom root folder, merging it with the "lib" in there. At the moment it is only tested with Meshroom-2019.1.0-win64, but it probably works with 2019.2.0, too.
The ZIP also contains a sample Meshroom graph "Meshroom-2019.1.0-win64", which shows the necessary nodes and connections.
So you need only to drop your images in Meshroom, and put the folder where your masks are in the "DepthMaskFilter/Mask Folder" attribute. Masks are named like the source image they belong to, and have to be grayscale .PNG files. Every pixel with value 0 (black) is masked (see here).
If you try this, please post your experience over there: #566.
At the moment, this is only my solution for a specific problem I have. If other people can successfully use it, I am going to submit a pull request to meshroom. No idea if or when this is going to be in the distribution.
I found this paper: Perspective-consistent multifocus multiview 3D reconstruction of small objects. They generate their masks using two images, one in normal front lighting and one with a strong back light. This method can be useful for turntable setups.
This would work fine, but requires twice as many images to be taken.
I have tentatively implemented a "MakeMaskFilter" for meshroom, which uses background subtraction to generate masks. Additionally I look for a continuous silhouette by only filling the mask from the image corners inward, but this only works because my objects to not have holes in them.
A complete pipeline now looks like this:
When I have the time I am going to upload this here.
@ALfuhrmann It could make sense to integrate background subtraction option into https://github.com/alicevision/AliceVision/pull/715. Currently there is only one algorithm implemented for HSV keying, but the option is already there to add other strategies. Would you be interested to make such contribution?
In the PR, the node has an option grow/shrink for post-processing the mask at the end. The FeatureExtraction node has a new input "masksFolder" to retrieve features only in the selected areas. The PrepareDenseScene has also a new option "Masks Folders" to only compute the depth map in the selected areas, which reduce the computation time.
I'd have to take a look at #715 first. Since I am lazy, my nodes are python only, I would have to change to C++ for a usable implementation. I'll look into it over Christmas.
If I have enough time, I may be able to integrate it. Could you share your python script somewhere?
Sure: https://pastebin.com/rFf14QUY
This has worked fine for me, but I am sure there is lots of room for improvement.
It even has some comments(!) but message me if you need additional information