ChromaKeying works in RenderView but not in UIImageView

[florianbuerger]

Hi,

I am playing around with the chroma key filters. I have an image with a green screen background and I am trying to isolate the minion. This works well when displayed in a render view but doesn't work at all when I display the filtered image in an UIImageView or when I write the PNG to disk. Am I using it right?

let output = PictureOutput() // instance variable

private func process() {
    output.imageAvailableCallback = { image in
        NSOperationQueue.mainQueue().addOperationWithBlock {
            self.imageView.image = image
        }
    }
    picture --> filter
    filter --> renderView
    filter --> output

    picture.processImage()
}

The top view is the RenderView, the bottom on the UIImageView showing the original image, even though it should show the processed output?

[BradLarson]

Does your UIImageView support transparency? The chroma keying shader only drops the alpha channel of the resulting image in response to they keyed colors, so the RGB components are the same. If the resulting image is then placed in something that doesn't respect alpha channels, it will look like the original image.

If you instead save this image to a PNG file, does the resulting PNG file have the correct reduced alpha channel in the keyed areas?

[florianbuerger]

Yes, UIImageView respects the alpha channel. I double checked with an exported artwork from Sketch.

Writing the image to disk does produce the same image as the image view.

    private func process() {
        output.imageAvailableCallback = { image in
            NSOperationQueue.mainQueue().addOperationWithBlock {
                self.imageView.image = image
                if let imageData = UIImagePNGRepresentation(image) {
                    let dir = NSSearchPathForDirectoriesInDomains(.DocumentDirectory, .UserDomainMask, true).first!
                    let url = NSURL(fileURLWithPath: dir).URLByAppendingPathComponent("image.png")
                    do {
                        try imageData.writeToURL(url, options: NSDataWritingOptions())
                    } catch let error {
                        print(error)
                    }
                }
            }
        }
        picture --> filter
        filter --> renderView
        filter --> output

        picture.processImage(synchronously: true)
    }

I've uploaded the complete project if you want to have a look.

[ggua5470]

@florianbuerger This seems not fixed yet. Any solution from you?

After chromakeying or chromakeyblend, the resulting UIImage in imageAvailableCallback() or the imageData in encodedImageAvailableCallback() with format setting to .png all lost the alpha channel, so the background is black (I used chromakeyblend to replace .green to .clear)

As @florianbuerger said, it only shows correct image with transparent background when showing the result to a RenderView (which means the chromakeying itself is working). When showing in a UIImageView, the background become black. It is similar to getting a JPEG output.

Any way to get it a UIImage with transparent background?

[florianbuerger]

I ended up using GPUImage version 1, which does not have that issue.

[ggua5470]

I end up using a small hack, still using GPUImage2. Because I know the chromakey is working (i.e. detecting the green pixels and set alpha to 0), but the resulting UIImage does not have the right alpha, So I replace the green pixels in the original image with pure-green pixels, and use color masking to remove them. This gives the "transparent" image I need to put into a UIImageView and process it further.

This hack is only good for one-time processing. For real-time processing, just use RenderView provided by GPUImage2.

Sample code below:

        //Use a ChromaKeyBlend to blend original image with a pure-green image
        let pictureInput1 = PictureInput(image:scaled)
        let filterChromakey = ChromaKeyBlend()

        //==> A trick here, use .green to replace green background, then replace it with transparent!
        //1X1 image to blend in (no need to create image of same size)
        let rect = CGRect(origin: .zero, size: CGSize(width: 1, height: 1))  //scaled.size
        UIGraphicsBeginImageContextWithOptions(rect.size, false, 0.0)
        UIColor.green.setFill()
        UIRectFill(rect)
        let greenImage = UIGraphicsGetImageFromCurrentImageContext()
        UIGraphicsEndImageContext()
        let pictureInput2 = PictureInput(image:greenImage!)
        
        pictureInput1.addTarget(filterChromakey)
        pictureInput1.processImage()
        pictureInput2.addTarget(filterChromakey)
        pictureInput2.processImage()
        
        let output = PictureOutput()
        output.imageAvailableCallback = {image in

            //replace green color to transparent
            let rawImageRef: CGImage = image.cgImage!
            let colorMasking: [CGFloat] = [0, 5, 250, 255, 0, 5]
            UIGraphicsBeginImageContextWithOptions(image.size, false, 0)
            let maskedImageRef=rawImageRef.copy(maskingColorComponents: colorMasking)
            UIGraphicsGetCurrentContext()?.translateBy(x: 0.0, y: image.size.height)
            UIGraphicsGetCurrentContext()?.scaleBy(x: 1.0, y: -1.0)
            UIGraphicsGetCurrentContext()?.draw(maskedImageRef!, in: CGRect(x: 0.0,y: 0.0,width:image.size.width,height: image.size.height))
            
            result = UIGraphicsGetImageFromCurrentImageContext()
            UIGraphicsEndImageContext();
    
        }
    
        pictureInput1 --> filterChromakey
        pictureInput2 --> filterChromakey --> output
        pictureInput1.processImage(synchronously: true)
        pictureInput2.processImage(synchronously: true)`

[allenkiehl]

Is there any solution to this? I am having same issue filters fine on live view but saved image not filtered.

[geoffhufford]

I posted a similar question on Stack Overflow prior to finding this issue discussion. https://stackoverflow.com/questions/52952776/gpuimage-chroma-key-export-png-file-with-alpha

I am going to try @ggua5470 workaround above, but hope someone may know the fix.

[allenkiehl]

@geoffhufford did the above solution work for you? I tried with no success after using thresholdSensitivity and smoothing does not give desired result. Filtering live works perfect though.

[allenkiehl]

@ggua5470

I end up using a small hack, still using GPUImage2. Because I know the chromakey is working (i.e. detecting the green pixels and set alpha to 0), but the resulting UIImage does not have the right alpha, So I replace the green pixels in the original image with pure-green pixels, and use color masking to remove them. This gives the "transparent" image I need to put into a UIImageView and process it further.

This hack is only good for one-time processing. For real-time processing, just use RenderView provided by GPUImage2.

Sample code below:

        //Use a ChromaKeyBlend to blend original image with a pure-green image
        let pictureInput1 = PictureInput(image:scaled)
        let filterChromakey = ChromaKeyBlend()

        //==> A trick here, use .green to replace green background, then replace it with transparent!
        //1X1 image to blend in (no need to create image of same size)
        let rect = CGRect(origin: .zero, size: CGSize(width: 1, height: 1))  //scaled.size
        UIGraphicsBeginImageContextWithOptions(rect.size, false, 0.0)
        UIColor.green.setFill()
        UIRectFill(rect)
        let greenImage = UIGraphicsGetImageFromCurrentImageContext()
        UIGraphicsEndImageContext()
        let pictureInput2 = PictureInput(image:greenImage!)
        
        pictureInput1.addTarget(filterChromakey)
        pictureInput1.processImage()
        pictureInput2.addTarget(filterChromakey)
        pictureInput2.processImage()
        
        let output = PictureOutput()
        output.imageAvailableCallback = {image in

            //replace green color to transparent
            let rawImageRef: CGImage = image.cgImage!
            let colorMasking: [CGFloat] = [0, 5, 250, 255, 0, 5]
            UIGraphicsBeginImageContextWithOptions(image.size, false, 0)
            let maskedImageRef=rawImageRef.copy(maskingColorComponents: colorMasking)
            UIGraphicsGetCurrentContext()?.translateBy(x: 0.0, y: image.size.height)
            UIGraphicsGetCurrentContext()?.scaleBy(x: 1.0, y: -1.0)
            UIGraphicsGetCurrentContext()?.draw(maskedImageRef!, in: CGRect(x: 0.0,y: 0.0,width:image.size.width,height: image.size.height))
            
            result = UIGraphicsGetImageFromCurrentImageContext()
            UIGraphicsEndImageContext();
    
        }
    
        pictureInput1 --> filterChromakey
        pictureInput2 --> filterChromakey --> output
        pictureInput1.processImage(synchronously: true)
        pictureInput2.processImage(synchronously: true)`

DId you get this to work with smoothing and thresholdSensitivity?

[sonvuhwg]

Have you resolve the problem with UIIMageView? @florianbuerger

[florianbuerger]

Have you resolve the problem with UIIMageView? @florianbuerger

I did not look into further when working on this project. Seems like I reverted to using GPUImage 1 as mentioned here.