How to handle alpha in the manual overlay "Overlay"?

Question

I play with manual-image processing (walk-the-pix) and I recreate the standard overlay mix. I look at the "Photoshop math" macros here:

http://www.nathanm.com/photoshop-blending-math/ ( See also here for a more readable version of Overlay)

Both source images have a fairly standard RGBA format (8 bits each), as well as the destination. If both images are completely opaque (alpha 1.0), the result will blend correctly, as expected:

But if my “blend” layer (top image) has transparency in it, I burst out laughing a bit at how to correctly reflect this alpha in the blending equation . I expect it to work in such a way that the transparent pixels in the blend layer do not affect the result, the opaque pixels in the blend layer overlay the mixture as usual, and the translucent pixels of the blend layer have some effect on the result.

Can someone explain to me the mixing equations or the concept behind this?

Bonus points , if you can help me do this so that the resulting image will correctly alpha alpha (which only comes into play for pixels that are not opaque in both layers, I think.)

Thanks!

// factor in blendLayerA, (1-blendLayerA) somehow? resultR = ChannelBlend_Overlay(baseLayerR, blendLayerR); resultG = ChannelBlend_Overlay(baseLayerG, blendLayerG); resultB = ChannelBlend_Overlay(baseLayerB, blendLayerB); resultA = 1.0; // also, what should this be?? 

Answer 1

Just a guess, but I would try

 resultA = 1 - (1-baseAlpha) * (1-blendAlpha) 

Answer 2

After mixing the base color and the mix color, mix the original base color and the color obtained as a result of blending using alpha blend colors:

 vec4 baseColor = ...; vec4 blendColor = ...; vec4 blendedColor = blend(baseColor, blendColor); vec4 fragmentColor = (1.0 - blendColor.a) * baseColor + blendColor.a * blendedColor; 

I use this for blending, blending an opaque base color and a blended texture that has many (semi) transparent pixels.

Answer 3

I experimented with this problem exactly until I found out that it’s best to have a base and a blending layer like with direct alpha, and then multiply only the result with basic alpha.