Pages

Sunday, 10 March 2013

Chroma Keying (transparent background) with Vuforia AR Video Playback

It is possible to make augmented reality video playback with the Vuforia platform just that little bit more realistic by chroma keying the video. Chroma keying, aka the green screen effect, is an old trick used in the television industry of putting a solid colour background behind the person being filmed so they can replace that one colour in post-production with a completely different scene. Obviously this requires the person in the foreground to avoid wearing anything close to that colour otherwise they'll appear patchy/transparent in places.

I managed to come up with a basic way to implement this with the video playback functionality offered in the Vuforia SDK. Here's a snippet of the OpenGL ES shader code that I used:

static const char videoPlaybackFragmentShader[] =
  "#extension GL_OES_EGL_image_external : require \n"
  "precision mediump float; \n"
  "uniform samplerExternalOES texSamplerOES; \n"
  "   varying vec2 texCoord;\n"
  "   varying vec2 texdim0;\n"
  "   void main()\n\n"
  "   {\n"
  "       vec3 keying_color = vec3(%f, %f, %f);\n"
  "       float thresh = 0.8; // [0, 1.732]\n"
  "       float slope = 0.2; // [0, 1]\n"
  "       vec3 input_color = texture2D(texSamplerOES, texCoord).rgb;\n"
  "       float d = abs(length(abs(keying_color.rgb - input_color.rgb)));\n"
  "       float edge0 = thresh * (1.0 - slope);\n"
  "       float alpha = smoothstep(edge0, thresh, d);\n"
  "       gl_FragColor = vec4(input_color, alpha);\n"
  "   }";
UPDATE: I forgot to add I also needed to run these commands in order for the fragment shader above to work:
glDepthFunc(GL_LEQUAL);
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
UPDATE #2: 2013/04/04:
Some people were reporting they had problems with their video content turning black. Re-reading this article I realised I'd missed out the vertex shader code as well so here it is. If I recall rightly, the variable "texCoord" that is used in the fragment shader must have the same name as in the vertex shader. Using the sample Video Playback app with the Vuforia SDK, if there are any problems with compiling your shader source code then there'll be some errors printed in LogCat that you can use to help debug.
static const char* videoPlaybackVertexShader =
    "attribute vec4 vertexPosition; \n"
    "attribute vec4 vertexNormal; \n"
    "attribute vec2 vertexTexCoord; \n"
    "varying vec2 texCoord; \n"
    "varying vec4 normal; \n"
    "uniform mat4 modelViewProjectionMatrix; \n"
    "void main() \n"
    "{ \n"
    "   gl_Position = modelViewProjectionMatrix * vertexPosition; \n"
    "   normal = vertexNormal; \n"
    "   texCoord = vertexTexCoord; \n"
    "} \n";
In the VideoPlayback sample code, then in VideoPlayback.cpp, inside the renderFrame() function I inserted these 3 lines above all the calls to SampleUtils::scalePoseMatrix. For those of you who have studied the source code of the Video Playback project that can be downloaded with the Vuforia SDK you can see this is a modified version the source code for the fragment shader for the video frames. What it does is, for any colour that is close to the specified colour that is processed by the fragment shader (any pixel to render on the screen from rasterization), that colour will be made transparent by modifying its alpha channel. The same code can be used for the keyframe's fragment shader too. Notice I have 3 %fs. These, in order, are the RGB values of the colour to chroma key - i.e. to make transparent. I used sprintf to replace the string with actual float values. Note the range of these numbers is 0.0-1.0. They're float numbers, not the integer range you often see of 0-255.
Unfortunately I don't have a project I can share this time as the source code I originally implemented this in is mixed in with other code I'm not able to release publicly.

To be a little more adventurous and dynamic you could consider implementing a function to dynamically change the value of the colour that is made transparent on-the-fly using a colour picker dialog or some other input method. Other input methods could be a text input field for the 0x RGB value or a different kind of colour picker like the ones often found in paint programs where the user touches a part of the screen and the colour value of that pixel is read in as the input. This would be handy so users could just touch the area of the background they want to remove and it would automatically detect the colour.

40 comments:

  1. Splendid code you have there..
    But unfortunately it isn't working for me.. The video either becomes black or stays the same..

    ReplyDelete
    Replies
    1. Hi. How are you setting the value of keying_color? Are you using sprintf? If so try hardcoding a value, for example white, so you'll have:
      " vec3 keying_color = vec3(1.0, 1.0, 1.0);\n"
      Most videos will have some degree of white in them so if this works then you'll have patchy transparent areas.

      Oh! I just recalled, I forgot I also had to run some extra gl* commands. I'll update the blog article so read above! Sorry about that ;-)

      Delete
    2. hi soliax,
      thanks a lot for your code, its works well,
      but my one prblm is that if i set vec3 keying_color = vec3(0.21, 0.71, 0.29);
      ///green
      then it also make transparent the black shadows in video,
      what could be reason for that ? plz help..

      Delete
    3. i had success with vec3(0.0, 1.0, 0.0);

      Delete
    4. This comment has been removed by the author.

      Delete
  2. This comment has been removed by the author.

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. great work sollax, appreciate it :)

    ReplyDelete
  5. This is working correctly for me but, when I press "play" on the video, the background (camera stream) and the part of video that I'm filtering (all that have green color) becomes black...do you have any idea about how to solve it ?

    Thanks in advance

    ReplyDelete
    Replies
    1. I got this problem too. any way to solve it?

      Delete
  6. I added UPDATE #2. I'm not sure if this will solve your problems but compare the vertex shader source codes and also check LogCat for any compile time problems OpenGL might be having trying to load your shaders.

    ReplyDelete
  7. Hi, your program is great. I managed it work now. sorry, I am very new to GL and Android.

    the tricks to Danno's problem, I got it work after I add some "cleaning" code:

    glDepthFunc(GL_LESS);
    glDisable(GL_BLEND);

    so after glEnable(GL_BLEND), you must disable it. or you will see anything else is black

    ReplyDelete
    Replies
    1. Worked perfectly! After line QCAR::Renderer::getInstance().end();, at the end of the function Java_com_qualcomm_QCARSamples_VideoPlayback_VideoPlaybackRenderer_renderFrame

      Thnk you so much you and Sollax, great code.

      Delete
  8. Thanks alot for your reply, I'm going to test in the next days your solution and will give you a feedback.

    ReplyDelete
  9. I've just added:

    glDepthFunc(GL_LESS);
    glDisable(GL_BLEND);

    at the end of "renderFrameQCAR" function (as last comment says), followed Sollax instructions and it's all working perfectly.

    Thanks alot!

    Note: I didn't tested yet what Sollax says in "UPDATE #2: 2013/04/04", but it's all working good anyway !

    ReplyDelete
  10. Hi, I'm trying to apply this on iOS, but I have a "glError (0x502)" runtime error, if I comment :

    #extension GL_OES_EGL_image_external : require
    and
    uniform samplerExternalOES texSamplerOES;

    the error disapear, but the video still with the color I want to remove.

    Any idea?

    Thanks!

    ReplyDelete
    Replies
    1. Hi,
      The error 0x502 is the OpenGL error for INVALID OPERATION. OES is required in order to play videos on a surface with Vuforia, so my best guess is your phone's OpenGL is too old to support this. From my experience only Android 4 and above support this feature.

      Delete
    2. Can you solve the problem?? I have the same issue and i can´t solve it!!

      Delete
  11. Hi soliax,
    I made my project in unity. and it works with shaders. but when i tried to use with vuforia it doesnt work. Video just disappeared. I googled and iam here. Honestly i didnt understood your solition.
    Could you pls share your project source? That would be so helpful for me.
    Thanks for nice tutorial, have a nice day.

    ReplyDelete
  12. Sorry but the project is a commerical product so I can't release the source code.

    ReplyDelete
  13. hi soliax, can you help me to show the code that can make my 3d object transparent ? i'm using ImageTarget sample by vuforia.

    thanks before,
    denmasmr

    ReplyDelete
  14. Hi, great article, thanks.
    Im interested in the Vuforia video playback tool, and im not a developer . do you happen know which iOS app i can download to test this feature and see how the video playback is working in the Vuforia sdk ?
    I have used other AR platforms and now would like to check this one

    Thanks
    Lior

    ReplyDelete
    Replies
    1. I don't know of any iOS apps like that sorry.

      Delete
  15. Thanks Soliax...this worked like a charm on Android.

    ReplyDelete
  16. So it works fine for me when I hard-code the keying colors. Would appreciate if you can provide some pointers on where to look for help on making it dynamic like you mentioned in the last paragraph. I understand we use %f to accept the value as input at runtime, but how do I pass this value to my VideoPlaybackFragmentShader? Any help would be appreciated.

    ReplyDelete
  17. Exactly what i want to do but seems Vuforia does not support Webcam! :(

    ReplyDelete
  18. hi soliax,
    thanks a lot for your code, its works well,
    but my one prblm is that if i set vec3 keying_color = vec3(0.21, 0.71, 0.29);
    ///green
    then it also make transparent the black shadows in video,
    what could be reason for that ? plz help..

    ReplyDelete
  19. Hi Soliax,

    Can we do frame animation using Vuforia SDK..

    ReplyDelete
  20. Hi,
    I'm facing the following problem:
    extension 'GL_OES_EGL_image_external' is not supported
    what could be reason for that ? help

    ReplyDelete
  21. Replies
    1. https://unexpectedend.wordpress.com/2014/11/04/unity3d-chroma-key-shader/

      Delete
  22. Hi Soliax!!Thanks for the code, its work for me on android device, but I dont know how integrate it on ios device. Can you help me?

    ReplyDelete
    Replies
    1. hi frenando..
      where you use this code either u use to make shader or to add this code in videoplaybackbehavior script
      ]

      Delete
  23. Hello is this still working in unity vuforia ?

    ReplyDelete
  24. Can you please write it
    for iOS too?

    ReplyDelete
  25. Please share the shader file for iOS too,i have implemented the same using the above code,but i get gl error

    ReplyDelete
  26. where to use this is code in videoplaybackbehaviour or to create/make a shader and use its material.
    plz tell

    ReplyDelete
  27. compartís el código modificado a gabriel.gar@hotmail.com.ar.

    saludos

    ReplyDelete