I am working on a project in vvvv at the moment, and wondered if anybody here had any input or ideas, or if anyone knew if something like this had been done before (I had a search but couldn’t find anything)
Goal:
Take any camera, any projector. Put them in a dark room. Connect them to our vvvv box.
When a person moves into the beam of the projector they can be ‘keyed’ with other content. Like a live version of a chroma key effect. So you could project only on the actor, or only on the background, or different content on both. Ideally this could be made to work regardless of the geometry of the background, so it doesn’t have to be done against a flat screen.
This means that vvvv will have to work out the outline of the person, and how it relates to the pixels coming out of the projector.
Application:
In a theatre environment you could design lighting without worrying about where performers were. I.E you could just tell the system to make the actors red and the background green. If you had several projector/camera pairs you could even apply a texture to an actor in the same way as you might texture a 3D model.
Process:
I have two ideas on how to do this:
The camera and the projector are in the same position. vvvv is calibrated to understand what the normal relationship between the projected image and the camera image is, including the changes in contrast/brightness as the camera automatically adjusts for brightness and contrast. vvvv then compares each captured frame to its simulation of what should be projected, and abnormalities represent areas that are now interrupted by a figure standing in the beam. The edges of this area are identified and all the pixels inside are earmarked for projection of content B. The entire image with new content is fed back into the simulation so it can analyse changes in the next detection frame.
The camera is at nearly 90 degrees to the projector, and is calibrated as before but instead looks for the shadows cast by the person in front. The pixels it can no longer see are the pixels on top of the figure. I feel that this is a weaker solution as it relies on having a flat enough background to be able to capture the beam in a single image.
I’m not even sure if this is possible without specialised equipment or a highly controlled environment.
On the other hand it might have been done before and everything I need is out there already.
What do you guys think?
Issues:
There is a variable time delay between camera input and projector output depending on CPU load. Makes it difficult to predict when the camera will see the pixels that have been sent.
At this stage I also have no idea about how to translate the data from the ‘abnormal pixels’ into a closed figure.
Any and all feedback is appreciated.
Hoping to have something working by June this year.
Cheers
-Derformer
Hi Derformer, I have had similar thought but have not had the time to try and realise this yet! Looking into it you should be able to utilise the Background Subtraction shader to take a Snapshot of the empty space and then using some lever patching reproject onto the “Changed pixels”. In theory its relatively straight forward but as always its not untill you get the gear to try it when you find the quirks!
Jannis has done something similar using infra red light and luma keying a thresholded image, if you scroll to the bottom of his pages theres some screenshots and his personal web site used to have information about it too (not his wiki page though I think?)
Which illustrates pretty well what I’m trying to achieve, although using a different method.
Jannis used two projectors, a front one for the bodies and a rear for the screen. The actors wore white jumpsuits so they would appear on the infrared camera. However this means the faces aren’t lit.
I want to use just one projector, and not use infra-red if possible. This way there are no restrictions on what the set is as you don’t have to rear-project on it. There are no restrictions on costumes. And other infrared sources won’t cause problems (Eg. other theatre lights)
The dream is that you could set two projector/camera sets opposite each other, similar to boom (side) lighting for a ballet, and they would project only on the dancer in the middle.
Hmmm… seems like I just have to make it complicated.
^^^ I could be wrong about the white jumpsuits being seen on the infrared camera. It could be that the infra-red floods the screen and the actors appear as negative space in front of it.
depending on your desired quality it will be huge project.
Some thoughts by me:
-What happens when actors overlap - in camera sight or in projector beam? How to calculate missing information? How to beam actor-shaded areas?
-How to bridge the framedelay? Especially if actors dance/move fast.
-How to handle (fast)-changing lighting situations?
-How to identify an actor - what happens if two overlap?
-Whats about hardware? How far a cameras, how many pixels resolution do you get and is it sufficent for projection then. In other words - how big (in mm) are projected pixels in real and a filmed pixel by camera. How fast are cams?
Here is a project done by Art+Com some years ago, might be interesting for you.
I suggest to go into infrared-lightend background as you have less troubles with shadows then. Even a clever position for cams and projectors can help a lot.
And if you get somehow information about the light situation (by osc/midi/dmx) then you could, in theory, calculate the light out of the image by remapping it.
Those are good questions. And I can’t answer them all yet.
Regarding multiple actors/occlusion:
Not too concerned at this stage, just want to get an A area that is unchanged and a B area that is changed. In the future I might look at a 3d model of actor positions from multiple camera information.
Regarding Framedelay:
My biggest concern. Hopefully VVVVers can help me write fast code!
I think I am going to need to learn shader language. Current delay with capture card and no load is 3 frames. If I can keep it that low should be reasonable.
Regarding other lighting:
At this stage I don’t know.
Regarding camera/projector hardware
I have accepted it will be impossible to get a perfect 1:1 ratio of camera pixels to projector pixels. I will have to work out some clever maths to get around this.
The Art+Com link is brilliant. Great video. Makes me optimistic given they did it with 1996 hardware.
