depth camera face gesture pose
here's a pretty much complete plugin for Intel's "Intel Perceptual Computing SDK" (here).
It works with two cameras:
Creative Interactive Gesture Camera
(the two are the same device in different housings...)
You need to install the Intel PCSDK from above URL.
Note: Currently, the Intel PCSDK (and thus this plugin) does not support
2012-11-15: Initial release. Proudly presented by MESO.
2013-01-30: Changed to beta 2.
2013-05-07: Changed to release version of PCSDK. Added support for voice recognition.
Mmmh time to invest in a new piece of tech.
soo i see it works in close situations and i see 1280*720 rgb image but what's different from kinect? or why should one invest in this one if Leap comes out (hopefully) 4 months later?
Just bought it, 2 of them! Thanks for the patch...
@microdee: the difference is the close interaction, and its quite stable for hand/finger position and orientation. (Actually, depending on how good the Leap works, this might of course be kind of redundant).
Oh, and you get head tracking with "landmark detection" (eye and mouth corners).
I attached some images. What you see there is
Works best in a standard desktop environment. Works, but not as good when camera isn't in front of you (e.g. above or somewhere else).
What is the usable range?
apparently the range is 15cm to 1 meter... but that is good for most gestural (finger/hand) interaction.
There is an activation range of 40cm. Once a hand is found inside that 40cm, it can move back to at least 1m.
My camera has just arrived. Thanks for the patch!
The patch works perfect, including both hands at once. Thanks again for the work.
Not having any joy here, the examples all work ok, but in v4 the light stays red and the plugin does put out any data, its not red, there are no error messages...
What v4 version are you all using?
I used it on vvvv_45beta28.1 and Win 7 64. Of course had to install the Intel SDK first. I had red node at first then I decided to re-install v4 and put above patch next to VVVV.exe that made it working without red node.
Thanks for the plugin, got my camera today!
But sadly, no workey: Win7 x64, SDK installed, samples work fine, 28.1. No red node, but "DS325 (Devices DS325) help.v4p" never shows enabled or info, and console gives:
00:00:01 ERR : Exception occured in TMPluginWrapperNode.Evaluate: Access violation at address 6F651B27 in module 'libpxcupipeline.DLL'. Read of address 6679B3F0
Reinstalling, moving things, made no difference.
Same result on 2nd machine with clean Win7x64 install and 28.1. I am using the Perceptual Computing SDK 2013 Beta2 (Version 5831).
That's strange. It's hit and miss now, it was working well before but now works sometimes and other times not working. I've got the vvvv_45beta28.1 folder installed in c:/Program Files(86)/vvvv_45beta28.1
I noticed it's working when I start vvvv.exe first and then click on one of the 02_PCSDK DS325.zip patches. it needs work but it's a good start.
Same error as mediadog, is this linked in the plugin meso? Where does it come from and where is it meant to live?
@catweasel: have you tried firing up vvvv.exe first and then starting the patches? Like I mentioned I'm having problems too but hit and miss it works when I start vvvv root first and the double click the DS325 patches. The images above are the results. The location I put the DS325 folder is next to vvvv.exe in C:\Program Files (x86)\vvvv_45beta28.1\DS325
I'm hopeful @herbst updates these patches soon as they are very good efforts and a good start...
Still no joy moving the DS325 folder. I tried moving the SDK DLLs into the DS325 folder as well since they are in the same folders as the example .EXEs.
I thought maybe this was a 32/64 bit issue, so I pulled out a laptop with Win7 x32, installed 28.1 and the x32 PCSDK, examples run fine, same behavior with the patches.
The only difference I did notice when running vvvv.exe first is that it takes longer to fully initialize, as if it is initializing the hardware, but I still get the same fault. When I run the patches directly, the camera can be unplugged and it acts the same (immediate fault).
@herbst: I'd be happy to help debug this. I've got an outrageous art project in mind this camera is perfect for, and I'd rather help get this working than make my own plugin.
It's very strange that it runs sometimes here. One last suggestion. Un-register the vvvv using the crack file and re-register it then start the vvvv.exe first. Perhaps the order of this counts. But I don't see how, just a random try.
@herbst: help with this please it can be potentially a very good patch, especially that DS325 is USB only, hence portable.
What is your installed version of Intel Perceptual SDK? I'm still using intel_pc_sdk_intel64_5057.msi and I haven't installed the latest one which is intel_pc_sdk_intel64_5831.msi maybe that's the problem.
I just tested the DS325 with the new vvvv_45beta29_x64 and 32bit. it worked fine but when I shut on patch the next one won't run unless I quit vvvv and start again. I managed to test all of the patches.
Ahhh, yes, that's why I listed I was using 5831 in my initial posts, I thought a PCSDK mismatch might be the problem.
Unfortunately, Intel does not offer specific downloads that I can find, just the latest which is 5831, so no way for me to try an older one. Can someone post 5057 here, purty please? Or can anyone confirm things working with 5831? Thanks folks!
SUCCESS! This plugin does require the first SDK beta, not beta2. After some thrashing, I found it here:
Uninstall the previous PCSDK, remove the PCSDK folders, install this, and it seems to work with 29 with no further thrashing.
Excellent so I shouldn't update my SDK to the next version? I'm glad it works for you now, perhaps you can improve this patch and if interesting share it. I know not too many have the DS325 camera yet, but I like the little thing, especially its portability.
is anyone else having this problem?
i installed the beta1 sdk as linked in this thread, tried vvvv and it didn't work, but neither did any of the depth related samples that installed with the SDK. the harr tracking and landmark distribution worked, but no depth demos. the camera info exe says it cannot find the camera. it looks fine in the windows device manager.
i tried it on my big beefy machine, and also my laptop. both are win7 64bit. same deal.
beta2 samples worked fine when i uninstalled beta1 and tried beta2.
i went back to beta1, same deal... no camera working in the samples, and along with that, no vvvv use either.
additionally, thank you @meso and/or @herbst for your efforts on this.
also, any chance of seeing beta2 support? because then i'd be all set :)
I'll try to upload the SDK version that worked for me, but I think it is the first one as mediadog mention. I'll post the link here shortly.
Thanks... if it is the one linked in this thread, that is the one i tried. i hope you do have a different one. i'm not sure what to do next.
i found on the intel forums that some ppl had trouble in the past unless they disabled their onboard webcam (i guess it will show up as id 0 to opencv)... it didn't help me.. but i am posting that solution here in case it can help someone else.
Here is the link to the original Intel SDK:
I hope it works for you.
thanks metrowave, i wish i had my camera with me now, but i'll be able to give it a go later today. thanks again.
The first beta works for me now too, thanks guys!
I just installed the Intel Driver (beta1) 32bit from metrowave's link,
but when I run the examples I got that:
c:\Program Files (x86)\Intel\PCSDK\bin\win32>gesture_viewer.cs.exe Failed to locate a capture module
in vvvv I get exception
Hey everyone, i found out my camera's firmware is definitely not backward compatible with beta1... so that may mean some of you will have trouble getting this vvvv patch up and running as you will be locked into beta2 as I am.
with the help of Seth Gibson (Intel) and Justin Maurer (coworker at http://Supertou.ch ), we got an app written in Cinder that is based on beta2. it simply spits out OSC for X and Y data off of every finger. I am using it now. fun stuff. I will share it as a contribution soon, as soon as it is less primitive looking. I'll also make an effort to learn how to make it a node (it currently relies on a config file for things like host and udp port, etc)
Good stuff, Nice work!
I was also thinking of OSC for sound generating making a gestural noise synth with the DS325.
my first goal is sound triggering... samples and synthesis... and I must say, this camera does not disappoint! the Kinect was never good for this because of the lag. it was ok for graphical onscreen stuff but when a drum sound drops a half second after I expect it to, it feels very very broken. the ds325's response rate is amazing! really good for music. though it gets really confused on id'ing the different fingers. it tracks them all really well but gets confused with which is which. I had hoped to use index finger to hit a hit-test area as a sound trigger, then maybe rotate 2 fingers to tweak the sound. sadly, it confuses the fingers too often for this to be useful.
Still, I will take that fancy response rate and make something useful. so I am happy.
I'll share this stuff sometime this week. on vvvv.org and git.
So everyone, updated to beta 2. Seems to be a lot faster and more stable. Currently, the download for the voice recognition/synthesis addon from beta 2 is broken, but I hope it will work again soon, so I can incorporate it.
Thanks a lot herbst. great efforts, so this new update is for the latest Intel® Perceptual Computing SDK 2013 Beta. Looking forward to testing it, have to upgrade the SDK first. I think I do have the voice recog/synthesis v.2 addon, it's 524MB. with Intel's permission I'm uploding it on 4shared and will post the link here, if you don't have it you can try that first.
Edit: here is the Intel voice addon SDK:
Most excellent! Thanks herbst! It didn't work when I installed the intel64 SDK2 version, which I thought installed both the 32 and 64 versions. So I uninstalled and installed the ia32 version and that works! Muchos gracias amigo!
intel sdk beta3 is out. any chance to get the plugins updated?
not working anymore :(
There is a new 2013 major version?
Sorry for letting you all wait - didn't have access to such a camera for quite some time and finally bought one for myself. So, here are the changes to make the release version work. And I added voice recognition support!
Please test, as I had to do quite some changes and didn't test very thoroughly.
hello thank you for the update.
I tried all the help patches,
i got working the voice recognition. At first it was showing strange letters but later started to work quite good with lots of words and phrases.
The face patch it,s the only one that got the camera to turn on the green light.
In this one i saw some activity a square in the left corner of the renderer sort of responding from the face.
The rest did not work.
I tested with last alpha 29.5
Thanks again herbst, everything works well for me using alpha29.3 except that the help patches work one at a time, meaning after closing one you have to quit vvvv before starting another one of DS325 help patches.
The voice recognition and the accelerometer effect are very good and interesting indeed. The face recognition also works.
@colorsound, just retry it but quit vvvv and restart between the patches, also the lights on DS325 is not always on the same way, but the unit works.
hi thanks for suggestions, i have tested with other computer installing a fresh driver and vvvv but still not working , maybe my device is malfunctioning. also opening close v4 for each help patch.
although camera works with the D,fusion studio computer vision from intel
did you already manage it to use the face recognition in VVVV?
I got the Sample in the SDK working, but i am not sure, how you could implement that feature in VVVV. Any Ideas on that? I want to save different Persons (give them different IDs) and want to recognize them later.
Great if somebody could help or even send a testpatch ...
Thanks in advance
is it possible to display the video feed from the RGB camera while at the same time accessing the sensor data?
@herbst, this is a very useful patch, great work!
I have the same question regarding the RGB feed from the cam; How do I access it?
It would be great to have a pin for rgb output.
Herbst said it is possible, but needs additional development. If you need the addition for a commercial project it may be an idea to fund the development. I'm hoping to have a project that needs it soon!
hi guys , is there any update or plans to update for beta33 and DepthSenseSDK-184.108.40.2067-VS2010 or with iisu?
does anyone have an offline installer for the intel-sdk? intel took them offline :(
the older version is here, if you look above here on this post, there is the link for audio SDK as well:
anonymous user login