Skinned Skeleton and Kinect

Has anyone successfully used a skinned mesh in vvvv with the kinect skeleton tracking?
If you have, how did you do it! Any tips or examples, would be lovely!

So I’ve just imported astroboy into cinema4d ,and exported it to see what the differences are there, and they are many,so it looks like c4d will never export a form the vvvv can use, but has anyone had any maya exports that work?

+1 on that question.

It seems this guy has done it:

Hmm yeah from the start of last year too!

you can use maya for that not sure if newest version. i used 2008 maya
still controlling it is bit rocket sicence

+1 for this thread.

The example in the Youtube video seems work fine.

@antokhio any chance of an example patch? Last time I tried it was with lots of lookats, as the rotations didnt work as expected, it would be a very useful thing to have!

gonna dig where it buried
main problem was keeping skeleton from breaking a part from inferred joint’s

i can upload something that addresses this issue tonight. just let me do a quick bugfixing… its ust for doing the animation and i found a working model. but i still didnt find the perfect way to export, too.

yea sorry for delays, i checked it out but in general
there is nothing optimistic quite, few topics:

i cannot tell you how to set up pivots proper.
we used reseted pivots for our biped, but i can see on mesh they not reset everywhere.
A guy animator told me the biped pivots should be aligned so thay never rotate on Z.

In general we need someone who know how to rig character for cg proper, it should be maya without scripts. All i have so far is this book http://shiba.hpe.cn/jiaoyanzu/wuli/soft/Hlsl/Character-Animation-With-Direct3D.pdf
and if i’ll manage to find the guy riger/animator who does this i’ll try to ask him…

problem with hierarchy that you need to use orientation from privious point to compute next one…
If anyone can rip off fbx from mssdk example, pls post it since might shade some light on orientation of the bones.

Hey, It took me a while, sorry. But can you try it with this characters / method? animatedcharacter-(assetsdb)

I have no Kinect to test it right now. Can be that the problem is more that there are twisted rotations…

+1 for this thread! Let’s sort this out guys COME ON! We know it can be done. I have been trying to do this for 2 weeks and it’s driving me nuts!
I am now really really close to having a workflow working from knowing exactly how to rig the model to how to export in collada, to importing it with correct skeleton.

THIS IS WHAT WE NEED, we all want to do this right? We need to find a workflow for this from the actual model, how to rig it properly for collada export, what collada export to use and then the correct patch in v4. I am almost there but “not quite”. Here are some things I have learnt along the way, please CORRECT ME if any of the following information is WRONG, I am not claiming to know it all, just observations really…

  1. Anthokio, as usual, is RIGHT! not just in this thread but in the other threads. A lot of problem seems to come form the Z orientation(XYZ) of the collada model not working with the kinect skeleton AND the general Z-up thing in 3d modelling softwares. Also you’re right to point about locking z rotation for bones in 3d software, and maybe even the inheritance of bone orientation. HOWEVER, when I lock bone orientation or lock inheritance of bone orientationin 3ds max: it doesn’t make any difference to the collada model. My educated guess is this: locking orientation and its inheritance is a 3ds max feature which does NOT translate into collada, but then maybe we should be doing all this shizz in v4 itself? Anyway keep reading I will get back to this later…
  2. I am a cinema 4d guy, however I have given up using c4d for this, simply because the collada exporter does NOT work with bones, it seems, I tried everything… although I would love to be proved wrong if anyone managed to do it.
  3. 3ds MAX SHOULD work. I know what Anthokio is saying about Max having a z-up orientation as opposed to v4 using Y up, BUT it is the same in UNITY and there are ways around it, I will get to how after but first some tips on HOW TO RIG or prepare your model FOR COLLADA EXPORT (this is after literally hundreds of trials and experiments to just get the right collada export with bones into v4):

-Don’t use BIPED or any automated rigging techniques, go back to BASICS as collada is limited! use the very simple BONE tool along with the Animation -> Bone tools panel for ease of use. Create the same number of bones as the Kinect openNI skeleton: 15 bones… Here is the list of bones but keep in mind you should create them in 5 “hierachies” or “chains”:
Torso -> Neck -> Head
LeftShoulder -> LeftElbow -> LeftHand
RightShoulder -> RightElbow -> RightHand
RightHip -> RightKnee -> RightFoot
LeftHip -> LeftKnee -> LeftFoot

ROOT BONE! Very important, when you have created these 5 hierarchies, one bone must be set a ROOT BONE the top most bone, to do that either create a single new bone at feet or in middle of your model OR use TORSO bone (which method is best guys?) AND link every top bones of each chain to the ROOT BONE (so select LeftShoulder, rightShoulder, RightHip, LeftHip, and if not using torso as root bone, select torso as well and in 3ds max click on LINK(little chain icon) and drag from one of selected object onto the root bone, release and it will briefly light up to show the linking has been done).

-Export using OPENCOLLADA plug in for 3ds max, tick just about everything apart from animation (unless you want animation imported but let’s keep it simple for now, Maybe untick export tangeant thingy, doesn’t seem to make a difference.

-PIVOT ORIENTATION: I have managed to get my bones “facing the right way” in v4 when viewing the model’s skeleton using the “selectjoint” node by changing the ROOT bones PIVOT ORIENTATION (in max go to hierachy -> pivot -> affect pivot only -> rotate y axis up). Moreover if you go through each bones and painstakingly rotate pivot Y-up instead of Z-up and then export, (untick “always deform” in SKIN so that rotating joints won’t fuck up your model/skinnning, then re tick it), then when I apply a transformation to a joint of this new exported model using the “setjoint” node, for example some translate Y transform, it works the right way, as in, the joint moves UP if I increase the Translate Y, therefore proving that my bone is now correctly facing the right way.

HOWEVER I am guessing that the orientation of bones is still not correctly Y-up cos it still doesn’t work. I get the “selectJoint” visualization of my skeleton to actually work really well, it faces the right way and moves correctly acording to kinect movementm but it doesn’t “apply it properly” to the collada skeleton weirdly enough -> I am guessing it is the orientation of bone indeed that is still not right.

THEN I HAD THIS IDEA: Instead of making all these tricky changes in 3dsmax or other 3ds software to really just “hack” our collada model to actually be Y-up, why don’t we “flip” all data coming from the Kinect openNI skeleton to be Z-up instead of Y-up!?! Isn’t it easier to model numerical data like v4’s data then rotating bones and joints…? So , theoretically the idea would be taking position XYZ from openNI kinect skeleton node and vector split it to get XYZ, then connect it back to a vector join node now, but this time connecting Y pos to Z pos and Zpos to Y pos (making the skeleton position XZY instead of XYZ), then obviously we would do the same for X orientation XYZ, Y orientation XYZ and Z orientation XYZ, not forgetting, finally to also swap Y orientation XYZ(now XZY) to Z orientaiton XYZ(now XZY) to completely flip the whole kinect skeleton Data from XYZ to XZY. or am I talking absolute shit here guys?

LOCKING ORIENTATION, MAPPING DATA: shouldn’t we do this inside v4? I am honestly not sure that collada will export orientation locks and orientation inheritance limitation, not sure, just saying.
Couldn’t a simple mapping of data sort most of the problems, if we correctly map, manually maybe to start, each and every X, Y and Z in openNI skeleton output data to be smaller or bigger couldn’t that help, let’s say, with my models arm’s being 3 meters long and super twisted (map z orientation right down to almost nothing)…?

BIG UP TO TEKCOR for this new module, I am trying it out now and we will see if this could be solution on v4 side anyways.
But we need a complete workflow from 3d modelling software right down to v4 patch. Let’s do it guys!

PS: Forgive me for writing such a long post, but I wanted to share my humble knowledge on the subject and my trials and errors so that we can find a solution. As shown in the picture below things go really wrong when bone orientation is not correct!

well first of all you can’t rotate biped without of special script, since skinning is locks orientation of your model,
second yes we spend 3 month on that and the problem is not only orientation of bones but also kinect not stable enough
third there is a way, but it’s a programming way described in mssdk avataring
fourth don’t waste your time on nonsense you can’t acidentaly solve 20 bones with root pivots and orientation, hierarchy is much more difficult to work with then anything since every second bone you wont see a result
is super geeky way to sort that, you need o know the codding and know how the biped works
its no way you sort that stuff without knowing that stuff, its comes from pices its not gonna work without them

hi guys,

i did this in unity3d in the beginning of kinect.
Back then the Joint Data from openNI had to be flipped.
In Unity i had no Problems getting it working with various 3D Models.
Don’t know if it helps but i just post the snipped anyway:

void TransformBone(uint userId, NiteWrapper.SkeletonJoint joint, Transform dest)
{
	NiteWrapper.SkeletonJointTransformation trans = new NiteWrapper.SkeletonJointTransformation();
	NiteWrapper.GetJointTransformation(userId, joint, ref trans);

	// only modify joint if confidence is high enough in this frame
	if (trans.ori.confidence > 0.5)
	{
		// Z coordinate in OpenNI is opposite from Unity. We will create a quat
		// to rotate from OpenNI to Unity (relative to initial rotation)
		Vector3 worldZVec = new Vector3(-trans.ori.m02, -trans.ori.m12, trans.ori.m22);
		Vector3 worldYVec = new Vector3(trans.ori.m01, trans.ori.m11, -trans.ori.m21);
		Quaternion jointRotation = Quaternion.LookRotation(worldZVec, worldYVec);
		
		Quaternion newRotation = jointRotation * Quaternion.identity;

		//Set rotation
		//dest.rotation = newRotation;
		// Some smoothing
		dest.rotation = Quaternion.Slerp(dest.rotation, newRotation, Time.deltaTime * 20);
		//Set position (unit length) -  1000 is the distance set in C code
		dest.position = Vector3.Slerp(dest.position, new Vector3(trans.pos.x/1000, trans.pos.y/1000-1, -trans.pos.z/1000), Time.deltaTime * 20);
	}
}

yo guys,

@Tekcor)) discovered/told me last night that the COLLADAFILE node actually has hidden pins relating to the Z-up/Y-up orientation of the model, allowing conversion from z-up to y-up! God, I could really punch myself in the face for not noticing this before! HOWEVER it doesn’t solve all problems. BUT it helps a bit, click on ((node:colladafile-(ex9.geometry) node and in Inspektor, select input model Up axis as Z and right axis as X, then for model output select Y axis as up and X as right. Now if you apply transformation using “setjoint” on a collada model simply rigged Z-up normally in 3ds max, then it works, I mean:it will move up if you translate Y and deep if you translate Z. So showing that even though the model is Z-up, it gets “converted” to Y-up by the collada node. But orientation is STILL not working when we apply the openNI kinect skeleton orientation.

So I then thought about checking if the data from openNI skeleton is orienting correctly, just on its own, without any linking to any transform or collada skeleton. So I created a very simple patch(attached to this post) that gets all skeleton openNI data (positionXYZ, Xorientaiton XYZ, Yorientation XYZ, Z orientaiton XYZ) and apply that simply to a set of BOX(ex9 geometry) by using the PARALLELEPIPED node. Guess what? the position of boxes is correct and it “almost” looks correct, but some of the orientation of boxes are weird/inverted/flipped. I think this highlights the fact that the kinect openNI skeleton itself is NOT y-up like V4, OR simply it just doesn’t work when put through a parallelepiped-(transform-vector) node (yeah ok it should work, but it doesn’t, I am guessing some of the orientations are flipped).
So is there a way to correct this weird parallelepiped orientation isssue?
OR would the solution be to use axisangle-(quaternion-set) or axisangle-(quaternion-get) node instead of parallelepiped-(transform-vector) node? So would using quaternion be a good approach?

@ @ethermammoth: Great post! really helpful man, thankx. That code seems to confirm that the quaternion approach is what we should do! also you highlight that openNI orientation is flipped/inverted on some axis. just learning unity as we speak but I pretty much understand your code, but how do we do this simply and most processor efficently in v4? I attached a simple kinect skeleton tracking patch, which highlights the orientation problem of openni skeleton when using parallelepiped node. Could you help on how to apply your code to this simple patch? so that we can correct the openni skeleton joint orientations to be aligned with v4. If we can do that I am quite confident that we can apply it to skinned model, yes it might take some time to find the correct settings/workflow in 3d software, but we can do it i’m sure. (the point is: there is no point in applying this openni skeleton data to a model now as it is already simply wrongly oriented in v4, let’s fix that and then we can think of applying it to model)

@ @antokhio: Sorry mate I am not sure I understand everything, but on first point if you mean in 3dsMax you can’t rotate biped without messing the skin, actually look at “advanced options” on the SKIN modifier you will find a check box “always deform”, untick that and then you can move your bones without messing the whole skin up, then re-tick that check box once you are done with your bones and it should still work. ANYWAY I agree with you though, we should find a workflow that is simple in 3dsMax/Maya/Whatever to rig models and export even if it is z-up… something that doesn’t involve pivoting joints, or rotating bones. The fact the “colladafile” node has hidden pins letting you change axis orientation, really should help the workflow from 3ds max to be simple. We convert Z-up axis to Y-up in v4 , not in 3ds max. I know what we are trying to do sounds fairly simple but in reality is really hard, I agree with you Antokhio and I am also frustrated by this but, still man, I feel like it is doable in V4, it must be… we can do this!

Finally if “all fails”, I guess we could look into doing the skinning “manually” in v4 using vertexbuffer-(ex9.geometry-join) node etc, as explained in this tutorial:
skinningwithskeletonnodes

So tonight I will try and apply your code ethermammoth to this simple kinect tracking patch attached, if you can try as well, or let me know what you think…

Thankx a lot for your contribution guys.

kinect_openni_skeleton_tracking.v4p (17.7 kB)

hey thanks guys.
I think the best thing would be to write ethermammoth’s code into a plugin.
should work or not?

Yeah good point tekcor, should work! I’ve never written a plugin for v4, however it’s on my “to-learn” list. I might have a go at this tonight. Do you otherwise/also think there is a way of doing this with a module/patch using axisangle node instead of writing a plugin? Or do you think even if we do manage to do it with a patch it will still be much slower, CPU-wise?

cheers man.

propably it would be possible using the quaternion nodes as well. for me it is just hard to imagine 4d vector math so it would be easyer to make a plugin for me :) but takes some time as well.

All this is sounding good, I’m distracted but a couple of other projects at the moment, unfortunatly!
I have a feeling that an FBX loader for v4 might help, I’m a c4d user, and dont really want to have to use max or maya just to build a skeleton, especially as any characters are likely to come from DazStudio anyway! (colladas from daz dont load correctly in v4 either!)

@antokhio @evvvvil @ethermammoth @tekcor
does any of you have a rigged model as .dae that has as many joints as the kinect V1 (20) or V2 (25) you could share for some testing?

“…as many joints as the kinect…” because it seems to me this would be necessary to get this to work at all?! i saw the models in tekcors assetdb contribution but they have higher joint-counts which we cannot really map to the kinect, right?

or has this been solved in the meantime?