Interactive Book

Hello everybody,

I’m Working on a project in which I’m trying to create an Interactive book. The book is going to have at the one page written content about History of Ancient Greece and at the other page is going to have either a video or an interactive game. I want to use Kinect to track the hands of the user so he can play the game. Also when the page changes, the content has to change so I thought to use fiducial tracking to track the pages of the book.

My main concern is the games…The games that I will have in the book are going to be simple…for example like this: http://www.miniclip.com/games/short-history-of-the-world/en/ but imagine it with Kinect use…Is it possible? The games will be created in Flash.

How the interactive flash game will be interactive with Kinect?..What I mean is, in flash the game will be interactive when the mouse is clicking on several points…when is projected on the book how the interaction will be achieved…??

Thanks

so the essential challenge is to:

  • Create a ‘touchscreen’ using a kinect and a projector
  • Send touch interactions into Flash and render to VVVV

Anybody got experiences with either of these?

(Vasilis was a student on my VVVV Omniversity course at Madlab UK)

Hello,

This article has some info

I prototyped a system which used a kinect and some transforms to view the point cloud mesh in the 3D region defined by a 2D screen + an extruded layer above it (e.g. 1cm). Then used contour to track interactions in this region.

This type of system might be possible here, but requires going from GPU (rendered transformed mesh) to CPU (OpenCV), which can work with AsTexture and VVVV’s native Contour node but would be pretty janky.

i’d make some tests at the beginning… tryout the 2 flash-solutions vvvv brings (the flash renderer and the flash plugin. test them with some flash content that covers all of your problems, like user input from vvvv or sending arbitrary values to vvv for example) is one of them fullfilling all your needs? if yes, then you could head towards the kinect-task. be sure to define what you want … then again, some small tests with the 2 native kinect-plugins available until you get proper xy-coordinates and click events that can be fed into flash.
If all that’s done you have your poc and can “start” the project.

if you face any problems in any step, feel free to use this thread to ask for help.

cheers

Thanks for the replies…

I will try what sebl suggested and I’ll get back to you with any problems I found.

how’s it going vasilis?

I’ve almost finished the flash game…I also ordered a Kinect and I’m waiting for it next week…

thanks

perhaps, you should consider doing the whole thing purely in vvvv…

If it isn´t really trivial I´d refrain from it. Gamelogic in vvvv is a pain in the ass.

Because I am new in VVVV I’m not familiar on creating games in VVVV instead of Flash…The game is 80% finished so now the next thing if I got it right is to take it in VVVV and see if it’s working there?

How much easier would it be if i was doing it entirely in VVVV?

Depends on what exactly the game has. Here based on your screenshot I suppose user is allowed to click on 9 different icons and get a respective pop-over message and close the same? If this is all then I would say easy in VVVV but if its more please explain…

basically yes…that’s all…but my question is why to do this in VVVV instead of Flash?

With the use of Flash(EX9)chrismo I’ve insert the flash game in VVVV…

It’s a choice of comfort. This much in flash is very easy but it’s convenient in VVVV considering further processing. Flash is as good for playing the swf not for receiving any info from… however you may retrieve info based on need in your application architecture.

The problem is that because I don’t have a lot of time for this project(is my final project for Master’s degree) and I need to make another two games at least, with flash it would take me only a few days to make them but in VVVV because I’m new it would take me much more time…So now the only thing that I have achieved is to play the swf in vvvv…Can you tell me the next steps I need to follow? or what is your approach to this project?

thanks

the game is very easy…Tne Book in the first page is going to have the story of Alexander the great…the second page is going to show the game and a guideline that will say “Lead Alexander the great to the end of his empire” (the stops are 9 and to go from one stop to another you just press the next stop…so the only clickable buttons in the whole game are the stops…)when the user clicks to the stop Alexander will move to that stop and a message will appear with some history info of the specific stop…

I can write what I think but it’s only my openion and I haven’t tested it out. You may also consider the way Mr. elliot suggested if possible.

So, i think it would be good if you can print the fiducial marker on the book for good reason, let me tell you why in a bit.

  1. you need to find a way to recognize the finger tip and use your finger tip as the XY coordinates. Certainly seems possible with OpenCV, also have a look into Hand (Kinect OpenNI)

http://www.kinecthacks.com/kinect-finger-detection/

http://www.youtube.com/watch?feature=fvwp&NR=1&v=lCuItHQEgEQ

  1. You need to calculate the Z about 1cm off the book like when your finger tip crosses the line and based on XY you trigger the click in flash node

  2. It can be a pain to calibrate each time the XY&Z off the book in every place you try because you need to map those value to -1,1…and this is where fiducial tracker may be useful, you may use apply transform node to see where the fiducial tracker is in space and do the math for map node.

All the best, hope it helps :)

Ok I will consider all this…do a research for a few days and I will get back to you…

Does anyone know a way to send arbitrary values from flash to vvvv when the swf is in actionscript 3.0?? It seems that with actionscript 2.0 is possible but with 3.0 it’s not…

Should I redesing the game but with actionscript 2.0 code?