General Status

As you know, efforts have been going for the last year and a half into bringing a computer vision nodeset to VL.

The goal was to incorporate as many of the features covered by the world renowned ImagePack; contributed by Elliot Woods some years ago; while bringing in as many new features and improvements as we could.

In the winter of 2018, after setting a roadmap and having patched a decent initial nodeset, we happily announced the pre-release version of VL.OpenCV.

Since then, listening to your needs and constant feedback, we have tried to polish every corner, fix every bug, document every odd scenario, add plenty of demos and specially we tried to give you a clean, consistent and easy to use nodeset.

At this point in time, we are happy to announce that the goal has been nearly met. Most of the features available in the ImagePack made it into VL.OpenCV with the exception of Structured Light, Feature Detection, StereoCalibrate and some of the Contour detection functionality. At the same time, newer features such as YOLO, Aruco marker detection and others have been brought in for you to play with.

So what's next? Even better documentation and loads of examples!

In the mean time, here is a summary of the new things that have been brought into the package in the last couple of months:

CvImage

The new CvImage wrapper around OpenCV's Mat type allows for some optimizations, specially when dealing with non-changing images.

CvImage allows nodes to run only once when the image has changed, significantly reducing CPU usage

Cache Region lovvvves CvImage!

Since it is now possible to detect if an image has changed, CvImage is a perfect candidate to benefit from Cache regions.

Cache regions can now make proper usage of image inputs and outputs

Renderer

The Renderer was re-built from the ground up to improve usability and to fix bugs and issues. Title, Render Mode and Show Info features were added. Renderer also remembers its bounds on document reload.

New Renderer implementation introduces Title, Renderer Mode and Show Info pins

Histograms

Histogram analysis has been added to VL.OpenCV. A useful tool in many scenarios.

Histograms allow you to analyze pixel value tendencies per channel

Homography

Homography and reverse homography are now available in VL.OpenCV.

Homography (vvvv used only for point IOBox)

Stereo Matchers

Two new Stereo Matchers were added, these allow you to create a depth map from a set of stereo images. For more see the StereoDepth demo patch in VL.OpenCV.

Depth map obtained from a pair of stereo images

Serialization

Serialization support was added for CvImage and Mat types, allowing you to use CvImage as part of more complex data structures which get serialized with no effort. This can be a heavy operation so make sure to trigger it when needed only.

For a hands on demonstration check out the Serialization VL demo that ships with VL.OpenCV.

Breaking Changes

As part of this final effort to clean everything even further and make the nodeset consistent and properly organized, we needed to rename and move a few things around which as you can imagine means the introduction of breaking changes. We understand this is an annoying thing to cope with, but this was basically the reason why we chose to keep this pack in a pre-release state until we felt confident with its structure and approach.

In summary yes, you will get red nodes when you upgrade your VL.OpenCV projects to the latest version, but in most cases it should be as easy as to double-click and find the node in its new category.

An exception to this are the nodes that got renamed, which we list below:

• Reader (Intrinsics) -> IntrinsicsReader
• Reader (View Projection) -> ViewProjectionReader
• Writer (Intrinsics) -> IntrinsicsWriter
• Writer (View Projection) -> ViewProjectionWriter
• GetByteArray -> ToByteArray
• GetValues -> ToValues
• GetValues (Custom) -> ToValues (Custom)
• FiducialTracker -> FiducialDetector
• Update (FaceRecognizer) -> FaceRecognizerUpdate
• VideoFile (Append) -> VideoWriter (Append)
• VideoFile - VideoPlayer
• Merge -> Join

Summarized Changelog

General

• Introduced a new wrapper for Mat called CvImage
• Cache region additions to improve performance on non-changing Images
• ImageReader re-implementation
• Renderer re-implementation
• VideoIn re-implementation
• Reintroduced "Supported Formats" output pin on VideoIn node
• Upgraded to VL.Core 0.95.37
• DirectX/OpenCV transformation documentation and cleanup
• Added Blocking, Reactive and HoldLatestCopy versions of VideoIn node
• Added Blocking and Reactive versions of ImageReader node
• Reimplemented Extrinsics as a data type
• Stabilized VideoFile Source node and moved from Experimental into Sources
• Added Serialization support for Mat type
• Added Serialization support for CvImage type

New in VL

New in vvvv

Remember that in order to use VL.OpenCV you first need to manually install it as explained here. Also, until we move away from pre-release you need to use the latest alpha builds.

We hope you find good use for this library in your computer vision projects and as always, please test and report.

Since a while, VL comes with the idea that you can organize node and type definitions in your VL document.

But now, we want to give you back another, alternative way to look at things - an organization structure, which is more intuitive and also well known from vvvv beta: The application side of things...

And also, we did this in reaction to the feedback we got from Link festival:

You want to be able to navigate the running object graph, where it's about the instances of patches, not about their definitions. You want to be able to navigate into a running patch and see the values that flow in this instance, not in another instance of the same patch...

Also, typically you approach your project top-down and just add more details to it since this is the basic idea of rapid prototyping: patching a running system that you incrementally grow and modify.

So we took the chance to shift the focus a bit so that in VL you again get confronted with the application side of things first.

The Application Side of Things

This is what you know from vvvv beta: a patch can contain a sub-patch - you navigate into it and inspect the values flowing. You go outwards - to the caller - via "Ctrl-^". With the ^-Key we actually refer to a key at a certain position on the keyboard.

In VL this now is just exactly the same. Navigating into a process node shows you the right values. Ctrl-^ brings you back to the caller. So you are basically navigating the living node tree of the application. In VL it's been hard to think in these terms, but now it's the default. We also made sure that this works when navigating between vvvv beta and embedded VL nodes.

Also, try to use the back and forth mouse buttons if you happen to have a 5-button mouse. Ctrl-MouseBack will bring you to the calling patch and Ctrl-MouseForth will travel back into where-ever you were coming from.

Every VL document comes with an Application patch, which will open by default. You can start patching right away. A bit as it is like in vvvv beta.

Patching top-down never has been easier. Creating an Ape simulation from scratch:

You can run many applications at the same time, e.g. several example patches in addition to your project app. The application menu lists all documents that actually make use of the application patch.

The Definition Side of Things

Definitions in vvvv beta basically correspond to the .v4p files, in VL you can have more of them per document.

Library developers or more advanced users will of course still want to organize types and nodes and approach them from the definition side. This is like saying "There is one idea of a wheel, but if you feel like you can instantiate three of them".

For an overview of the definitions, each document comes with a separate Definitions patch - basically what's been the document patch.

Here you see what happened during patching top-down: on the definition side, we now have two Processes.

That's where you would from now on also place your Classes, Records...

Navigation within the current document structure works with Ctrl-Shift-^, Ctrl-Shift-MouseBack, Ctrl-Shift-MouseForth.

When navigating into a patch like that you will see some instance of the patch or maybe none, if none is instantiated or currently running. In this case, you will not be able to see any values.

If the patch is not yet inspecting a particular instance it will wait for the moment an instance gets executed and then attach to this particular instance.

Some more minor cleanup

We took the chance to clean up some bits in the node browser and the patch explorer as well.
The application patch e.g. now doesn't offer confusing options, but basically only shows the properties stemming from pads, the Process Node Definition now is called that way (was "Order"), Process Nodes in the node browser look a bit like process nodes in the patch, choices like "Input", "Node" appear at the top of the list of choices in the node browser...

That should be it for now!
Thanks, yours devvvs

Holy patchers!

To answer the often contemplated question of "Where the hell is this all going?" we first have to establish that vvvv is different things for different people. Let's have a look at some of these different perspectives:

￭ vvvv - a generative designers best friend
￭ vvvv - a data visualizers wonder weapon
￭ vvvv - an interaction designer's magic wand
￭ vvvv - the multiscreen solution
￭ vvvv - physical computing covered
￭ vvvv - a projectionmapper's dream
￭ vvvv - rapid prototyping on steroids
￭ vvvv - your procedural lighting desk
￭ vvvv - computervision for everyone
￭ vvvv - stage? show? control!
￭ vvvv - fish feeding for the lazy
￭ vvvv - L.A.S.E.R zzzz!1!!

That's all?, we hear you say...and you mumble on, but can I please has:

￭ vvvv - a tool for machines to learn
￭ vvvv - easy logic for gamers
￭ vvvv - visual scripting done right
￭ vvvv - klickklickklick your mobile app
￭ vvvv - the no-brainer IOT toolbox
￭ vvvv - the final webdev solution
￭ vvvv - an admin's batching buddy
￭ vvvv - robots dig this
￭ vvvv - no controller too micro
￭ vvvv - from sound to noise in one patch
￭ vvvv - any shaders of gray
￭ vvvv - scratch that: kids taken seriously

Well, vl, vl, we hear ya.. and that's exactly why at the heart of it all, we have always been, and will continue to work on:
￭ vvvv - a multipurpose toolkit

Introducing vvvv gamma

When we started working on VL, we never thought about creating anything different than vvvv. We had our issues with vvvv and wanted to solve those. But mostly we loved it and wanted to keep the good things we still enjoy about it. Essentially we set out to rewrite vvvv to make it better, nothing more.

So it is our great pleasure that we can finally annouce the impending release of what we've so far been referring to as "VL standalone":

For a start, vvvv gamma will focus on 3 main scenarios:

• 2d interactive motion graphics using VL.Skia
• computervision using VL.OpenCV
• controlling devices, ie. everything IO (Midi, UDP, OSC, RCP, HTTP...)

Frankly it will be gorgeous and above all shine through the following features:

So beta vs. gamma, really?

Nope, don't think vs., we prefer to think ♥!
Here is how to read this: Over the past years we've been R&D'ing a next generation visual programming language that we dubbed: VL. And we still call it that: VL is a programming language, just like c#, that now happens to power two of our products:

• vvvv 50betaX
• vvvv gamma 2019.X

While for vvvv beta, VL is just another language it supports, for vvvv gamma, VL is the core. As such, both beta and gamma will continue to profit as we progress with VL. We are well aware of the fact that gamma is not going to replace beta for everyone anytime soon. We're therefore commited to support both equally for the foreseeable future.

If you're interested in a few more details regarding how we'll be polishing vvvv gamma for its initial release, please head over to our roadmap.

What will gamma cost?

We're still working this out and will update you on licensing early next year.

Wait and what about Xenko/3d?

As we've previously reported from our lab, our initial efforts of integrating the Xenko 3d engine couldn't have gone much smoother. We have helped realize a complex multiuser VR project within just 4 months besides our research and now know that this is certainly the way for us to go forward. VL.Xenko will complete vvvv gamma to a fancypantsy interactive 3d authoring environment.

We're confident to be releasing an initial public version of VL.Xenko before 2019.Q4 and we have some ideas for a prerelease phase that we'll announce soon.

Thanks for using vvvv, thanks for your feedback on VL, thanks for your patience, thanks for the licenses you buy which makes this journey possible for all of us.

Have a nice holiday, get some rest, because 2019 will be the year of the vvvv desktop!

yours,
devvvvs.

Welcome back to the second sneak peek into our adventures with xenko. Together with MLF we've been busy patching the first project done entirely with vl and xenko: Ocean of Air. So far the combination works superbé, and you can experience that for yourself until the 20th of January, if you are in the London area. Alongside the project we explored the xenko code base and now that it's live, we can give you some more insights into our research.

In the last blog post we used predefined entities to set up a little scene graph. This time we will dig a little bit deeper.

Having primitive objects like Box, Sphere, Plane etc. is nice for casual patching and quickly visualizing something. But you will need more sophisticated objects for the final output of your project. What you want to do is designing your own objects that are specific to your use case.

Luckily, game engines have quite similar requirements and came up with a good solution, and they call it entity/component/system, short ECS, which is also the latest hype in Unity. Xenko has a good documentation page if you want to go into detail. But for now let's stay on topic and keep two things in mind:

• Entities are the nodes in the scene graph
• Components are the features/properties of an entity

We found two appealing ways to create custom entities that can also be combined with each other in any way that suits you. You can either patch them or design them in xenko's game studio using their prefab workflow. Here is a simple example for both cases:

1. Patching Entities

Let's look inside the BoxEntity from the last blog post:

As you can see, it adds a BoxComponent to the entity on Create (white) and exposes parameters like Color, Transformation, Enabled etc. as input pins on Update (gray). This is more or less an arbitrary choice of how the BoxEntity is designed and it will probably change a bit before it becomes official. The patch is also an example of how vl's process nodes work nicely together with the entity component model. Each instance of an entity or a component can be represented by a process node and connected with each other in an understandable way.

In the patch we saw the EmptyEntity node, which is a general entity object that contains nothing more than a TransformComponent, hence the transform input pin. To make something useful with it, we add more components (e.g. model, material, audio, physics etc.) to it. There are many of them and you can combine them as it suits your use case. The big advantage here is, that the components are able to interact with each other via the common parent entity and that the scene graph system automatically processes them in an optimized way. This is where it gets interesting!

Let's say we want the box from the patch above to emit a sound from its current position. In order to do that we only have to add a SpatialAudioComponent to the same entity as the box component:

Since the SpatialAudioComponent and the BoxComponent have a common parent entity they will share the same transformation. Also, if an entity has child entities, the children get transformed by the parent. We could use that feature to add an AxisEntity to our custom entity:

Again, there is no need to connect the input transformation to the AxisEntity since it gets added as a child to the main entity and gets transformed automatically.
Here is what a little scene could look like:

Let's add a second one and let them rotate in the scene to hear the spatial audio effect. Aaaaaand action! (works best with headphones):

2. Prefabs

There is also a super easy way to design custom entities in xenko's game studio and use them in your patch. Suppose we have a 3d model with animation and skinning imported and edited in game studio. All we have to do now is to create a prefab from it and give it a meaningful name:

Learn more about xenko's prefab workflow here. Once we have that it's as simple as this to use it in your scene:

Ridiculous!
And finally we will start the walk animation:

Starting the animation is also patched in this case, but let's save that one for another post.

Conclusion

The entity component model of xenko works very well together with vl's process node feature. VL's automatic recompile and instant stateful hot reload allows to dynamically combine and configure entities and the scene graph in real-time while the application is running. You can combine different workflows with each other, simple primitives, custom patched entities or imported prefabs. There is no right or wrong, just build up the scene in a way that suits your way of thinking and the requirements of your project.

We still have only scratched the surface here, there is much more to come.

Yours,
devvvvs