With the release of vvvv gamma 2019.2 we introduced a new backend compiling patches in real time using Roslyn. This blog post is primarily intended for a technical audience, if you're solely interested what new features it brings to the table have a look at the before mentioned blog post.

Background

In the past VL (the language behind vvvv gamma) compiled in-memory directly to CIL using CCI. With the recent changes in the .NET ecosystem and CCI being superseded by Roslyn it became more and more apparent that at some point we'd also have to make the switch to keep up with the latest developments happening at Microsoft.

What finally pushed us into making the switch was two-folded:

1. An executable written by CCI wouldn't work at all the moment it referenced a library based on .NET standard. This was a major set back as nearly all future libraries would be targeting .NET standard and days of trying to find a workaround brought us nowhere. Making modifications to the already abandoned project CCI seemed like a very bad idea.
2. For a long time, we didn't know how to directly translate VL's adaptive nodes feature to CIL (or C# for that matter). For those who're not familiar with adaptive nodes, it allows one to build functions solely on their intent. For example, a LERP function can be expressed with a PLUS a MINUS a ONE and a SCALE node without having to specify what data will be provided in the end. The exact implementations will be looked up by the compiler when applying the function let's say on a float or a vector. This lookup is done from an entry point (like an executable) and while traversing from such an entry point all adaptive nodes will be replaced with their respective implementation. The emitted CIL code will then end up with two LERP functions, one for float and one for vector. This approach was working fine but it had a major drawback as it prevented us from pre-compiling our libraries and also it prevented any user to build a proper .NET library with VL.

Until this year in march @sebl came to the rescue by randomly dropping us a link in the chat pointing to a neat little "trick" which suddenly made it possible to translate our adaptive nodes feature directly.

After initial tests in March and April and having the patched tooltip feature still pending for the final release, we decided to let myself jump into the rabbit hole which I've finally crawled out of after more than half a year ;)

The neat little "trick"

Let's go back to the example of the LERP node and let's further try to write it down in C#:

T LERP<T>(T a, T b, float s) => a * (1 - s) + b * s;

Looks neat but sadly won't work, C# will tell us that the operators +, - and * and the constant 1 are not available on T.
The trick to make it work is to outsource those operators to a so-called "witness" which in turn will provide the implementation when the LERP gets instantiated with say a vector. So let's see how the actual needed C# code is gonna look like:

T LERP<TWitness, T>(T a, T b, float s) where TWitness : struct, IPlus<T>, IScale<T> 
{
    var w = default(TWitness);
    return w.Add(w.Scale(a, 1 - s), w.Scale(b, s));
}

with

interface IPlus<T> { T Add(T a, T b); }
interface IScale<T> { T Scale(T a, float s); }

and when applying it with say float we need to define a witness implementing the needed interfaces

struct MyWitness : IPlus<float>, IScale<float> 
{
    public float Add(float a, float b) => a + b;
    public float Scale(float a, float s) => a * s;
}

which finally allows us to call

LERP<MyWitness, float>(5f, 10f, 0.5f)

Fancy, no? The beauty is that when the JIT compiler hits such a code path it will be smart enough to inline all calls so in the end for the CPU the code to execute is the same as the initial naive attempt. But don't worry, this is all happening behind the scenes. In the patching world, it is as simple as it was before to patch generic numeric algorithms.

The implications

So now that we're able to translate patches directly to C# what are the implications apart from being able to export an application?

Easier to maintain

Well for us as developers it will be much easier to bring in new language features, because the code we generate will be be checked by the C# compiler and more important, we can fully debug the generated code with Visual Studio. That by the way is not only restricted to us, anyone can now attach a debugger to vvvv (or the exported app) and debug the patches.

Faster loading times

The generated C# code will make full use of .NET generics. So when building a generic patch the generated class will also be generic in the pure .NET world. As an example let's consider the Changed node, while in the CCI based backend a Changed class was emitted for each instantiation (Changed_Float, Changed_Vector, etc.), the new Roslyn based backend will only emit one Changed<T> class and it is left to the JIT compiler of .NET to create the different versions of the needed target code. This should lead to much less code the CPU needs to execute as the JIT compiler is much smarter on when to generate new code and when not.

But what's even more important is the fact that it opens up the world of compiling VL patches as pure .NET libraries. So we can finally pre-compile our libraries (like VL.CoreLib, VL.Skia, etc.) which in turn reduces the general overhead and leads to much quicker loading times and less memory usage. As an example loading the Elementa "All At Once" help patch takes ~15 seconds the first time (compared to ~33 seconds in the old backend) and thanks to caching to disk only ~8 seconds when opening at a later time.

Instantiate patches at runtime

Apart from better loading times, it also gives the patcher the ability to instantiate any VL patch during runtime. In the previous backend, one had to use a hack and put all possible instantiations into a non-executing if-region. This is not necessary anymore as all the patches get compiled. However, I should mention here that this is only true for non-generic patches. Generic patches usually require a witness which is not so straight forward to provide.

The drawbacks

Sadly the new backend also required some major internal changes in the frontend so it wasn't possible to guarantee existing patches would work the same way as they did before. Here follows a list of potential breaking changes:

• The type unification algorithm had some flaws and therefore needed modifications. In general, it is "smarter" than before, so when starting a patch from scratch fewer annotations should be necessary. But for existing patches, it sometimes finds a different solution than the previous algorithm leading to red links on the application side of a patch. In those cases, one needs to set a type annotation in the patch definition.
• When pause on error is enabled the old backend was able to show computed values up to the point where a node was crashing, this feature is sadly not available anymore due to local variable scoping rules of C#. Whether we'll bring this feature back or not is not yet decided.
• Naming rules are more strict, so it's not allowed anymore to have for example an "Update" and and "Update (Internal)" operation as both are considered to have the same name. In general instance-operation overloading is not possible.
• Static operation overloading - having operations with the same name but different pins - is still allowed as long as the types of the pins differ. So different pin names alone are not ok.
• When defining a generic interface all type parameters of its operations get assigned to the interface itself. This is necessary due to how internals of the emitted C# code work.
• The cyclic graph detection wasn't correct. This can also lead to red links now but should be considered a good breaking change as it makes the patcher aware of undefined execution order.
• Patches, where the execution order wasn't defined, might behave differently now.

Meanwhile...

...since we're terribly behind schedule regarding the release of a new website for vvvv.org (and we know that it will still take a while), we've created a preliminary site at:

This is mostly so that we can provide people who are new to the world of vvvv with a simpler entry point to our universe. You'll see that the site focuses mostly on vvvv gamma, which makes most sense for new people to explore. Obviously the existing vvvv.org remains our main portal to share news and updates with everyone. We consider visualprogramming.net a temporary landing page for vvvv gamma.

Nevvvvsletter

If you're old enough to remember, there is this thing called electronic mail (short: e-mail). In a classic retro-move we've also recently started an e-mail newsletter which you can sign up to from that page. Scroll to the bottom and find it in the left corner. Signing up will give you peace of mind to not miss any happenings around vvvv. Typically one mail per month.

Just in time!

Only a whopping 6 years and one and an half months after its first mention during Keynode 13 and to the day exactly 5 years after the release of the The Humble Quad Bundle, you can finally hold it in your own hands. Not exactly as the full release we had planned but as a preview:

To our own surpise we couldn't finish all the things we had planned to release today. Most notably the "windows executable export" didn't make it. We know this is a bummer, but we want to get this right and it just needs some more time.

Apart from that we figured there is no more need at this point, to keep it to ourselves. It is definitely good enough for a preview, definitely good enough to gather some feedback to incorporate into the final 1.0 release for which we take some more time to finish our plans. So let's have a look at what we got:

What's new compared to the vvvv beta series

Forum

To accommodate for the fact that from now on we essentially have 2 products, we added two main categories to the forum:

The existing question, feature, bug, general sections were moved into vvvv beta, and the vvvv gamma section got its own question, feature, bug and general sub-sections. Note that by default the search is constrained to the category you're currently viewing. When you're using vl in beta, still feel free to ask questions in the beta forum. We'll handle that.

Tutorials

Head over to this forum section to watch some video tutorials:https://discourse.vvvv.org/c/tutorials

Pricing

We've previously announced the upcoming pricing model for vvvv gamma, which we're currently refining and we'll update you on changes to it soon.

Until further notice, the previews of vvvv gamma are free of charge but due to its preview-nature we don't recommend using it in commercial scenarios yet.

Download

Here you go: vvvv gamma 2019.1 preview 975

Changelog:

975 26 11 19

• works with VL.Audio
• Fixed race condition between state disposal and OnNext calls in reactive ForEach regions. Was responsible for an access violation in OpenCV tooltips.

959 19 11 19

• equals version included with beta39
• can now create a node while linking by simply starting to type (double-click not required anymore)
• fixed SVGWriter
• fixes on bumpy Damper
• removed a few unnecessary Damper versions
• fixed time step calculation of Damper (Fast)
• Dampers can now be used in loops
• fixed image tooltip
• fix on tooltip layout
• some color operation improvements

930 02 11 19

• fixed annoyance with nodes being selected instead of dragged
• fixed problem with red nodes not updating after installing VL. nugets
• fixed time step calculation of Damper (Fast)

923: 31 10 19

• added ToVector2/3/4 (Float32)
• fixed Tokenizer (Frame)
• fixed register services method called too early
• fixed regression that timings wouldn't show up on regions
• fixed timings not working on Cache region
• fixed serializer not dealing with fields of type object correctly
• fixed serializer crashing with a stack overflow when data like vectors or colors stored in an object field were re-interpreted as arrays
• fixed IL emitter not picking up type dependencies on fields if the type was a generic type instance (https://discourse.vvvv.org/t/weird-reference-documents-behavior/17949)
• added Random implementation for RGBA
• fixed hue calculation and minor improvements in color space conversions
• added String <-> Array<Char> conversions
• shutdown dialog now lists unsaved documents
• installer adds windows defender exceptions

827: 09 10 19

• setup.exe is now signed and shouldn't trigger "windows protected your PC popup anymore"
• added Quad>Windows>Key/MouseDisplay
• fixed GrayScale Skia ColorFilter
• added Damper/Oscillator 2D, 3D
• Packages are now in AppData\Local\vvvv\nugets

703: 16 09 19

• tooltip performance improved
• ImageReader now returns correct format of images
• added FromBytes (SKImage)
• added Resize (SKImage)
• LinearSpread now has Phase input
• added midi ProgramChange node

667: 03 09 19

• added IsEven/IsOdd nodes
• added Morph node
• added MultiFlipFlop node
• added ConnectAll node
• added CounterFlop
• added Random (Centered)
• added Sort (FormerIndex)
• added OrderBy (FormerIndex)
• added IndexOf (KeySelector)
• added Search
• added Search (KeySelector)
• added Resample nodes (Point, Linear, Repeat, Hermite, BSpline)
• Switch node can now have more than 2 inputs
• Filter node now has TweenerMode exposed
• Nodebrowser now also looks for tags in nugets
• FromImage (Skia) now has options for the case of R16->R8
• FromImage (Skia) now handles 24->32bit conversions
• ADSRSettings has optional inputs for Attack, Decay and Release curve settings
• ADSR has an input to set a new clock at any moment
• fixed AddRange (Array) of SpreadBuilder
• updated VL.OpenCV to 0.2.129-alpha

618: 22 08 19

• SVG/PDFWriter now deal with background correctly
• improved some warnings
• increased max tooltip height

615: 21 08 19

• more tweaks for tooltips
• Ctrl+F now also considers nodes category
• updated to VL.OpenCV 0.2.122

573: 08 08 19

• pin tooltips now show their infos again when available
• copying messages from tooltips is now via ctrl+shift+c
• added simpler Mouse and Keyboard (Skia.IO) nodes
• skia primitives (rect, circle,...) now come in two versions, instead of as overloads
• updated to VL.OpenCV 0.2.121

552: 01 08 19

• reworked tooltips
• new settings: MouseMiddleButtonOpens to activate middleclick to open patches of nodes
• addded node: FromImage (MutableArray)
• added skia ColorFilter nodes: Transform, Brightness, Contrast, Grayscale, LUT

411: 12 06 19

• Value to bytes nodes now have defaults
• Fixed somehow newly introduced crash in patches making use of serialization (like Tilda) or reflection API (like the runtime-model-editor demos)
• Fixed accumulators on loops being auto-disposed causing object disposed exceptions in more complex patches (ike Tilda)
• Fixed Sampler (Reactive) getting stuck in an endless loop if upstream observable crashed (also seen in Tilda)

406: 10 06 19

• Fixed crash when creating IOBoxes in regions while linking
• Fixed pin highlights when linking via region border point
• Fixed application restart with F8/F5
• Skia gradient nodes rework

398: 05 06 19

• fixed couple of regressions in compiler introduced between 369 and 380
• fixed splash screen flicker
• fixed a null exception on startup

380: 01 06 19

• fix for Tokenizer (Frame/Postfix) with empty separators

369: 27 05 19

• Skia PerfMeter (F2) now measures full paint time
• fixed "Countdown" output of Trigger node
• fixes in compiler
• performance improvements in compiler
• added pixel format R32G32F to imaging
• fixed a freeze in Tokenizer
• added Tab, CR, LF, CRLF nodes
• added serializer for Range
• theme and interaction improvements
• added Skia checkerboard style that can be used as a paint
• fixed removing .vl doc references

344: 14 05 19

• PerfMeter in Skia renderer (F2) now shows UpdateTime and RenderTime
• added checkerboard style that can be used as a paint for any layer
• improved scrolling behaviour for sliders
• CoreLib improvements
• sped up RepellingCircles demo patch
• several compiler fixes
• compiler performance improvements

318: 09 05 19

• frames now let you choose colors from a palette instead of the color chooser
• frames now move their content along as you drag on their titlebar
• frames now only move elements that are fully contained
• frame is now included in the "surround with" context menu
• press SPACE to force-include frames in selections
• in inspektor changing precision for floateditor now also sets precision for min, max and stepsize.
• can now grab border control points on regions properly without interfering with region resize
• default culture setting is now invariant

303: 08 05 19

• fixed missing dependency for VL.OpenCV

301: 07 05 19

• windows timer is set to 1ms on start
• mainloop uses less performance and doesn't block windows messages
• Skia Renderer has PerfMeter build in, toggle with F2 when selected
• fixed dpi problem with text in SymbolFinder
• ctrl+T/ctrl+shift+T to bake/clear type annotations on datahubs
• fixed "invalid cast in typeunification" error

287: 06 05 19

• shortcuts now work with all tabs closed
• Renamed action "Assign->Pop" to "Assign->Clear assignment" to make it easier to understand what the action does
• Firmata: Tokenizer was stuck in an endless loop
• fixed null exception in ResizeSelectedMouseHandler
• Typewriter: Shift+PageUp/PageDown - select to the beginning/end of the document, cursor stays at the same column.
• OverlayEditor now has minimumsize (again)
• ImageEncoder doesn't have the bmp option anymore as skiasharp can't encode into bmp

273: 02 05 19

• fixed another problem with editors/tooltips and high dpi settings
• fixed "ReguarExpression" typo
• AllAtOnceEditor for vectors now sticks to value of first component
• fixed problem with enum-editors on pins getting stuck
• no more duplicate "Horizontal" entry in IOBox inspektor
• inspektor now also shows elementtype properties for Spread<Vector>
• serialization for custom types doesn't throw errors for inspektor/defaults
• upstream dis/connected iobox no longer looses its settings
• added GroupBy (Length) and GroupBy (Count) nodes to split a spread into spread-of-spreads
• added Clean node: removes slices with empty strings
• added RepellingCircles skia demo

252: 27 04 19

• fixed dpi handling for fonts in editors
• can now set ApartmentState of BusyWaitTimer to make UI threads
• mainloop now has high precision
• added PerfMeter to VL.Skia
• editing comment/string now keeps size of editor
• comments now have correct initial size
• StringEditor on pin now has wider fixed width
• fixed problem with paddings differing between single and multiline textbox
• fixed setting bool pin value via dragging
• fixed interaction in signature view of patch explorer
• fixed deadlock when implementing interfaces

230: 24 04 19

• fix for regions inside operation definitions disappearing
• fix for patches with more than 10 operations showing later operations as black
• quad icon now works for all themes
• previous/next icons now colored correctly in all themes
• string editors/comments now have a configurable "Max Visible Characters" to prevent low performance with too long lines

222: 18 04 19

• VL.Skia Camera 2d is not experimental anymore
• fixed pin interaction in signature view
• fixed an edge case when then node browser wouldn't show up
• fixed IOBox rendering freezes
• added many tags to VL.CoreLib to find nodes faster
• VL.Skia is referenced by default for new documents
• toggle toggles on every mouse click
• IOBox values are not applied while typing anymore

200: 15 04 19

• inputs/outputs of definitions/regions and groups can be moved (again)
• fixed problem with documents not opening anymore
• fixed file path serialization of dependencies when the path couldn't be made relative to the document itself
• fixed coloring of pads and region bordercontrol points

191: 13 04 19

• a comment that only holds a link can be right-clicked to open in the browser
• recent sketches now show in reverse order: most recent is topmost
• fix: improved recizing of nodes, regions and ioboxes
• fix: input/output indicators on pins and pads are now in sync with tooltip (again)
• fix: selected spread ioboxes can now be deleted with backspace when hovered with the mouse

180: 11 04 19

• fixed background for definition patches
• Skia ButtonBehaviour now lets you specify which buttons to listen to

177: 10 04 19

• new setting DocumentAskOnFirstSave sets whether to ask for save location on first document save
• added "Show Intro Patch" to quad menu, to recall intro patch even if it's not shown on startup
• reactivated play/pause mode visualization
• various coloring/theme fixes
• active tab is underlined (again)
• definition patches now have a hatched background
• removed RestructureDocument from patch context menu
• default count of a collection pin group can now be configured
• Skia Group defaults to 2 inputs (again)

150:

• VL.OpenCV now comes with demo patches in Help Browser!
• fixes for Skia ImageReader and ImageWriter
• added '-m' or '--allowmultiple' command line arg to allow running multiple instances side-by-side
• shortcuts are deactivated for patch when Finder box is open
• several fixes for IOBox drawing and interaction

139:

• fixes various assembly not found exceptions when using nodes of the Midi category, the Script region or binary serialization: a, b, c

137:

• fixes problems with pin-editors: a, b, c
• enables spread-editors directly on pins

134:

• Info.vl in now called Intro.vl
• double-clicking .vl files will open with the already running instance
• Skia renderer goes fullscreen via F11 or Alt+Enter
• many fixes and tweaks

Apart from the promised and still missing parts, we're aware of quite some little glitches still and will update the download link above periodically. So please check back often and report your findings!

Yours truely,
devvvvs.

vvvv looks back on a history spanning almost 20-years of wonderful projects. Many are the first of its kind and have influenced and defined trends in creative coding, digital arts, projection mapping, and interactive installations.

Over time we've become so familiar with all of this, we forget that many people aren't.

To address this, we proudly present our first showreel:

This video is even safe to share with your friends.

See credits or Vimeo description for the full list of projects.
Thanks to everyone and happy projecting. We'll probably do this again next year.

Yours,
devvvvs