» New Tokenizer Nodes
This site relies heavily on Javascript. You should enable it if you want the full experience. Learn more.

New Tokenizer Nodes

Dear jugglers of the bytes,

sorry this took so long! Almost five years ago we introduced the new datatype Raw for easier handling of byte streams in vvvv. It came with all the nodes you'd need, except probably the most important one, a Tokenizer. So for collecting incoming bytes via e.g. TCP or RS232 and making sure to separate them into the correct message-tokens you'd still have to fall back to the good old Tokenizer (String). Possible, but annoying.

Enter the new series of Tokenizer nodes:

  • Tokenizer (Raw FixedLength)
  • Tokenizer (Raw LengthPrefix)
  • Tokenizer (Raw Postfix)
  • Tokenizer (Raw Frame)

Users of Tokenizer (String) may remember that it was always a bit tricky to configure since it had quite a few options to configure it and you'd have to make sure to get those all right for your specific use-case. So now we've separated those use-cases and spent each of them an individual node. The nodes versions should be self-explanatory. If not, they all come with help-patches!

Now all of the Tokenizers always return a spread of tokens found in the last frame. So in order to simulate the Queue Mode of the original Tokenizer (String) here is how you do:

  • Spread: take the spread as is
  • Discard: take the last slice of the spread
  • Enqueue: take the new Enqueue (Raw) node, which collects incoming spreads of raw and always only returns one of the collected slices per frame
Classic scenario tokenizing by CRLF as postfix. Easy.

Also, on the other side, if you're in the business of sending out a stream of bytes here are the counterpart nodes to frame your messages accordingly:

  • PrefixLength (Raw)
  • PostfixBytes (Raw)
  • FrameBytes (Raw)

What the VL?

Of course. All patched in VL and even more practical to use over there because (once again) datatypes and delegates. The Tokenizer in VL is much more primitive in that it only collects all incoming bytes and then executes a delegates on the buffered bytes. The delegate allows you to easily implement more complex scenarios than the four preconfigured ones mentioned above.

Elegantly tokenizing Firmata with VL

The Firmata protocol for example fits non of the above mentioned simple cases as it has different types of tokens. Still the basic Tokenizer can be used to implement the firmata peculiarities on top of it. Further the delegate allows you to return the tokens already in your desired datatype. So instead of returning a Spread<Spread<Bytes>> as the Tokenizers in vvvv can only do, in VL it can readily return a Spread<MyToken> which is just so much more modern..

If you feel anything missing here or have any questions, please let us know in the comments!

The nodes are now available in Alpha Builds.

joreg, Tuesday, Aug 22nd 2017 Digg | Tweet | Delicious 0 comments  
  • 1

anonymous user login

Shoutbox

~10h ago

eno: @vux I'd like to become a patreon, but is there a possibility for a onetime donation? I'd rather pay 1x 1000 than 20x50.

~12h ago

Gareth.Griffiths: There's a really nice 3D text demo with the text being made from lines, anyone remember the name?

~19h ago

catweasel: @eglod, thats why many of us are working on timelines in VL...

~20h ago

eglod: Is there any possibility to copy and paste tracks in timelinerSA. I need it very much, is it very difficult to program this?

~1d ago

joreg: wanna make sure your projects will run with upcoming beta36? test them against the release candidate and report: beta36-release-candidate

~2d ago

colorsound: @vux thanks a lot for the update , my eternal gratitude.@ andresc4 @skyliner very happy you liked it. Big scale coming soon ;D

~2d ago

andresc4: I just gave my 2 cents tu Vux. BTW Laser things made by colorsound and dominikkoller are amazing!!!

~2d ago

dottore: thanks vux

~3d ago

~3d ago

Noir: big up vux