» New Tokenizer Nodes
This site relies heavily on Javascript. You should enable it if you want the full experience. Learn more.

New Tokenizer Nodes

Dear jugglers of the bytes,

sorry this took so long! Almost five years ago we introduced the new datatype Raw for easier handling of byte streams in vvvv. It came with all the nodes you'd need, except probably the most important one, a Tokenizer. So for collecting incoming bytes via e.g. TCP or RS232 and making sure to separate them into the correct message-tokens you'd still have to fall back to the good old Tokenizer (String). Possible, but annoying.

Enter the new series of Tokenizer nodes:

  • Tokenizer (Raw FixedLength)
  • Tokenizer (Raw LengthPrefix)
  • Tokenizer (Raw Postfix)
  • Tokenizer (Raw Frame)

Users of Tokenizer (String) may remember that it was always a bit tricky to configure since it had quite a few options to configure it and you'd have to make sure to get those all right for your specific use-case. So now we've separated those use-cases and spent each of them an individual node. The nodes versions should be self-explanatory. If not, they all come with help-patches!

Now all of the Tokenizers always return a spread of tokens found in the last frame. So in order to simulate the Queue Mode of the original Tokenizer (String) here is how you do:

  • Spread: take the spread as is
  • Discard: take the last slice of the spread
  • Enqueue: take the new Enqueue (Raw) node, which collects incoming spreads of raw and always only returns one of the collected slices per frame
Classic scenario tokenizing by CRLF as postfix. Easy.

Also, on the other side, if you're in the business of sending out a stream of bytes here are the counterpart nodes to frame your messages accordingly:

  • PrefixLength (Raw)
  • PostfixBytes (Raw)
  • FrameBytes (Raw)

What the VL?

Of course. All patched in VL and even more practical to use over there because (once again) datatypes and delegates. The Tokenizer in VL is much more primitive in that it only collects all incoming bytes and then executes a delegates on the buffered bytes. The delegate allows you to easily implement more complex scenarios than the four preconfigured ones mentioned above.

Elegantly tokenizing Firmata with VL

The Firmata protocol for example fits non of the above mentioned simple cases as it has different types of tokens. Still the basic Tokenizer can be used to implement the firmata peculiarities on top of it. Further the delegate allows you to return the tokens already in your desired datatype. So instead of returning a Spread<Spread<Bytes>> as the Tokenizers in vvvv can only do, in VL it can readily return a Spread<MyToken> which is just so much more modern..

If you feel anything missing here or have any questions, please let us know in the comments!

The nodes are now available in Alpha Builds.

joreg, Tuesday, Aug 22nd 2017 Digg | Tweet | Delicious 0 comments  
  • 1

anonymous user login

Shoutbox

~18h ago

joreg: Two #vvvv workshops this week in #berlin: Friday: Physical Computing, Saturday: Computer Vision. Tickets: upcoming-full-day-vvvv-gamma-workshops-in-berlin

~1d ago

joreg: @cznickesz also feel free to join our chat for such questions: chat

~1d ago

cznickesz: @joreg: Yeah, I´ll give it a try! Thx

~1d ago

joreg: @cznickesz: we're hoping this week. but depends on feedback. please test your project and let us know if it works.

~2d ago

cznickesz: @joreg: I should stop asking my questions so cryptic ;-) So right away: is final Beta39 hours, days or weeks away?

~2d ago

joreg: @cznickesz: the idea of an RC is exactly to make sure your projects will work with it. your last chance to report problems with b39.

~2d ago

cznickesz: So how "RC" is the "RC3"? I have to update an old project soon and I´m wondering if I should wait and directly update to Beta39

~2d ago

joreg: @takuma, the "one frame" issue was fixed in RC3 for beta39, see: beta39-release-candidate

~2d ago

Takuma: @joreg thank!! Ill try this. Are there a way to send video texture from vvvv to vl? fromimage node in vl somehow only sends 1 frame

~2d ago

cznickesz: @ravasquez thx, so I need to stop avoiding nuget in the end ;-)