» Tokenizer (String)
This site relies heavily on Javascript. You should enable it if you want the full experience. Learn more.

Tokenizer (String)

Help concatenates the incoming string stream and splits out defined tokens
Author vvvv group

Here you are supposed to see an image (rendered with SVG) describing the pins of this node. However, this does not work with Internet Explorer. You may want to consider using another browser, like FireFox, Chrome, Safari or Opera that do support SVG.


related discussions

start a forum thread about this node

about Tokenizer (String)


What is this Tokenizer for?

Its for communicating receiving Data over a serial or IP line.
They key to have 100%error free transmission is understanding that the commands which get sent in one bunch from the device are not guaranteed to arrive in vvvv in the exact same frame. Depending on the speed of both perties there might be one message, two or more messages, or only parts of messages received.
To solve that problem vvvv has the Tokenizer (String) node, which allows queuing up received messages until they are complete and then delivering them one by one (or as a spread). Make sure your sender sends something like a message end character after each message (usually a carriage return 13, but it can be anything) and make the Tokenizer split after that characters. In fact most off-the-shelf serial devices or internet protocols have provisions for this, as it is a general problem.

Not using this technique will principally lead to spurious random glitches with their probability depending on a lot of factors..

What is the enqueue mode for?

if you will receive a complete packet or even multiple packets during one frame depends on the frame rate of your patch and the frame rate of your sending party. therefore you will like to split the incoming stream of data into so called Tokens.

The QueueLength output of the tokenizer will report the received characters which were not output yet. This should always stay under the maximum length of your Tokens. if it rises forever something in your patch might be wrong - for example if the incoming strings would never contain the given Delimiter.

Depending on the ratio between the senders frame rate and the vvvv frame rate on the receiving end, you should decide on the QueueMode of the Tokenizer.

Enqueue output always one complete packet per frame - all other packets are saved in the Tokenizer, and get output in a later frame. this is fine if the frame rate of your patch is definitely higher then the packet rate of the incoming data, or if the data arrives only in bursts. Obviously if your sender sends 100 message per seconds and the Tokenizer outputs only 60 messages per second, the data will queue up in the tokenizer (therfore the name) up potentially until you run out of memory. It will also faithfully spit out all the queued messages when the input stops. So observe the QueueLength pin, as it will notify you of possible delays.

If the Queue Mode is set to Discard all packets, that cant be delivered right now will be discarded. So you would be sure to never get a delay, but certain data packets may be missing. So if your device would send absolute values, this might be the preferable way.

The most flexible mode is the Spread mode, which outputs all complete packets which were decoded in one frame as one spread of strings. So you can basically decide what to do with multiple packets which get received during one frame - like if you want them added up (ideal for relative data) or averaged, maxed, parsed separate or whatever.

anonymous user login


~7d ago

andresc4: gamma+stride+fuse+standaloneapp+soputoutput+oscinput = lovvvvvveeeeeee!!!!

~9d ago

joreg: @pechart thanks for the flowers!

~10d ago

pechart: Just a big shoutout to the whole vvvv devteam and the friendly community! It's fantastic to export patches to standalone apps. THX!

~18d ago

daviddenker: We are looking asap for 3x Azure Kinect to rent or buy. Any kind of help appreciated! :-)

~21d ago

mediadog: Hmmm md.ecosystem link on staff pics page is not found

~23d ago

schlonzo: physics constraints and forces plz?

~25d ago

karistouf: thank you @gml but im thinking about and old version of vvvvv. There was a redraw with HSL of each pixels

~26d ago

schlonzo: no more previews? is there a 4.0 stable?

~27d ago

sunep: Live now. I perform live AV with Thomas Li after kiloton who is performing now. https://youtu.be/IG5oC6JvMeM

~27d ago

ggml: @karistouf ColorAndDepth help file