Help concatenates the incoming string stream and splits out defined tokens
Author vvvv group

Credits
Warnings
Bugs

related discussions

start a forum thread about this node

about Tokenizer (String)

FAQ

What is this Tokenizer for?

Its for communicating receiving Data over a serial or IP line.
They key to have 100%error free transmission is understanding that the commands which get sent in one bunch from the device are not guaranteed to arrive in vvvv in the exact same frame. Depending on the speed of both perties there might be one message, two or more messages, or only parts of messages received.
To solve that problem vvvv has the Tokenizer (String) node, which allows queuing up received messages until they are complete and then delivering them one by one (or as a spread). Make sure your sender sends something like a message end character after each message (usually a carriage return 13, but it can be anything) and make the Tokenizer split after that characters. In fact most off-the-shelf serial devices or internet protocols have provisions for this, as it is a general problem.

Not using this technique will principally lead to spurious random glitches with their probability depending on a lot of factors..

What is the enqueue mode for?

if you will receive a complete packet or even multiple packets during one frame depends on the frame rate of your patch and the frame rate of your sending party. therefore you will like to split the incoming stream of data into so called Tokens.

The QueueLength output of the tokenizer will report the received characters which were not output yet. This should always stay under the maximum length of your Tokens. if it rises forever something in your patch might be wrong - for example if the incoming strings would never contain the given Delimiter.

Depending on the ratio between the senders frame rate and the vvvv frame rate on the receiving end, you should decide on the QueueMode of the Tokenizer.

Enqueue output always one complete packet per frame - all other packets are saved in the Tokenizer, and get output in a later frame. this is fine if the frame rate of your patch is definitely higher then the packet rate of the incoming data, or if the data arrives only in bursts. Obviously if your sender sends 100 message per seconds and the Tokenizer outputs only 60 messages per second, the data will queue up in the tokenizer (therfore the name) up potentially until you run out of memory. It will also faithfully spit out all the queued messages when the input stops. So observe the QueueLength pin, as it will notify you of possible delays.

If the Queue Mode is set to Discard all packets, that cant be delivered right now will be discarded. So you would be sure to never get a delay, but certain data packets may be missing. So if your device would send absolute values, this might be the preferable way.

The most flexible mode is the Spread mode, which outputs all complete packets which were decoded in one frame as one spread of strings. So you can basically decide what to do with multiple packets which get received during one frame - like if you want them added up (ideal for relative data) or averaged, maxed, parsed separate or whatever.