» Tokenizer (String)
This site relies heavily on Javascript. You should enable it if you want the full experience. Learn more.

Tokenizer (String)

Help concatenates the incoming string stream and splits out defined tokens
Author vvvv group

Here you are supposed to see an image (rendered with SVG) describing the pins of this node. However, this does not work with Internet Explorer. You may want to consider using another browser, like FireFox, Chrome, Safari or Opera that do support SVG.

Credits
Warnings
Bugs

related discussions

start a forum thread about this node

about Tokenizer (String)

FAQ

What is this Tokenizer for?

Its for communicating receiving Data over a serial or IP line.
They key to have 100%error free transmission is understanding that the commands which get sent in one bunch from the device are not guaranteed to arrive in vvvv in the exact same frame. Depending on the speed of both perties there might be one message, two or more messages, or only parts of messages received.
To solve that problem vvvv has the Tokenizer (String) node, which allows queuing up received messages until they are complete and then delivering them one by one (or as a spread). Make sure your sender sends something like a message end character after each message (usually a carriage return 13, but it can be anything) and make the Tokenizer split after that characters. In fact most off-the-shelf serial devices or internet protocols have provisions for this, as it is a general problem.

Not using this technique will principally lead to spurious random glitches with their probability depending on a lot of factors..

What is the enqueue mode for?

if you will receive a complete packet or even multiple packets during one frame depends on the frame rate of your patch and the frame rate of your sending party. therefore you will like to split the incoming stream of data into so called Tokens.

The QueueLength output of the tokenizer will report the received characters which were not output yet. This should always stay under the maximum length of your Tokens. if it rises forever something in your patch might be wrong - for example if the incoming strings would never contain the given Delimiter.

Depending on the ratio between the senders frame rate and the vvvv frame rate on the receiving end, you should decide on the QueueMode of the Tokenizer.

Enqueue output always one complete packet per frame - all other packets are saved in the Tokenizer, and get output in a later frame. this is fine if the frame rate of your patch is definitely higher then the packet rate of the incoming data, or if the data arrives only in bursts. Obviously if your sender sends 100 message per seconds and the Tokenizer outputs only 60 messages per second, the data will queue up in the tokenizer (therfore the name) up potentially until you run out of memory. It will also faithfully spit out all the queued messages when the input stops. So observe the QueueLength pin, as it will notify you of possible delays.

If the Queue Mode is set to Discard all packets, that cant be delivered right now will be discarded. So you would be sure to never get a delay, but certain data packets may be missing. So if your device would send absolute values, this might be the preferable way.

The most flexible mode is the Spread mode, which outputs all complete packets which were decoded in one frame as one spread of strings. So you can basically decide what to do with multiple packets which get received during one frame - like if you want them added up (ideal for relative data) or averaged, maxed, parsed separate or whatever.

anonymous user login

Shoutbox

~1d ago

benjawmino: so are there no splines/curves in gamma yet??

~5d ago

skyliner: wanna do drone shows or applications? then check this super cool project of our man e1n

~9d ago

NoseBleedIndustries: Thanks Joreg! The few minutes I was able to see, very good workshops!

~9d ago

joreg: @NoseBleedIndustries please give us some days, we'll have an announcement soon...

~9d ago

NoseBleedIndustries: I could not assist the Node20 (workshops ) Any Idea when we will have access to the links of the recordings?

~11d ago

bjoern: unity has c# bindings for usd, under apache license: https://github.com/Unity-Technologies/usd-unity-sdk

~19d ago

ravazquez: @synth yes they are being recorded and will be available for future consumption

~19d ago

synth: Another stupid question: Are all #NODE20 sessions recorded and accessible for later viewing in case someone missed something?