Tokenizer Bug

Hi all,

I´m sending from arduino characters through the RS232 to vvvv.
As you can see in the image attached I use tokenizer for taking the data and later the AsValue node, all of this for taking the position of a stepper motor.

What I´ve realized in a certain moment is that Tokenizer throughs an empty string (almost invisible) and later keeps working good, this spurious empty strings are represented as a ‘zero’ -> ‘0’ in the output of the AsValue node.

I send this:
23z23z23z23z23z23z23z…

I receive this in hypertrm:
23z23z23z23z23z23z23z…

I receive this in vvvv:
23 23 23 23 0 23 23 23 23 23 23 0 23 23 23 0 23 23 23 …

As you can see I have put a S+H which will avoid the empty strings

I think that this error depends on the frecuency in which I send the values…
Another point to say is that I connect arduino with a 115200 baud rate.

Is this a bug? or something I´m missing?
In the image you can see the two scenarios (one with the bug and the other free of bug)

Just thought to report it. I dont think tokenizer should do that…

could you upload the images once more pls?

yep, I forget to click in the insert button… Newbie things ;)

your solution looks fine to me. your suggestion would mean to put some part of your patched logic inside of the tokenizer node. that might be ok in your case but in other use cases one would like to have a different logic. so being able to patch that part is a good thing.
the “smarter” a node is, the harder it is to reuse it.
if you think your approach is the way to go, put it inside of a module and add that module to your personal node library or even make a little contribution out of it and share it with others :)

well, thanks for the answer! :) But I think I´ve not been very clear explaining the reason of this post… The point is that Tokenizer should never throw an empty string… so it´s supposed to be a bug.

If the node ‘+’ drops a result of 4 when making ‘2+1’ it´s obviously wrong, well I think Tokenizer does not work as expected… or am I wrong?

Can you tell me why tokenizer in that case throws empty strings?
Would you like me to send a vvvv file with the scenario for testing?
Only want to report the bug :s

Best Regards

or… maybe there is a way to modify the source code of tokenizer? or maybe post it here? in that way I could help.

i didn’t write that node so i can’t tell you why in particular it outputs an empty string but what else would you like it to output? the last token it saw? but then you got a different behaviour for the start of your patch as there wasn’t any token to be seen. and as you’ve already shown such a behaviour can easily be patched.
btw. you talk about empty strings in your posts but in your screenshot it shows nil (which is an empty spread). i suppose you set the tokenizer queue mode to spreads then?

Ammm, I get it now.

When Tokenizer receive “something” which does not have a ‘z’ in this case, the output throws an empty string, queueing the value for later…

In this case I´m sending the number ‘2’ and the output goes to an empty string, later I put ‘23z’ and tokenizer throws the number 2 plus the 23z -> ‘222222223z’ but if I keep sending ‘23z’ the output remains then in ‘23z’
(222222223z is only for the first frame when the interruptor goes ‘1’)

I think then this is not a bug but interesting speech cause RS232 node splits sometimes the received characters in two frames… So if I send 23z… RS232 will sometimes get ‘2’ and the other frame ‘3z’ thats why the output throws an empty string depending of the velocity of transmission…

You are true, Tokenizer can be used in other scenarios where this might help… I hope this will help RS232 user for understanding how to catch the values.

Thanks Elias!

PD: About the spreads, the image of the first post was not running the program, thats why the nill values