seems good news for patcher in Japan... how many? ;-)

are the database nodes broken, too?

and, what is the exact reason for that issue?

@u7angel: please test and report

@roger: as bjoern said. they'll have to be rewritten which shouldn't be too much of an issue seeing all the other database nodes implemented by vux...

@sebl: they shouldn't be. please report if you find troubles

Tested it on lots of strings (> 50000 in a spread, different languages, including french). Autodetect encoding didn't work (but they are utf8 without BOM, which is hard - but Notepad++ gets it right by heuristics). But after I selected utf8, all IOBoxes with strings showed the french characters correct (had to manually select encoding for them before).

BUT: just found a bug, I think. Reading in an XML file with encoding set to utf8 in the Reader, the XML nodes do not work anymore. If I set it to auto, they do, but comparing the strings extracted from the XML (now in cp1252 or so) with the lot of 50k utf8 strings fails for all the french characters. Which means that to get it to work again I had to switch all Readers to Auto => all the comparing works, but the string IOBoxes do not show french characters correctly, as everything isn't utf8 internally.

So, UTF8 works great, but breaks the newly-introduced XML nodes. At least with UF8 without BOM.
I made two patches, one in 28-2 and nearly the same in 28-3-unicorn, so see for yourself.
http://dl.dropbox.com/u/36620736/WEB/VVVV/Encoding_XML_Bug.7z

And a request: please show what encoding Reader thought the file was. Makes debugging easier, I think, as "auto" does not work for everything (e.g. no BOM, which is the official standard for utf8 as far as I know).

Great, the XML reading is working now with the latest unicorn build.

One thing I'm definitely missing though: any way to tell the writer (or some other text-out-node node) "don't touch anything, leave every single bit as it was". Or, when does that happen? If I explicitly set the writer to utf8, as that's the internal format? As I understand, it crushes everything it does not understand as utf8 to some "unknown" character.

To clarify: what if I want to read some binary file with the Reader, and write it back to disk after some operations? Or, much more probable, want to fetch some images from the web and save them? Or fetch some binary data and save it? Reader needs a "raw" encoding, or an "Encode" toggle.

I attached two new scenes to test - one reads & writes an image, the other one fetches an image from the web. I wasn't able to save either one back to disk in the correct format, it was always garbled (and definitely no image anymore).

http://dl.dropbox.com/u/36620736/WEB/VVVV/Encoding_images.7z

Please do not assume that everyone uses Reader & Writer for text files only :).

Edit:
I think selecting the encoding at input/output and not where the content is used/generated may be a design flaw. In current beta, you read a file, you write a file, and every single byte simply goes through. If you want some specific encoding, you tell the IOBox what you want, and you can switch between encodings with Convert before you send it to disk, to web or to some device. Power where power is needed. And now, you are forced to select the encoding at the in and outs and hope that nothing gets destroyed inbetween - what does + (String) do internally now, for example? Does it convert every character to utf8? What if I want to construct commands for devices, byte by byte, as possible before? What if I want to use ord and char? And so on, and so on... I hope you will answer all of these questions before finally releasing, because it WILL cause a lot of confusion if strings do not behave like "appending bytes to some other bytes" anymore.

sorry for not ranting earlier.. i completely agree with herbst here..

so you are saying the beauty of misusing strings as arrays of byte (and the didactical beauty of having the difference between the two only in the visualization) is for now changed into a errorprone and subtle conversion step at all input and output nodes for strings? this doesnt really sound like an advantage tp me.

and without the Byte node type (and the bunch of duplicated nodes like +(byte), the unicorn update will mostly disallow any clean binary data manipulation, right?

this surely sounds like a showstopper, right?

so please sell me some of the advantages of the whole thing - unicode with utf8 was supported for ages.. so whats the big plus?

Well, this is for sure going to take a while until it sinks into heads...

@joreg: I did what you suggested and changed the encoding to cp1252, and both examples work. Then I sat there, starring at my patch. I understand why it works, but for that you have to know all the internal and tricky details of encoding - the patch as it is (an image read and written with cp1252) is just ridiculous.

(Wouldn't it be easier to implement (and easier to understand) a "raw bytes" encoding method in Reader than to build entirely new nodes? I don't think that a change that is meant to make life easier is supposed to introduce dozens of new nodes (an entire new node class!).)

And I would like to suggest to make the encoding pin visible by default. Asking yourself "why the heck can't I store downloaded images anymore?" isn't great, you would basically force everyone to google and find this post here with explanations (be it "use Reader (bytes)" or "set the hidden encoding pin to cp1252").