{"issue":{"created_at":"2011-09-26T10:52:18Z","state":"closed","updated_at":"2011-11-19T18:14:57Z","user":{"gravatar_id":"6dcb96a9b44bb7f04fa4e4542c502075","id":111141,"url":"https://api.github.com/users/ferd","login":"ferd","avatar_url":"https://secure.gravatar.com/avatar/6dcb96a9b44bb7f04fa4e4542c502075?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"comments":13,"body":"This issue is derived from a bug we have in socket.io-erlang, which turns out to be a problem with misultin. \r\n\r\nSomeone reported to us that strings would fail when containing complex unicode characters and sent us some sample. The user would input a string like \"~m~7~m~Привет!\" and have crashes happening all the time.\r\n\r\nLooking at the stack trace he'd send us, I saw the following data in the request body: `<<\"data=%7Em%7E7%7Em%7E%D0%9F%D1%80%D0%B8%D0%B2%D0%B5%D1%82%21\">>`. Decoding it with misultin, it returns:\r\n\r\n    3> misultin_utility:parse_qs(S).\r\n    [{\"data\",\r\n       [126,109,126,55,126,109,126,208,159,209,128,208,184,208,178,208,181,209,130,33]}]\r\n    4> io:format(\"~ts~n\",[[126,109,126,55,126,109,126,208,159,209,128,208,184,208,178,208,181,209,130,33]]).              \r\n    ~m~7~m~ÐÑÐ¸Ð²ÐµÑ!\r\n    ok\r\n    5> io:format(\"~ts~n\", [list_to_binary([126,109,126,55,126,109,126,208,159,209,128,208,184,208,178,208,181,209,130,33])]).\r\n    ~m~7~m~Привет!\r\n    ok\r\n\r\nThe issue, as far as I can see, is that the unicode is parsed as a binary, which turns all code points in bytes (0..255). Then the binaries are blindly turned into lists, but they don't have the same unicode format in Erlang -- you actually need to convert them to codepoints greater than 255, and then output them with a `~ts` combination instead of just `~s`. \r\n\r\nThe problem is that you're likely using `binary_to_list` to convert strings, but when you have unicode data with grapheme clusters, what you need is something more like this:\r\n\r\n    6> io:format(\"~ts~n\",[unicode:characters_to_list(<<126,109,126,55,126,109,126,208,159,209,128,208,184,208,178,208,181,209,130,33>>)]).\r\n    ~m~7~m~Привет!\r\n    ok\r\n\r\nThis use of `unicode:characters_to_list(Str)` will work with IO lists and convert to whatever format you use. Note that the strings returned might now become invalid IO lists and should need to be converted back with `unicode:characters_to_binary/1` before being pushed back in a socket.","assignee":null,"id":1738522,"pull_request":{"patch_url":null,"html_url":null,"diff_url":null},"number":61,"labels":[],"title":"Critical issue with Unicode parsing","closed_at":"2011-11-19T18:14:57Z","html_url":"https://github.com/ostinelli/misultin/issues/61","milestone":null,"url":"https://api.github.com/repos/ostinelli/misultin/issues/61"},"action":"closed"}