{"issue":{"milestone":null,"user":{"id":212107,"login":"asl2","url":"https://api.github.com/users/asl2","gravatar_id":"0033cf0774f6eea956f096a1e10cb999","avatar_url":"https://secure.gravatar.com/avatar/0033cf0774f6eea956f096a1e10cb999?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"title":"Item hashes differ between unicode and str, and 32 / 64 bits","created_at":"2011-12-19T22:44:13Z","labels":[],"updated_at":"2011-12-20T04:03:59Z","state":"open","id":2606984,"pull_request":{"patch_url":null,"html_url":null,"diff_url":null},"html_url":"https://github.com/axiak/pybloomfiltermmap/issues/16","closed_at":null,"body":"I noticed that unlike the built-in containers, pybloomfiltermmap differentiates between unicode and str, eg\r\n<pre>\r\n>>> from pybloomfilter import BloomFilter\r\n>>> filter = BloomFilter(100000, 0.01, '/tmp/bloomfilter')\r\n>>> filter.add(u'foo')\r\nFalse\r\n>>> 'foo' in filter\r\nFalse\r\n>>> u'foo' in filter\r\nTrue\r\n>>> 'foo' in set([u'foo'])\r\nTrue\r\n</pre>\r\n(The only documentation I can find for this is [PEP-100] [1], which makes no guarantees about hashes of unicode containing non-ascii characters.)\r\n[1]:http://www.python.org/dev/peps/pep-0100/ \"PEP-100\"\r\n\r\nI also noticed that the default python hash on 32-bit python is the low 32 bits of the hash on 64-bit python.  For example, on 32-bit:\r\n<pre>\r\n>>> hex(hash(\"foo\"))\r\n'-0x2c217945'\r\n</pre>\r\nvs 64-bit:\r\n<pre>\r\n>>> hex(hash(\"foo\"))\r\n'-0x39f8634c2c217945'\r\n</pre>\r\nThis caused me some head-scratching: my workaround was to encode unicode strings via utf-8, so they'd go through the string hashing instead of python hashing, and it works splendidly now.\r\n\r\nAt the next file-format incompatibility, it might be nice to hash unicode strings via PyString_AsStringAndSize, and mask off the top 32 bits on 64-bit systems. ","number":16,"url":"https://api.github.com/repos/axiak/pybloomfiltermmap/issues/16","comments":1,"assignee":null},"comment":{"user":{"id":110029,"login":"axiak","url":"https://api.github.com/users/axiak","gravatar_id":"c2233920bcc1ab0f8a877474c0ccdc4b","avatar_url":"https://secure.gravatar.com/avatar/c2233920bcc1ab0f8a877474c0ccdc4b?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"created_at":"2011-12-20T04:03:59Z","updated_at":"2011-12-20T04:03:59Z","id":3214134,"body":"This is done because hashing strings is separate from hashing unicode strings for performance. I'm not sure how to fix this problem since I don't know what encoding the strings are in, so I can't hope to have the raw bytes in the correct format. I'll think about this issue.","url":"https://api.github.com/repos/axiak/pybloomfiltermmap/issues/comments/3214134"},"action":"created"}