[Zope3-checkins] SVN: Zope3/trunk/src/zope/publisher/ Many mainstream browsers don't send an HTTP_ACCEPT_CHARSET header.

[Philipp von Weitershausen]

Log message for revision 25999:
Many mainstream browsers don't send an HTTP_ACCEPT_CHARSET header.
zope.publisher uses this header to deduce the encoding of form values;
if this header is missing though, it didn't convert them at all to unicode.

Since Zope's fallback is 'UTF-8' everywhere whenever an encoding is not
specified, it should also fallback to trying to decode incoming form
data as UTF-8.

Added a test to verify this behaviour.

Thanks to Marius and Bjorn for their advice.



-=-
Modified: Zope3/trunk/src/zope/publisher/browser.py
===================================================================
--- Zope3/trunk/src/zope/publisher/browser.py	2004-06-29 15:26:31 UTC (rev 25998)
+++ Zope3/trunk/src/zope/publisher/browser.py	2004-06-29 17:26:35 UTC (rev 25999)
@@ -229,7 +229,7 @@
         """Try to decode the text using one of the available charsets."""
         if self.charsets is None:
             envadapter = IUserPreferredCharsets(self)
-            self.charsets = envadapter.getPreferredCharsets()
+            self.charsets = envadapter.getPreferredCharsets() or ['utf-8']
         for charset in self.charsets:
             try:
                 text = unicode(text, charset)

Modified: Zope3/trunk/src/zope/publisher/tests/test_browserrequest.py
===================================================================
--- Zope3/trunk/src/zope/publisher/tests/test_browserrequest.py	2004-06-29 15:26:31 UTC (rev 25998)
+++ Zope3/trunk/src/zope/publisher/tests/test_browserrequest.py	2004-06-29 17:26:35 UTC (rev 25999)
@@ -208,6 +208,20 @@
         self.assertEqual(request.form,
                          {u'a':u'5', u'b':6})
 
+    def testFormNoEncodingUsesUTF8(self):
+        encoded = 'K\xc3\x83\xc2\xb6hlerstra\xc3\x83\xc2\x9fe'
+        extra = {
+            # if nothing else is specified, form data should be
+            # interpreted as UTF-8, as this stub query string is
+            'QUERY_STRING': 'a=5&b:int=6&street=' + encoded
+            }
+        request = self._createRequest(extra)
+        # many mainstream browsers do not send HTTP_ACCEPT_CHARSET
+        del request._environ['HTTP_ACCEPT_CHARSET']
+        publish(request)
+        self.assert_(isinstance(request.form[u'street'], unicode))
+        self.assertEqual(unicode(encoded, 'utf-8'), request.form['street'])
+
     def testFormListTypes(self):
         extra = {'QUERY_STRING':'a:list=5&a:list=6&b=1'}
         request = self._createRequest(extra)