[Zope] utf-8 over xmlrpc with zope2.9

Kevin plone at kevinkal.com
Thu Sep 13 11:47:21 EDT 2007


Took a big step back and wrote a simple method to test adding some
ldap attributes via xml-rpc.  It's seems that there is some issue in our
'real' methods that is causing the strings to be treated as ascii, but
in the test method, everything is working well.

But now it becomes even stranger, if zope is run with zopectl fg, even 
the original methods work.  utf-8 strings remain encoded properly, but 
with zopectl start or runzope or runzope &, the method fails...same old 
error...I'll spare the whole traceback but the final error is

UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in 
position 4:
 > ordinal not in range(128)

Doyon, Jean-Francois wrote:
> Looks like a bug in the LDAPUserFolder implementation to me ... Seems to explicitely expect a regular string, and not a unicode one.
>
> What you could do is look at encoding the strings you get from XML-RPC with something like u''.encode('utf-8') (which would return a plain string).
>
> Or go digg in LDAPUserFolder and figure out how it's messing with the encoding of your strings.  LDAP v3 requires UTF-8 encoded strings as part of the spec.
>
> J.F. 
>
> -----Original Message-----
> From: zope-bounces at zope.org [mailto:zope-bounces at zope.org] On Behalf Of Kevin
> Sent: September 12, 2007 12:44
> To: zope at zope.org
> Subject: Re: [Zope] utf-8 over xmlrpc with zope2.9
>
> Doyon, Jean-Francois wrote:
>   
>> To be more explicit.
>>
>> The problem is that the strings received by XML-RPC where the encoding is utf-8 end up internally as unicode u'' strings.
>>
>> Other strings you're working with internally might not be however.
>>
>> When you're trying to render a ZPT, if you have a mix of u'' and '' strings, you might end up with such UnicodeDecodeErrors ...
>>
>>   
>>     
> we're checking this now, but what about print statements that are used for debugging other strings that are formed for return values?
> I would think that if the error were related to ZPT we would see the traceback lead to publisher.
>
> here's the traceback:
> Traceback (most recent call last):
>   File "/var/lib/zope001/Products/web_site/webXMLRPC.py", line 124, in reg isterUser
>
> status=reg.register(email,activate,returnto,firstname,lastname,middlename)
>   File "/var/lib/zope001/Products/web_site/RegistrationService.py", line 84, in register
>
> ldap.setAttributes(email,{'givenName':[firstname],'sn':[lastname],'middleNam
> e':[middlename]})
>   File "/var/lib/zope001/Products/web_site/LDAPServices.py", line 58, in set Attributes
>     result= self.userManage.manage_editUser(user_dn, REQUEST, attributes)
>   File "/var/lib/zope001/Products/LDAPUserFolder/LDAPUserFolder.py",
> line 1601,
> in manage_editUser
>   File "/var/lib/zope001/Products/LDAPUserFolder/LDAPDelegate.py", line 500, in modify
>     values = map(to_utf8, values)
>   File "/var/lib/zope001/Products/LDAPUserFolder/utils.py", line 101, in
> to_utf8
>     return encodeUTF8(decodeLocal(s)[0])[0]
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 4:
> ordinal not in range(128)
>   
>> In my case, I make sure I use u'' strnigs everywhere, and set character sets to utf-8 everywhere, and so on.
>>
>>   
>>     
>
>   
>> Basically if needed you can make your whole system utf-8, avoiding the risks of those errors cropping up.
>>
>> J.F. 
>>
>> -----Original Message-----
>> From: zope-bounces+jdoyon=nrcan.gc.ca at zope.org 
>> [mailto:zope-bounces+jdoyon=nrcan.gc.ca at zope.org] On Behalf Of Doyon, 
>> Jean-Francois
>> Sent: September 11, 2007 14:01
>> To: Kevin; zope at zope.org
>> Subject: RE: [Zope] utf-8 over xmlrpc with zope2.9
>>
>> Should work, I use XML-RPC pretty heavily with utf-8 without much problems.
>>
>> This error might be occurring because you're mixing unicode and non-unicode strings on the same page.
>>
>> Send the full traceback and we'll know more ...
>>
>> J.F. 
>>
>> -----Original Message-----
>> From: zope-bounces at zope.org [mailto:zope-bounces at zope.org] On Behalf 
>> Of Kevin
>> Sent: September 11, 2007 13:56
>> To: zope at zope.org
>> Subject: [Zope] utf-8 over xmlrpc with zope2.9
>>
>> I am trying to send unicode encoded utf-8 or latin1 strings over 
>> xmlrpc to a method in zope.  The method in zope tries to decode the 
>> string and it seems detects ascii.  I then recieve an error that
>> "UnicodeDecodeError: 'ascii' codec can't decode byte 0xf3 in position 4: 
>> ordinal not in range(128)"
>>
>> I can re-create this error in a python shell by doing something like:
>>  >>> s = 'helló'
>>  >>> s2 = unicode(s,'ascii')
>> so after looking at code, I assume this is what is happening instead 
>> of  >>> s2 = unicode(s,'iso-8859-1')
>>
>> If  I  submit  to this method from  an html form, I have no error and the data is stored perfectly.
>>
>> I have tried setting the encoding of xmlrpclib.Server to utf-8 and
>> latin1(iso-8859-1) and making a custom transport for xmlrpclib where I add charset='utf-8' or charset='iso-8859-1'.
>>
>> Is is not possible to send unicode characters > 128 to zope?
>>
>>
>> The method is LDAPUserFolder.manage_edit_user()
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Zope maillist  -  Zope at zope.org
>> http://mail.zope.org/mailman/listinfo/zope
>> **   No cross posts or HTML encoding!  **
>> (Related lists -
>>  http://mail.zope.org/mailman/listinfo/zope-announce
>>  http://mail.zope.org/mailman/listinfo/zope-dev ) 
>> _______________________________________________
>> Zope maillist  -  Zope at zope.org
>> http://mail.zope.org/mailman/listinfo/zope
>> **   No cross posts or HTML encoding!  **
>> (Related lists -
>>  http://mail.zope.org/mailman/listinfo/zope-announce
>>  http://mail.zope.org/mailman/listinfo/zope-dev )
>>   
>>     
>
>
> _______________________________________________
> Zope maillist  -  Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev )
>   




More information about the Zope mailing list