[2] This line change from 'iso-8859-1' to 'utf-8' lib/python/App/dtml/manage_page_header.dtml <dtml-call "REQUEST.set('management_page_charset','iso-8859-1')">

Bad news. That will cause all management form submissions to encode strings in utf8. 99% of the methods to which the strings are being submitted will not be expecting this, and will corrupt characters whose unicode code point is >127.

If you have a ZMI form that *is* expecting this then you need to make some other changes to avoid breakage. Essentially just adding :utf8: marshalling tags, possibly some :strings: and :ustring: too.

(yes, this sucks. The problem is that browsers dont specify the character encoding used in form submissions. At some point we need to discuss the way forward on this issue....)

[1] These 3 filses (total 3 line) change from "encode('latin1')" to "encode('utf-8')" $find . -name '*.py' -exec grep -l 'encode.*latin1' {} ; ./lib/python/ZPublisher/Converters.py ./lib/python/ZPublisher/HTTPRequest.py ./lib/python/ZPublisher/HTTPResponse.py

Even more bad news. Suppose a dtml page it not yet prepared to handle unicode (because it hasnt had the changes described above) but it 'accidentally' encounters a unicode attribute. This happens often in the ZMI when objects have a unicode 'title', because many products render the title attribute of *other* objects. We cant force the response to utf8 because this will cause the same breakage described above.

For more details see:

http://www.zope.org/Members/htrd/howto/unicode http://www.zope.org/Members/htrd/howto/unicode-zdg-changes

But I have not enough test.

obviously. ;-)

I want mechanism to change encoding dinamically.

the manage_properties page that Arnar Lundesgaard has been working with is a good example. It switches between latin-1 and utf-8 automatically depending on whether any unicode properties have been defined. (to support *really* old browsers that dont understand utf8)

...

(yes, this sucks. The problem is that browsers dont specify the character encoding used in form submissions. At some point we need to discuss the way forward on this issue....)

There is a standard accept-charset attribute of forms, which says what encodings are accepted by the form handler (Zope here). I think we should use it and set it to UTF-8 in those cases.

Just to be clear, this is an HTML attribute of the <form> tag. For instance:

<form action="foo" ... accept-charset="UTF-8"> ... </form>

This instructs the browser it should send the content of the form in the accepted charset. As a default, it is recommended that user agents use the encoding of the document, but this is not a strict requirement in HTML4.

Florent

On Mon, 2002-09-30 at 09:17, Toby Dickenson wrote:

On Saturday 28 Sep 2002 4:38 pm, Florent Guillaume wrote:

...

<form action="foo" ... accept-charset="UTF-8"> ... </form>

This instructs the browser it should send the content of the form in the accepted charset.

Yes, accept-charset could be part of a full solution to this problem, but I dont think it is a whole solution....

Are you suggesting that a method could assume its form submissions would always be made in utf-8? That would cause problems if a submission was made from:

• some other form that didnt have an accept-charset
• some non-browser code that synthesizes http requests

Yes, there is no good way.

A further problem is that we want this decoding to be performed in ZPublisher, but that that point in the publishing process it doent know which method is going to be called. That means the utf8 assumption cant be made independantly for each method.

One answer to this problem is when browsers include the charset attribute in "multipart/form-data" POST requests. ZPublisher knows unambiguously what encoding was used by the browser.

This really sucks, you'd think that by 2002 all recent browsers would send a content-type:text/plain;charset=foobar in multipart/form-data, as the spec (from 1998) recommends... But even Mozilla 1.1 doesn't do it.

Sadly I cant see a nice way to do the same for GET requests

As an aside, one interesting tidbit about Mozilla: if you paste Unicode into a field of a form without an explicit encoding (accept-charset or document-charset), it encodes Unicode characters into &#xxxx; and sends that on the wire.

Anyway, in the near future I see no alternative to putting :utf8: into field names, and using accept-charset="UTF-8" or an utf-8 encoding for the document.

Florent