UTF-8 Lines Generated Using Restlet on GAE

Question

I have a simple Restlet service hosted on AppEngine. This performs basic CRUD operations with strings and works well with all types of UTF-8 characters when I test it with curl (for all verbs).

This is consumed by a simple client client hosted in a servlet in another AppEngine application:

// set response type resp.setContentType("application/json"); // Create the client resource ClientResource resource = new ClientResource(Messages.SERVICE_URL + "myentity/id"); // Customize the referrer property resource.setReferrerRef("myapp"); // Write the response resource.get().write(resp.getWriter()); 

Above all that I have in the servlet. Very simple.

The servlet is called via jquery ajax, and the json that I return is well formed and that’s it, but the problem is that the UTF-8 encoded strings are returned scrambled, for example: Université de Montréal becomes Universit?? de Montr??al Universit?? de Montr??al .

I tried adding this line to the servlet (first of all):

 resp.setCharacterEncoding("UTF-8"); 

But the only difference is that instead of getting ?? I get Universitￃﾩ de Montrￃﾩal (I don’t even know which characters are what I assume).

I am 100% sure that the break service is ok, because besides debugging it line by line, I can check it from cmd line using curl and return well formed lines.

Having looked at the http header of the response from firefox (when calling the servlet via javascript), I see that the encoding is really UTF-8, as expected. After several hours trying to read every possible related article, I met this restlet style discussion and noticed that I really have Transfer-Encoding: chunked in the http-header response. I tried the suggested solutions (overriding ClientResource.toRepresentation, did nothing good, so I tried restlet 2.1, as being compliant with ClientResource.setRe​questEntityBuffering​(true) , there was no luck either), but I'm not sure if my problem is related to Transfer-Encoding: chunked at all .

At this moment I have no ideas, and I would really appreciate any suggestions! O_o

UPDATE

I tried doing a GET with the classic UrlConnection, and the line is returned:

 URL url = new URL(Messages.SERVICE_URL + "myentity/id"); URLConnection conn = url.openConnection(); InputStream is = conn.getInputStream(); StringWriter writer = new StringWriter(); IOUtils.copy(is, writer, "UTF-8"); resp.getWriter().print(writer.toString()); 

So much to be RESTful and bizarre ... but still I don't know why the original version does not work!: /

Answer 1

I tried doing a GET with the classic UrlConnection, and the line is returned:

 URL url = new URL(Messages.SERVICE_URL + "myentity/id"); URLConnection conn = url.openConnection(); InputStream is = conn.getInputStream(); StringWriter writer = new StringWriter(); IOUtils.copy(is, writer, "UTF-8"); resp.getWriter().print(writer.toString()); 

So much to be RESTful and bizarre ... but still I don't know why the original version does not work!: /

Answer 2

Does your answer contain the appropriate Content-Type header? It should be something like " Content-Type: application/json; charset=UTF-8 " (note the encoding).

Try starting the development server and extract your resource from the command line using cURL and check the headers, for example. curl -i http://localhost:8080/myentity/id . In theory, browsers should assume UTF-8 for JSON, but I wouldn’t trust it.