JSON URL from StackExchange API returning jibberish?

Question

I have a feeling that I'm doing something wrong here, but I'm not quite sure that I am missing a step or just a problem with the encoding or something like that. Here is my code:

URL url = new URL("http://api.stackoverflow.com/0.8/questions/2886661"); BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream())); // Question q = new Gson().fromJson(in, Question.class); String line; StringBuffer content = new StringBuffer(); while ((line = in.readLine()) != null) { content.append(line); } 

When I print content, I get a whole bunch of wings and special characters, mostly nonsense. I would copy and go through it here, but that does not work. What am I doing wrong?

Answer 1

In this case, this is not a character encoding problem, it is a content encoding problem; you expect text, but the server uses compression to save bandwidth. If you look at the headers, when you take this URL, you will see that the server you are connecting to returns gzipped content:

 GET /0.8/questions/2886661 HTTP/1.1 Host: api.stackoverflow.com HTTP/1.1 200 OK Server: nginx Date: Sat, 22 May 2010 15:51:34 GMT Content-Type: application/json; charset=utf-8 <more headers> Content-Encoding: gzip <more headers> 

Thus, you need to either use a smarter client, such as Apache HttpClient, as suggested by stevedbrown (although you need to configure it to make it automatically say Gzip ) or explicitly unzip the stream obtained in your sample code. Try this instead of the line where you declare your input:

  BufferedReader in = new BufferedReader(new InputStreamReader(new GZIPInputStream(url.openStream()))); 

I checked that this works for the url you are trying to capture.

Answer 2

Use the Apache Http Client to properly perform character conversions. From these site examples :

 public final static void main(String[] args) throws Exception { HttpClient httpclient = new DefaultHttpClient(); HttpGet httpget = new HttpGet("http://api.stackoverflow.com/0.8/questions/2886661"); System.out.println("executing request " + httpget.getURI()); // Create a response handler ResponseHandler<String> responseHandler = new BasicResponseHandler(); String responseBody = httpclient.execute(httpget, responseHandler); System.out.println(responseBody); System.out.println("----------------------------------------"); // When HttpClient instance is no longer needed, // shut down the connection manager to ensure // immediate deallocation of all system resources httpclient.getConnectionManager().shutdown(); } 

In this case, see http://svn.apache.org/repos/asf/httpcomponents/httpclient/branches/4.0.x/httpclient/src/examples/org/apache/http/examples/client/ClientGZipContentCompression.java , in which shows how to work with gzip content.

Answer 3

Sometimes the response of an API call is compressed, for example. StackExchange API Please check their documentation and see what kind of compression they use. Some use GZIP or DEFLATE compression. For gzip compression, use the following.

 InputStream is = new URL(url).openStream(); BufferedReader in = new BufferedReader(new InputStreamReader(new GZIPInputStream(is)));