How to process / read response using "Transfer-Encoding: chunked"?

Question

Can I learn how to process / read the response using "Transfer-Encoding: chunked"?

Currently im is using common-httpclient.3.1

My current encoding as indicated (can only handle a response with content length in the header only): -

httppost = new PostMethod(localurl); httppost.setRequestHeader("Content-Type", "application/xml; charset=utf-8"); RequestEntity entity = new StringRequestEntity(in, "application/xml", "UTF-8"); httppost.setRequestHeader("Content-length", entity.getContentLength()+""); httppost.setRequestEntity(entity); for (int i=0; i<retryAttempt; i++) { try { httpclient.executeMethod(httppost); if (httppost.getStatusCode() == 200) { br = new BufferedReader(new InputStreamReader(httppost.getResponseBodyAsStream(), httppost.getResponseCharSet())); String reply = null; long len = httppost.getResponseContentLength(); if(len != 0) { char[] cbuf = new char[Integer.parseInt(len+"")]; if (br.read(cbuf, 0, Integer.parseInt(len+"")) != -1 ) { repOut = String.valueOf(cbuf); } }else{ while ((reply = br.readLine()) != null) { if (!reply.equals("")) repOut += reply+" \n "; } } 

Reply from URL (via httppost):

 Thread-9, READ: TLSv1 Application Data, length = 443 Padded plaintext after DECRYPTION: len = 443 0000: 48 54 54 50 2F 31 2E 31 20 32 30 30 20 4F 4B 0D HTTP/1.1 200 OK. 0010: 0A 53 65 72 76 65 72 3A 20 53 54 52 4F 4E 47 48 .Server: STRONGH 0020: 4F 4C 44 2F 33 2E 30 2F 4F 70 65 6E 42 53 44 0D OLD/3.0/OpenBSD. 0030: 0A 44 61 74 65 3A 20 53 75 6E 2C 20 32 32 20 4D .Date: Sun, 22 M 0040: 61 79 20 32 30 31 31 20 31 34 3A 30 31 3A 30 38 ay 2011 14:01:08 0050: 20 47 4D 54 0D 0A 43 6F 6E 74 65 6E 74 2D 74 79 GMT..Content-ty 0060: 70 65 3A 20 61 70 70 6C 69 63 61 74 69 6F 6E 2F pe: application/ 0070: 78 6D 6C 3B 63 68 61 72 73 65 74 3D 55 54 46 2D xml;charset=UTF- 0080: 38 0D 0A 53 65 74 2D 63 6F 6F 6B 69 65 3A 20 4A 8..Set-cookie: J 0090: 53 45 53 53 49 4F 4E 49 44 3D 34 34 42 35 33 45 SESSIONID=44B53E 00A0: 35 46 38 30 39 41 35 35 32 34 32 41 44 36 34 38 5F809A55242AD648 00B0: 33 41 39 33 32 42 30 46 44 36 3B 20 50 61 74 68 3A932B0FD6; Path 00C0: 3D 2F 3B 20 53 65 63 75 72 65 0D 0A 54 72 61 6E =/; Secure..Tran 00D0: 73 66 65 72 2D 65 6E 63 6F 64 69 6E 67 3A 20 63 sfer-encoding: c 00E0: 68 75 6E 6B 65 64 0D 0A 0D 0A 62 62 0D 0A 3C 3F hunked....bb..<? 00F0: 78 6D 6C 20 76 65 72 73 69 6F 6E 3D 22 31 2E 30 xml version="1.0 0100: 22 20 65 6E 63 6F 64 69 6E 67 3D 22 55 54 46 2D " encoding="UTF- 0110: 38 22 3F 3E 0A 3C 54 68 72 65 65 44 53 65 63 75 8"?>.<ThreeDSecu 0120: 72 65 3E 3C 4D 65 73 73 61 67 65 20 69 64 3D 22 re><Message id=" 0130: 50 41 54 72 61 6E 73 52 65 71 53 70 65 63 34 31 PATransReqSpec41 0140: 37 39 32 30 31 31 30 35 32 32 32 32 30 30 30 32 7920110522220002 0150: 36 30 31 37 32 39 33 33 37 36 38 36 35 22 3E 3C 6017293376865">< 0160: 50 41 54 72 61 6E 73 52 65 73 3E 3C 76 65 72 73 PATransRes><vers 0170: 69 6F 6E 3E 31 2E 30 2E 32 3C 2F 76 65 72 73 69 ion>1.0.2</versi 0180: 6F 6E 3E 3C 2F 50 41 54 72 61 6E 73 52 65 73 3E on></PATransRes> 0190: 3C 2F 4D 65 73 73 61 67 65 3E 3C 2F 54 68 72 65 </Message></Thre 01A0: 65 44 53 65 63 75 72 65 3E 0D 0A 76 67 28 7C B3 eDSecure>..vg(.. 01B0: C8 F9 7C A1 C9 77 2F F6 E8 8A 42 .....w/...B [Raw read]: length = 5 

Answer 1

commons-httpclient will cancel your server response out of the box, you do not need to handle blocks. However, you may run into problems because of the nature of the chunking you don't know the length of the content until you read everything. (Querying the length of the content will return -1.) Therefore, you cannot allocate a fixed-size buffer to hold all content.

If you change the test on the return value of getResponseContentLength to "if (len> 0)", I think you will see that you are in the case of readLine when you encounter a response that needs to be read, and that it should read the whole answer by queues.

Answer 2

Another library to keep in mind is the Jodd Http library ( http://jodd.org/doc/http.html ). It handles the Chunked transmission encoding for you, and you get the whole body as a string back.

Good thing the API is very simple:

 HttpRequest httpRequest = HttpRequest.get("http://jodd.org"); HttpResponse response = httpRequest.send();