URL encoding using httpclient

Question

I have a list of URLs that I need to get. A URL with special characters and therefore must be encoded. I use Commons HtpClient to get the content.

when i use:

GetMethod get = new GetMethod(url); 

I get an "Invalid" illegal escape character exception. "When I use

  GetMethod get = new GetMethod(); get.setURI(new URI(url.toString(), false, "UTF-8")); 

I get 404 when I try to get the page because the space turns into %2520 instead of just %20 .

I saw a lot of posts about this problem, and most of them advised to partially create part of the URI. The problem is that it is a list of URLs, not one that I can handle manually.

Any other solution for this problem?

thanks.

Answer 1

What if you create a new URL object for it, for example URL urlObject = new URL(url) , then urlObject.getQuery() and urlObject.getPath() to separate it correctly, urlObject.getPath() request passwords into a list or map, or whatever then do something like:

EDIT: I just found out that the HttpClient library has a URLEncodedUtils.parse() method, which you can easily use with the code below. I will edit it to match, but not verified.

With Apache HttpClient, it will be something like:

 URI urlObject = new URI(url,"UTF-8"); HttpClient httpclient = new DefaultHttpClient(); List<NameValuePair> formparams = URLEncodedUtils.parse(urlObject,"UTF-8"); UrlEncodedFormEntity entity; entity = new UrlEncodedFormEntity(formparams); HttpPost httppost = new HttpPost(urlObject.getPath()); httppost.setEntity(entity); httppost.addHeader("Content-Type","application/x-www-form-urlencoded"); HttpResponse response = httpclient.execute(httppost); HttpEntity entity2 = response.getEntity(); 

With Java URLConnection, it will be something like:

  // Iterate over query params from urlObject.getQuery() like while(en.hasMoreElements()){ String paramName = (String)en.nextElement(); // Iterator over yourListOfKeys String paramValue = yourMapOfValues.get(paramName); // replace yourMapOfNameValues str = str + "&" + paramName + "=" + URLEncoder.encode(paramValue); } try{ URL u = new URL(urlObject.getPath()); //here the url path from your urlObject URLConnection uc = u.openConnection(); uc.setDoOutput(true); uc.setRequestProperty("Content-Type","application/x-www-form-urlencoded"); PrintWriter pw = new PrintWriter(uc.getOutputStream()); pw.println(str); pw.close(); BufferedReader in = new BufferedReader(new InputStreamReader(uc.getInputStream())); String res = in.readLine(); in.close(); // ... } 

Answer 2

Please use the URLEncoder class.
I used it in the exact scenario and it worked perfectly for me.
What I did was use the URL class to get the part that comes after the host
(for example, at www.bla.com/mystuff/bla.jpg it will be "mystuff / bla.jpg" - you should URLEncode only this part, and then create the URL again.

For example, if the string orignal "http://www.bla.com/mystuff/bla foo.jpg", then:
Encode is "mystuff / bla foo.jpg" and get "mystuff / bla% 20foo.jpg" and then attach this to the parts of the host and protocol:
"http://www.bla.com/mystuff/bla%20foo.jpg"
Hope this helps