URL.openConnection () returns an already open connection a second time

Question

I am working on web site cleaning apps in Scala. The site I'm scraping is heavily session oriented, so I need to hit the site once to get the session id before I can do anything else.

I get a connection to get the session id, for example:

url.openConnection().asInstanceOf[HttpURLConnection] 

It works great. The associated field of the returned HttpURLConnection is false, and it turns to true when I call .connect () on it. No problems.

The first hint of the problem occurs when I end the connection and call .disconnect () on it. The .connected field remains valid. Hm.

So now I have a session id, and I'm going to get a page that has the form I want on it. I'm calling

 url.openConnection().asInstanceOf[HttpURLConnection] 

again, like last time - the same code, in fact - except that the HttpURLConnection that it gives me has a .connected field set to true! At first I thought that somehow it gives me the same object that it gave me before, but no, the memory identifier is different.

So now, when I call .setRequestProperty () on the connection, it explodes with IllegalStateException: already connected.

I do not understand how to use HttpURLConnection?

Notes: Scala 2.9.2, Java 6.0. Also, the URL objects that I call .openConnection () on are different objects, not the same.

Thanks...

Answer 1

It is called the connection pool, in search of HTTP Keep-alive. It's good. Do you want it. If you do not, call the disconnect() method.

Answer 2

The URL class is not well suited for session-based operation (especially for cookie-based sessions).

If you want to take advantage of this, I would suggest using something like Apache HTTPClient

IMHO

Answer 3

It seems that the HttpUrlConnection supports the connection for you under the covers.

Check out this article for some tips to get it to close the connection and not be too useful.

Although in your case it sounds as if you might want to use keep-alive, as this can speed up your calls to the site, avoiding unnecessary connections.