All geek questions in one place

Asynchronous web requests in Java?

I am writing a simple web crawler in Java. I want it to be able to load as many pages per second as possible. Is there a package out there that simplifies the execution of asynchronous HTTP requests in Java? I used HttpURLConnection, but it blocks. I also know that there is something in Apache HTTPCore NIO, but I'm looking for something easier. I tried using this package and I was getting higher throughput using HttpURLConnection for multiple threads.

+3
source share
2 answers

Typically, data-intensive protocols tend to work better in terms of bandwidth with classic blocking I / O compared to NIOs if the number of streams is less than 1000. At least that's the case, of course, with client-side HTTP based on (probably imperfect and possibly biased) HTTP test used by Apache HttpClient [1]

It might be much better to use a blocking HTTP client with threads if the number of threads is moderate (<250)

If you are absolutely sure that you want a NIO-based HTTP client, I can recommend the Jetty HTTP client, which I personally consider the best asynchronous HTTP client at the moment.

[1] http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore

+6
source

, : HTTP- Java

, " , ", , . , "robots.txt" , - .

+3

Source: https://habr.com/ru/post/1779112/

More articles:


All Articles