Testing modules for download

Question

I am writing a Java program that loads and then processes many web pages. What is the best practice for testing a component of a program that loads a page without hitting remote servers?

Answer 1

Thus, the component that performs the loading and the component that processes the page must be separate. Every time you have a problem testing part of the code, this is a sign that you can try to do too much in one component.

After you have done this, you will test part of the processing, however, it makes sense. Ask the processor component to take an InputStream or even just type String.

As for the download part, you probably need an integration test. Integration tests are often much more active and require setting up a local web server (maven can do this), or at least using a file: URL.

Answer 2

One thought would be to use an "InputStream" as the object you pass into your processing code. I believe that the HttpClient class (or equivalent) for reading data through HTTP gives you some stream to read in response. For testing, you can simply replace another type of read stream, such as a local FileStream.

Answer 3

If your code supports an HTTP proxy, you may have a network cache that acts as a proxy. Just run the code once with proxy caching, save the data, linger on the network, etc. Then after that you can run the code with a proxy just returning the data. Switching between them is just a matter of configuring an HTTP proxy.

The advantage of this approach is that you can unit test against an arbitrary number of sites. The network cache / HTTP proxy will be reused for future use.

Answer 4

Check Injection Dependencies
This is a technique in which you “introduce” different “dependencies” into your functions instead of starting with your function (simple explanation).

Read Martin Fowlers DI article
http://martinfowler.com/articles/injection.html

hope this helps / Jonas