Limit page size

Question

Is there a way to limit the amount of data loaded by the python urllib2 module? Sometimes I come across broken sites with my / dev / random as a page, and it turns out that they use all the memory on the server.

Answer 1

urllib2.urlopenreturns an object similar to a file, and you can (at least theoretically) .read(N)limit the amount of data returned in N bytes from such an object.

This approach is not completely insane, because an actively hostile site can go quite a bit to trick a reasonably reliable one, like urlib2 by default; in this case, you will need to implement and install your own knife, which knows how to protect itself from such attacks (for example, at the same time receive no more than MB from an open socket, etc.).