All geek questions in one place

URLFetch behind the proxy server when creating App Engine

Is there a way to specify a proxy server when using urlfetch in Google App Engine?

In particular, every time I make a call using urlfetch, I want the GAE to go through a proxy. I want to do this in production, not just dev.

I want to use a proxy server because there are problems using google outgoing IP addresses (speed limit, static outgoing IP address, sometimes blacklisted, etc.). Setting up a proxy server is usually easy if you can edit the http message itself, but the GAE API does not allow you to do this.

+4
source share
2 answers

You can always use your own:

  • In the case of a fixed destination: just configure fixed port forwarding on the proxy server. Then send requests from GAE to the proxy. If you have multiple destinations, set up forwarding to separate ports, one for each destination.

  • In the case of a dynamic destination (too much to handle through fixed port forwarding), your GAE application adds its own HTTP header ( X-Something ) containing the final destination, and then connects to a custom proxy. The user proxy checks this field and redirects the request to the destination.

+2
source

We ran into this issue and turned to Google Cloud support. They suggested using the flexible Google App Engine with some app.yaml options, a configurable network, and IP forwarding of the NAT gateway.

This did not work for us, because many key features from App Engine Standard are not implemented in App Engine Flexible. In fact, we will need to rewrite our product.

So, in order to make applicable requests for a selection of URLs, you have a static IP address, we created our own proxy server: https://github.com/csgactuarial/app-engine-proxy

For reasons of redundancy, I propose to implement this as a multi-zone, multi-zone system with load balancing.

0
source

Source: https://habr.com/ru/post/1390179/

More articles:


All Articles