I have a central server to which many distributed servers must transfer data in the form of several large files, 500 MB - 10 GB +. Servers are not in the same physical network and cannot be connected to each other via VPN. Although we are trying to open other ports, at present we can only talk about 443, HTTPS, which is great for our REST services, but terrible for transferring files between servers.
I know that this is not such a specific question as I would like for Stackoverflow, but I would like to know: which methods can work better than the ones I tried?
Server A -> generate file -> transmit via https -> DMZ -> proxy pass -> receive on server B
Both servers use Java 1.8, Tomcat, and Spring 4.1.4.RELEASE. DMZ is just Apache and has little to no control.
Things I tried ...
Make RPC calls for the service using Spring HttpInvokerProxyFactoryBean (this works great for small sites, but large sites often delete connections when transferring data)
Reporting a multi-page form using Apache HttpPost (this also works, but we have to set the file limits in apache / tomcat, as well as its connection is unreliable)
Using a library called RMIIO, which basically mimics RMI over HTTP when configured correctly. This seemed promising as it requests a stream from the server and writes it to the stream from the remote server. I still have not worked on HTTPS, and the library was written in 2007 (with some updates until July 2016), but it is very dated, but not supported, and I suspect there are better ways to do this (now I can’t find them)
I looked at gRPC, but realized that it was just a binary protocol, and I would have to basically handle file fragmentation if I wanted to get a streaming effect.
REST Spring MVC, http://callistaenterprise.se/blogg/teknik/2014/04/22/c10k-developing-non-blocking-rest-services-with-spring-mvc/ , , , .
, . Spark , , , - . , , HTTPS.
( ) , .
. .