Uploading files to Google Cloud Storage: 500 Backend Error

Question

We try to upload files to Google Cloud Storage before moving them to BigQuery, but on some downloads we often encounter a “500 Internal Server Error” or “410 Gone” (unprocessed messages below).

We use the official SDK and add a retry with exponential deviation, but the errors are always here. Do you have any tips?

This is how we load (scala):

val credential = new GoogleCredential().setAccessToken(accessToken)

val requestInitializer = new HttpRequestInitializer() {
    def initialize(request: HttpRequest): Unit = {
        credential.initialize(request)

        // to avoid read timed out exception
        request.setConnectTimeout(200000)
        request.setReadTimeout(200000)

        request.setIOExceptionHandler(new 
            HttpBackOffIOExceptionHandler(new ExponentialBackOff()))
        request.setUnsuccessfulResponseHandler(new 
            HttpBackOffUnsuccessfulResponseHandler(new ExponentialBackOff()))
    }
}

val storage = new Storage.Builder(
    new NetHttpTransport,
    JacksonFactory.getDefaultInstance,
    requestInitializer
).setApplicationName("MyAppHere").build

val objectMetadata = new StorageObject()
    .setBucket(bucketName)
    .setName(distantFileName)

val isc = new InputStreamContent("binary/octet-stream", fis)
val length = isc.getLength
val insertObject = storage.objects().insert(bucketName, objectMetadata, isc)
// For small files, you may wish to call setDirectUploadEnabled(true), to
// reduce the number of HTTP requests made to the server.
if (length > 0 && length <= 2 * 1000 * 1000 /* 2MB */ ) {
    insertObject.getMediaHttpUploader.setDirectUploadEnabled(true)
}
insertObject.execute()

Our scala dependencies:

"com.google.api-client" % "google-api-client" % "1.18.0-rc",
"com.google.api-client" % "google-api-client-jackson2" % "1.18.0-rc",
"com.google.apis" % "google-api-services-bigquery" % "v2-rev142-1.18.0-rc",
"com.google.apis" % "google-api-services-storage" % "v1-rev1-1.18.0-rc",
"com.google.http-client" % "google-http-client" % "1.18.0-rc",
"com.google.oauth-client" % "google-oauth-client" % "1.18.0-rc"

Original SDK error responses:

500 Internal Server Error
    {
        "code" : 500,
        "errors" : [ {
        "domain" : "global",
        "message" : "Backend Error",
        "reason" : "backendError"
    } ],
        "message" : "Backend Error"
    }

410 Gone
    {
        "code" : 500,
        "errors" : [ {
        "domain" : "global",
        "message" : "Backend Error",
        "reason" : "backendError"
    } ],
        "message" : "Backend Error"
    }

Answer 1

Backend error , .

- , 10 , , 1:1 .