Renewable Download in Drive Rest V3 API

Question

I am trying to create a revolving boot session using the API to control the drive on Android.

In accordance with the documentation, you must perform the following steps:

• Start a renewable session
• Save Updated Session URI
• Upload file

Step 1: I use the following code to start a renewable session.

File body = new File(); body.setName(fileName); body.setMimeType(mimeType); body.setCreatedTime(modifiedDate); body.setModifiedTime(modifiedDate); body.setParents(Collections.singletonList(parentId)); HttpHeaders header = new HttpHeaders(); header.setContentLength(0L); header.setContentType("application/json; charset=UTF-8"); header.set("X-Upload-Content-Type","image/jpeg"); HttpResponse response= driveObject .files() .create(body) .setRequestHeaders(header) .set("uploadType","resumable") .buildHttpRequest() .execute(); 

Step 2: Once the execution is completed, I print the request response header to see the location URI

 System.out.println(response.getHeader().toString()); 

The output is as follows:

 { cache-control=[no-cache, no-store, max-age=0, must-revalidate], content-encoding=[gzip], content-type=[application/json; charset=UTF-8], date=[Thu, 06 Oct 2016 02:20:18 GMT], expires=[Mon, 01 Jan 1990 00:00:00 GMT], alt-svc=[quic=":443"; ma=2592000; v="36,35,34,33,32"], pragma=[no-cache], server=[GSE], transfer-encoding=[chunked], vary=[Origin, X-Origin], x-android-received-millis=[1475720421761], x-android-response-source=[NETWORK 200], x-android-sent-millis=[1475720420804], x-content-type-options=[nosniff], x-frame-options=[SAMEORIGIN], x-xss-protection=[1; mode=block] } 

I did not find the location URI in the response header to start downloading filedata as indicated in the documentation, and I do not find Java patterns to reload.

How to get the location location id as indicated in the documentation?

Answer 1

I tried for most of the week, and I finally got renewable downloads to run. This does not work as I expected, but it will work.

Do not use the REST API for everyone.

What I found out is that the Google Drive REST API is, as far as I know, really not capable of doing pieces of downloads. It could be a mistake, or it could be by design. I can be too stupid too.

But I thought that I could not see the code examples anywhere. Everyone was talking about Http headers all the time. So this is what we will do below. We will only use headers.

So, how do you do renewable, hosted downloads using the Google Drive and Android REST APIs:

0) Initialization

 String accountName = "account_name"; GoogleAccountCredential credential = GoogleAccountCredential.usingOAuth2(context, Arrays.asList(SCOPES)).setBackOff(new ExponentialBackOff()).setSelectedAccountName(accountName); 

1) Start a renewable session

Follow the guidelines set forth by Google in this document :

 POST /upload/drive/v3/files?uploadType=resumable HTTP/1.1 Host: www.googleapis.com Authorization: Bearer your_auth_token Content-Length: 38 Content-Type: application/json; charset=UTF-8 X-Upload-Content-Type: image/jpeg X-Upload-Content-Length: 2000000 { "name": "My File" } 

Set all header fields exactly as in the Google example. Send it as a POST request. Use the credential variable to get the authorization token. The mime type for X-Upload-Content-Type not so important, it works without it ( this SO answer provides a nice function to extract it from the path). Set X-Upload-Content-Length to the total file length. Set Content-Type to JSON, as our body will provide metadata for Google in JSON format.

Now create the body of your metadata. I entered the file name and parent. Set the Content-Length to the length of your body in bytes. Then write your body to the request.getOutputStream() output stream.

 URL url = new URL("https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable"); HttpURLConnection request = (HttpURLConnection) url.openConnection(); request.setRequestMethod("POST"); request.setDoInput(true); request.setDoOutput(true); request.setRequestProperty("Authorization", "Bearer " + credential.getToken()); request.setRequestProperty("X-Upload-Content-Type", getMimeType(file.getPath())); request.setRequestProperty("X-Upload-Content-Length", String.format(Locale.ENGLISH, "%d", file.length())); request.setRequestProperty("Content-Type", "application/json; charset=UTF-8"); String body = "{\"name\": \"" + file.getName() + "\", \"parents\": [\"" + parentId + "\"]}"; request.setRequestProperty("Content-Length", String.format(Locale.ENGLISH, "%d", body.getBytes().length)); OutputStream outputStream = request.getOutputStream(); outputStream.write(body.getBytes()); outputStream.close(); request.connect(); 

2) Save the renewable session URI

Finally, connect() and wait for an answer. If the response code is 200 , you have successfully initiated a restarted, resumed download. Now save the location header URI somewhere (database, text file, whatever). You will need this later.

 if (request.getResponseCode() == HttpURLConnection.HTTP_OK) { String sessionUri = request.getHeaderField("location"); } 

3) Download the file

 PUT {session_uri} HTTP/1.1 Host: www.googleapis.com Content-Length: 524288 Content-Type: image/jpeg Content-Range: bytes 0-524287/2000000 bytes 0-524288 

Put the following code in a loop until the entire file is loaded. After each fragment, you will receive a response with the code 308 and the header range . From this range header, you can read the next run of the piece (see (4)).

Content-Type will again be the mime type. Content-Length - the number of bytes loaded into this fragment. Content-Range should be of the form bytes startByte-EndByte/BytesTotal . You put this in a PUT request.

Then you create a FileInputStream and set the position to the start byte (which you got from the last header of the range response) and read the other piece into your buffer. Then this buffer is written to the output stream of the connection. Finally, connect() .

 URL url = new URL(sessionUri); HttpURLConnection request = (HttpURLConnection) url.openConnection(); request.setRequestMethod("PUT"); request.setDoOutput(true); request.setConnectTimeout(10000); request.setRequestProperty("Content-Type", getMimeType(file.getPath())); long uploadedBytes = chunkSizeInMb * 1024 * 1024; if (chunkStart + uploadedBytes > file.length()) { uploadedBytes = (int) file.length() - chunkStart; } request.setRequestProperty("Content-Length", String.format(Locale.ENGLISH, "%d", uploadedBytes)); request.setRequestProperty("Content-Range", "bytes " + chunkStart + "-" + (chunkStart + uploadedBytes - 1) + "/" + file.length()); byte[] buffer = new byte[(int) uploadedBytes]; FileInputStream fileInputStream = new FileInputStream(file); fileInputStream.getChannel().position(chunkStart); if (fileInputStream.read(buffer, 0, (int) uploadedBytes) == -1) { /* break, return, exit*/ } fileInputStream.close(); OutputStream outputStream = request.getOutputStream(); outputStream.write(buffer); outputStream.close(); request.connect(); 

4) Reply to appeal

After that, you will receive a response with the code 308 (if successful). This answer contains the range header (mentioned).

 HTTP/1.1 308 Resume Incomplete Content-Length: 0 Range: bytes=0-524287 

You split it and got a new start byte.

  String range = chunkUploadConnection.getHeaderField("range"); int chunkPosition = Long.parseLong(range.substring(range.lastIndexOf("-") + 1, range.length())) + 1; 

5) The response code is not 308 ?!

It may happen that you get a 5xx response. Your internet connection may fail, the file may be deleted / renamed during download, etc. Etc. Do not worry. As long as you save your session URI and start byte, you can resume downloading at any time.

To do this, submit the header of the following form:

 PUT {session_uri} HTTP/1.1 Content-Length: 0 Content-Range: bytes */TotalFileLength URL url = new URL(sessionUri); HttpURLConnection request = (HttpURLConnection) url.openConnection(); request.setRequestMethod("PUT"); request.setDoOutput(true); request.setConnectTimeout(10000); request.setRequestProperty("Content-Length", "0"); request.setRequestProperty("Content-Range", "bytes */" + file.length()); request.connect(); 

Then you get 308 with the range header, from which you can read the last byte loaded (as we did above). Take this number and start the cycle again.

I hope I can help some of you. If you have more questions, just ask in the comments and I will edit the answer.

Answer 2

You do not need to care about all this logic. the documentation does explain the thread for completing a renewable download, but it is prone to errors if done manually.
Fortunately, Google exposes a dedicated class to handle this case, i.e. MediaHttpUploader .

This piece of code does the work of resuming boot on disk (the same thing can be done on GCS):

 public class Main { private static final JacksonFactory JSON_FACTORY = new JacksonFactory(); private static final NetHttpTransport HTTP_TRANSPORT = new NetHttpTransport(); private static final MemoryDataStoreFactory DATA_STORE = new MemoryDataStoreFactory(); public static void main(String... args) throws IOException { Credential credential = authorize(); MediaHttpUploader mediaHttpUploader = new MediaHttpUploader(new FileContent("application/json", Paths.get("/path/to/foo.json").toFile()), HTTP_TRANSPORT, credential); mediaHttpUploader.setProgressListener(uploader -> System.out.println("progress: " + uploader.getProgress())); GenericUrl genericUrl = new GenericUrl(new URL("https://www.googleapis.com/upload/drive/v3/files?name=toto")); GenericJson data = new GenericJson(); data.put("name", "title"); JsonHttpContent jsonHttpContent = new JsonHttpContent(JSON_FACTORY, data); mediaHttpUploader.setMetadata(jsonHttpContent).upload(genericUrl); System.out.println("Finished"); } private static Credential authorize() throws IOException { // load client secrets try (BufferedReader br = Files.newBufferedReader(Paths.get(Resources.getResource("client_secret.json").getPath()))) { GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(JSON_FACTORY, br); // set up authorization code flow GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder( HTTP_TRANSPORT, JSON_FACTORY, clientSecrets, Collections.singleton(DriveScopes.DRIVE)) .setAccessType("offline") .setDataStoreFactory(DATA_STORE).build(); // authorize return new AuthorizationCodeInstalledApp(flow, new LocalServerReceiver()).authorize("user"); } } 

}

Please note that there is no place where we indicate the location. All logic is hidden in the MediaHttpUploader class.
Therefore, I do not answer the question (where to find "Location"), but I point out that this is really not necessary when using classes from the Google library (and I am sure that there are other third-party libraries to do the same work).

UPDATE: mediaHttpUploader is what is used by the Drive v3 client under the hood. Therefore, we can think of something like this:

  File fileMetadata = new File(); fileMetadata.setName(UPLOAD_FILE.getName()); FileContent mediaContent = new FileContent("image/jpeg", UPLOAD_FILE); Drive.Files.Create insert = drive.files().create(fileMetadata, mediaContent); MediaHttpUploader uploader = insert.getMediaHttpUploader(); uploader.setDirectUploadEnabled(false); uploader.setProgressListener(new FileUploadProgressListener()); return insert.execute(); 

Answer 3

If you were able to get 200 Http status, it will provide Location as part of the header. But from what I saw on your System.print , there is no HttpResponse.getHeader , it could just be a typo, and you mean HttpResponse.getHeaders .

If so, I would suggest first determining if you have a 200 OK Http status code, and loop getAllheaders to determine if there is a Location header.

Hope this helps!

Answer 4

Maybe this https://github.com/PiyushXCoder/google-drive-ResumableUpload/blob/master/ResumableUpload.java will help you. However, it was written for servlets, but you can easily change it for Android.

Well, after receiving the comments, let me add a few additional descriptions.

However, the "resumableUpload.java" github repo link is well commented, and this is enough for you to understand how to perform this download on Google Drive. And you really don't need to read this long description.

As described in https://developers.google.com/drive/v3/web/resumable-upload on google on how to perform a resumable download

• We need to make a POST request to tell the server about this download and get the session URI to which we will send our pieces of data for the file. And yes, we need an access token to fulfill this request (here the Credential object has an access token, and we will use it). This request is executed using this method:

     public String requestUploadUrl (HttpServletRequest request, HttpServletResponse response, Credential credential, com.google.api.services.drive.model.File jsonStructure) throws MalformedURLException, IOException
         {
                 URL url = new URL ("https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable");
                 HttpURLConnection req = (HttpURLConnection) url.openConnection ();
                 req.setRequestMethod ("POST");
                 req.setDoInput (true);
                 req.setDoOutput (true);
                 req.setRequestProperty ("Authorization", "Bearer" + credential.getAccessToken ());
                 req.setRequestProperty ("X-Upload-Content-Type", jsonStructure.getMimeType ());
                 req.setRequestProperty ("X-Upload-Content-Length", String.valueOf (jsonStructure.getSize ()));
                 req.setRequestProperty ("Content-Type", "application / json; charset = UTF-8");

                 String body = "{\" name \ ": \" "+ jsonStructure.getName () +" \ "}";
                 req.setRequestProperty ("Content-Length", String.format (Locale.ENGLISH, "% d", body.getBytes (). length));
                 OutputStream outputStream = req.getOutputStream ();
                 outputStream.write (body.getBytes ());
                 outputStream.close ();
                 req.connect ();

                 String sessionUri = null;

                 if (req.getResponseCode () == HttpURLConnection.HTTP_OK) {
                     sessionUri = req.getHeaderField ("location");
                 }
                 return sessionUri; 
             }

• Now that we have the session URI, we can start sending our data for the requested file, the cartridge is wise. And let me execute PUT requests for each fragment. The size of each cartridge must be a multiple of 256 KB. For each fragment, the following method can be used.

     public int uploadFilePacket (HttpServletRequest request, HttpServletResponse response, String sessionUri, com.google.api.services.drive.model.File jsonStructure, java.io.File file, long chunkStart, long uploadBytes) throws MalformedURLException, IOException
         {
             URL url1 = new URL (sessionUri);
             HttpURLConnection req1 = (HttpURLConnection) url1.openConnection ();

             req1.setRequestMethod ("PUT");
             req1.setDoOutput (true);
             req1.setDoInput (true);
             req1.setConnectTimeout (10000);

             req1.setRequestProperty ("Content-Type", jsonStructure.getMimeType ());
             req1.setRequestProperty ("Content-Length", String.valueOf (uploadBytes));
             req1.setRequestProperty ("Content-Range", "bytes" + chunkStart + "-" + (chunkStart + uploadBytes -1) + "/" + jsonStructure.getSize ());

             OutputStream outstream = req1.getOutputStream ();

             byte [] buffer = new byte [(int) uploadBytes];
             FileInputStream fileInputStream = new FileInputStream (file);
             fileInputStream.getChannel (). position (chunkStart);
             if (fileInputStream.read (buffer, 0, (int) uploadBytes) == -1);
             fileInputStream.close ();

             outstream.write (buffer);
             outstream.close ();

             req1.connect ();

             return req1.getResponseCode ();
         }

The following method loads the file, dividing it into pieces.

     public void uploadFile (HttpServletRequest request, HttpServletResponse response, Credential credential, com.google.api.services.drive.model.File jsonStructure, java.io.File file) throws IOException, UploadFileException
         {
             String sessionUrl = requestUploadUrl (request, response, credential, jsonStructure);

             for (long i = 1, j = CHUNK_LIMIT; i = jsonStructure.getSize ())
                 {
                     j = jsonStructure.getSize () - i + 1;
                 }
                 int responseCode = uploadFilePacket (request, response, sessionUrl, jsonStructure, file, i-1, j);
                 if (! (responseCode == OK || responseCode == CREATED || responseCode == INCOMPLETE)) throw new UploadFileException (responseCode);
             }
         }

What all.