Calculating Post Content-Length for a Base64 encoded file

Question

I am trying to upload some large files from an Android device to the .Net Web Service. This web service is configured in such a way that it accepts these files as a POST parameter and that the files should be sent as a Base64 encoded string.

I was able to use this Christian d'Heureuse library to convert a file to a Base64 string, calculate the size of the string in bytes and send it earlier, however, the method I used earlier included loading the entire file into memory, which caused errors from memory when working with large files, which was not unexpected.

I am trying to convert a file in Base64 to chunks and transfer this data through the connection (using the data output stream object) as it is being converted, so the whole file does not need to be loaded into memory in one go, however I cannot determine the exact size of the Content-Length for the request before the file conversion - I usually look at about 10 bytes - disappointing, it sometimes works!

I also found that for a while, when this works, the server returns the following error message "Invalid size for Base64 char array". I think this is a problem with padding characters, however I don't see a problem with my code working, some advice on this problem would be much appreciated!

This is the code that generates the request and passes the data:

try { HttpURLConnection connection = null; DataOutputStream outputStream = null; DataInputStream inputStream = null; //This is the path to the file String pathToOurFile = Environment .getExternalStorageDirectory().getPath() + "/path/to/the/file.zip"; String urlServer = "https://www.someserver.com/somewebservice/"; int bytesRead, bytesAvailable, bufferSize; byte[] buffer; int maxBufferSize = 456; //The parameters of the POST request - File Data is the file in question as a Base64 String String params = "Username=foo&Password=bar&FileData="; File sizeCheck = new File(pathToOurFile); Integer zipSize = (int) sizeCheck.length(); Integer paddingRequired = ((zipSize * 8) / 6) % 3; Integer base64ZipSize = ((zipSize * 8) / 6) + ((zipSize * 8) / 6) % 3; Integer paramLength = params.getBytes().length; //Code to work out the number of lines required, assuming we create a new //line every 76 characters - this is used t work out the number of //extra bytes required for new line characters Integer numberOfLines = base64ZipSize / 76; Log.i(TAG, "numberOfLines: " + numberOfLines); Integer newLineLength = System.getProperty("line.separator") .getBytes().length; //This works out the total length of the Content Integer totalLength = paramLength + base64ZipSize + (numberOfLines * newLineLength) + paddingRequired; Log.i(TAG, "total Length: " + totalLength); FileInputStream fileInputStream = new FileInputStream(new File( pathToOurFile)); URL url = new URL(urlServer); connection = (HttpURLConnection) url.openConnection(); connection.setDoInput(true); connection.setDoOutput(true); connection.setUseCaches(false); connection.setRequestMethod("POST"); connection.setRequestProperty("Connection", "Keep-Alive"); connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;"); connection.setRequestProperty("Content-Length", "" + totalLength); // number of bytes outputStream = new DataOutputStream( connection.getOutputStream()); //Write out the parameters to the data output stream outputStream.writeBytes(params); bytesAvailable = fileInputStream.available(); bufferSize = Math.min(bytesAvailable, maxBufferSize); buffer = new byte[bufferSize]; bytesRead = fileInputStream.read(buffer, 0, bufferSize); Integer totalSent = paramLength; Integer enLen = 0; //Convert the file to Base64 and Stream the result to the //Data output stream while (bytesRead > 0) { String convetedBase64 = Base64Coder.encodeLines(buffer); convetedBase64 = convetedBase64.replace("=", ""); if (totalSent >= (totalLength - 616)) { Log.i(TAG, "about to send last chunk of data"); convetedBase64 = convetedBase64.substring(0, convetedBase64.length() - 1); } Log.i(TAG, "next data chunk to send: " + convetedBase64.getBytes().length); Log.i(TAG, "'" + convetedBase64 + "'"); enLen = enLen + convetedBase64.length(); outputStream.writeBytes(convetedBase64); totalSent = totalSent + convetedBase64.getBytes().length; Log.i(TAG, "total sent " + totalSent); Log.i(TAG, "actual size: " + outputStream.size()); bytesAvailable = fileInputStream.available(); bufferSize = Math.min(bytesAvailable, maxBufferSize); buffer = new byte[bufferSize]; bytesRead = fileInputStream.read(buffer, 0, bufferSize); // read // into // the // buffer } Log.i(TAG, "enLen: " + enLen); Log.i(TAG, "paddingRequired: " + paddingRequired); for (int x = 0; x < paddingRequired; x++) { outputStream.writeBytes("="); } InputStream is2 = connection.getInputStream(); String output = IOUtils.toString(is2); Log.i(TAG, "Got server response: " + output); fileInputStream.close(); outputStream.flush(); outputStream.close(); } catch (Exception ex) { Log.e(TAG, "caught an exception:" + ex.getMessage()); } 

I would be very grateful if anyone could point out any errors in my code that could cause this, or suggest a better way to convert and download the file.

Answer 1

Soooo ... I managed to find several solutions to this problem, just in case someone stumbles on this question, I will leave them here:

The first was to write the data to a temporary file, so I could get the size after the conversion in bytes - at first it seemed like a good idea, but it was inefficient and after finding other methods it seemed stupid.

Another method is to not indicate the length of the content (I did not know that you can do this!). Unfortunately, Android was still trying to allocate enough memory to boot, which caused problems.

If you specify a connection using the ChunkedStreamingMode Android, then it plays well and buffers the load, saving on the used basset (except for the strange 2.2 error).

The code for this as such:

 httppost.setDoInput(true); httppost.setDoOutput(true); httppost.setRequestMethod("POST"); httppost.setRequestProperty("Content-Type", "application/x-www-form-urlencoded"); httppost.setChunkedStreamingMode(0); //This specifies the size of the chunks - 0 uses the system default DataOutputStream dos = new DataOutputStream(httppost.getOutputStream()); dos.writeBytes(content); //This code write out the rest of the post body first, for example username or password try { BufferedInputStream dis = new BufferedInputStream(new FileInputStream("path/to/some/file"), 2048); byte[] in = new byte[512]; while (dis.read(in)) > 0) { dos.write(Base64.encode(in, Base64.URL_SAFE)); //This writes out the Base64 data //I used the web safe base64 to save URL encoding it again //However your server must support this dos.write(newLine); //this write out a newline character } dos.flush(); dis.close(); dos.close(); 

Hope this helps !!!