I implemented Java code to upload files to a server from org.apache.commons.net.ftp.FTPClient For several files, ftp download speed is very slow. How can I improve the speed.
-Change the library? What is a powerful FTP client class library for uploading multiple files?
- Use multiple threads? How can I implement ftp download function with multiple threads? Can someone show me an example? I am new to multithreaded programming.
After I read the whole answer, I will try to change my code and test it.
The following is an example FTPClient code:
// create instance of FTPClient FTPClient ftp = new FTPClient(); ftp.setControlEncoding("UTF-8"); ftp.setDefaultTimeout(30000); // connect to server try { ftp.connect("10.1.1.1", 990); } catch(Exception e) { System.out.println("Cannot connect to server"); return; } // login to server if (!ftp.login("username", "password")) { ftp.logout(); System.out.println("Cannot login to server"); return; } try { ftp.setFileTransferMode(FTP.BINARY_FILE_TYPE); ftp.enterLocalPassiveMode(); // ftp.setBufferSize(0); <-- someone suggest me to set buffer size to 0, but it throw error sometime. } catch(Exception e) { } // create directory on server // dirs is list of required directories on server for (String dir : dirs) { try { ftp.makeDirectory(dir); } catch(IOException e) { } } // files is a map of local file and string of remote file // such as // file on client is "C://test/a.txt" // location on server is "/test/a.txt" for (Map.Entry<File, String> entry : files.entrySet()) { File localFile = entry.getKey(); String remoteFile = entry.getValue(); FileInputStream input = null; try { input= new FileInputStream(localFile); ftp.storeFile(remoteFile, input); } catch (Exception e) { try { ftp.deleteFile(remoteFile); } catch (IOException e1) { } } finally { if (input != null) { try { input.close(); } catch (IOException e) { } } } } // disconnect if (ftp != null && ftp.isConnected()) { try { ftp.disconnect(); } catch (IOException f) { // do nothing } }
When I downloaded 1050 files (each file is about 1-20 KB), it took about 49406 - 51000 milliseconds (this is just the download time). I would like to improve speed.
Some people suggest I use ftp4j, but when I test a library with 1050 files, ftp4j download speed is slower than FTPClient, about 10,000 milliseconds. it took about 60,000 milliseconds.
The following is an example ftp4j code:
// create instance of FTPClient FTPClient ftp = new FTPClient(); ftp.setCharset("UTF-8"); // connect to server try { ftp.connect("10.1.1.1", 990); } catch(Exception e) { System.out.println("Cannot connect to server") return; } // login to server try { ftp.login("username", "password"); } catch (Exception e) { try { ftp.logout(); } catch (Exception e1) { } System.out.println("Cannot login to server") return; } try { ftp.setType(FTPClient.TYPE_BINARY); ftp.setPassive(true); } catch(Exception e) { } // create directory on server // dirs is list of required directories on server for (String dir : dirs) { try { ftp.createDirectory(dir); } catch (Exception e) { } } // files is a map of local file and string of remote file // such as // file on client is "C://test/a.txt" // location on server is "/test/a.txt" for (Map.Entry<File, String> entry : files.entrySet()) { final File localFile = entry.getKey(); final String remoteFile = entry.getValue(); BufferedInputStream input = null; boolean success = false; try { input = new BufferedInputStream(new FileInputStream(localFile)); // ftp.upload(localFile); <-- if I use ftp.upload(File), it will took more time. ftp.upload(remoteFile, input, 0, 2048, new MyTransferListener()); success = true; } catch (Exception e) { } finally { if (input != null) { try { input.close(); } catch (IOException e) { } } if (!success) { try { ftp.deleteFile(remoteFile); } catch (Exception e) { } } } } // disconnect if (ftp != null && ftp.isConnected()) { try { ftp.disconnect(); } catch (IOException f) { // do nothing } }
I am trying to use multiple threads.
The following is a multiple code:
final CountDownLatch latch = new CountDownLatch(files.size()); ExecutorService pool = Executors.newFixedThreadPool(10); for (Map.Entry<File, String> entry : files.entrySet()) { final File localFile = entry.getKey(); final String remoteFile = entry.getValue(); pool.execute(new Runnable() { public void run() { FileInputStream input = null; try { input= new FileInputStream(localFile); ftp.storeFile(remoteFile, input); } catch (Exception e) { try { ftp.deleteFile(remoteFile); } catch (IOException e1) { } } finally { if (input != null) { try { input.close(); } catch (IOException e) { } } latch.countDown(); } } }); } try {
Is it correct? It works correctly, but it cannot improve speed. it took about 49000 - 51000 milliseconds, the same as the code without the stream.
I am testing speed using an intranet. The Internet will take more time.
How can I do to increase download speed?