How to effectively control changes at a remote location?

Question

We need to track changes in the remote system file that we receive via FTP, SMB. We do not have SSH access to the remote / os system. Our only look at the remote system is what FTP or Samba allows.

What we do today:

periodically scan the entire catalog, create a view in memory to perform our actions, and then combine it with what we have in the database.

What we would like to do:

The ability to determine if a directory has changed, and therefore if parsing is required. Ideally, you never need to understand completely. We do not want to rely too much on the capabilities of the OS (inodes ...), because this can change from installation to another.

The main goal . This process starts to slow down when the amount of data is very large. Only a few% of this date are new and need analysis. How to disassemble and add only this part to our database?

The lines that we are discussing at the moment:

• Check Folder Size
• using checksum in file
• Checking the last folder / file change date

What we really want:

Some input and best practice, because this problem throws quite general goals and should have the bean already discussed, and we don’t want to end up doing something too complicated at this stage.

Thanks in advance, a bunch of fellow developers; -)

We use the java / spring / hibernate stack, but I don't think this is very important.

: , FTP- . , .

Answer 1

Java (rdp4j) FTP : // . lastModified .

. , FtpDirectory MyListener API:

package example

import java.util.concurrent.TimeUnit;
import com.github.drapostolos.rdp4j.DirectoryPoller;
import com.github.drapostolos.rdp4j.spi.PolledDirectory;

public class FtpExample {

    public static void main(String[] args) throws Exception {
        String host = "ftp.mozilla.org";
        String workingDirectory = "pub/addons";
        String username = "anonymous";
        String password = "anonymous";
        PolledDirectory polledDirectory = new FtpDirectory(host, workingDirectory, username, password);

        DirectoryPoller dp = DirectoryPoller.newBuilder()
        .addPolledDirectory(polledDirectory)
        .addListener(new MyListener())
        .setPollingInterval(10, TimeUnit.MINUTES)
        .start();

        TimeUnit.HOURS.sleep(2);

        dp.stop();
    }
}

Answer 2

, , . . , .

, , , .

- (, rsync, robocopy), / . " " - rsync.

Answer 3

, FTP SMB. , , :

• : , ,
• : ,

, , .

Answer 4

/ , SHA1 () SHA512. , . ( Git):

• , ;
• , - .

, f d .

You can also put the directory under version control using Git (or Mercurial or whatever), periodically git addeverything in it, use git statusto find out what has been updated and git commitchanges.