Apache Camel ftp consumer downloads the same files again and again

Question

I have the following spring configuration

<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd"> <bean id="downloadLogger" class="com.thomsonreuters.oa.sdi.camel.DownloadLogger" /> <bean id="fileFilter" class="com.thomsonreuters.oa.sdi.camel.IgnoreReadyFilesFilter" /> <camelContext xmlns="http://camel.apache.org/schema/spring"> <route> <from uri="ftp://url_to_ftp?password=*******&amp;noop=true&amp;stepwise=false&amp;binary=true&amp;consumer.delay=10s&amp;recursive=true&amp;filter=#fileFilter" /> <process ref="downloadLogger" /> <to uri="file:data/outbox" /> </route> </camelContext> </beans> 

On the ftp side, I have 3 folders with files that I want to download. I want to run the following script:

• On ftp, a fixed number of files (for isntance 5) from the first data transfer user uploads these files to the destination folder
• On the second attempt to upload files, the ftp state remains the same (5 files), and the ftp camel consumer simply does nothing (except checking for new files).
• 2 new files come to ftp, and with this pull data, the consumer downloads only these new two files

Currently my current solutions download all the files every time I start the dataload process, how can I manage the information about the downloaded files to prevent duplicates from downloading (I mean the files already copied from ftp), I can write my own filter, which will filter out already downloaded files, but I believe that there should be a built-in function that will give me control over this (maybe idempotentRepository, actually I'm not sure) ...

Answer 1

You need to use the constant idempotent repository if you want Camel to remember which files it previously downloaded between reboots.

You need to set this parameter on the ftp endpoint: idempotentRepository

For more details see: http://camel.apache.org/file2 (Note: FTP component inherits parameters from file component.)

There are several examples of using different stores on the wiki page. And you can also create your own store.

Answer 2

Finally, I get the following solution:

 public class SdiRouteBuilder extends RouteBuilder { @Override public void configure() throws Exception { from("ftp:// login@url _to_ftp/RootFolder?" + "password=******&noop=true&stepwise=false&binary=true&consumer.delay=10s&recursive=true&filter=#fileFilter") .idempotentConsumer(header("CamelFileName"), FileIdempotentRepository.fileIdempotentRepository(new File("data", "repo.dat"))) .process(new DownloadLogger()) .to("file:data/outbox"); } } 

Answer 3

Maybe @endryha will respond well in 2011, but not with Camel 2.20.1

In Camel 2.20.1, this code will create two idempotentRepository

• ftp default memory idempotentRepository
• idempotentConsumer custom idempotentRepository (in this case, the file)

So the correct way to use idempotentRepository is (I remove most of the options for readability)

 "ftp:// login@url _to_ftp/RootFolder?&idempotent=true&idempotentRepository=#myIdempotentRepo" 

and bean

 @Bean private IdempotentRepository<String> myIdempotentRepo() { return FileIdempotentRepository.fileIdempotentRepository(new File("data", "repo.dat"); }