How to Verify that MultiResourceItemReader Updates Resources Every Time a Job Runs

Question

I have a java application that uses spring task scheduling and batch jobs. I rely on MultiResourceItemReaderin jobto read certain files from the directory, process them and delete these files. The external process is responsible for periodically placing new files in this directory. But the problem is that every time it starts job, it tries to read the same file resources that were present during the launch of the application, and therefore fails because these resources have disappeared, and now there are new files.

The question is how to configure the application so that the resource property is evaluated for each scheduled execution of the given job.

The corresponding beans are inserted below:

  <bean id="multiResourceReader" class="org.springframework.batch.item.file.MultiResourceItemReader">
    <property name="resources" value="file:/opt/data/*.csv" />
    <property name="delegate" ref="testFlatFileItemReader" />
  </bean>

  <batch:job id="MyJob">
    <batch:step id="readandstore">
      <batch:tasklet>
        <batch:chunk reader="multiResourceReader" writer="oracleItemWriter" commit-interval="10" />
      </batch:tasklet>
    </batch:step>
  </batch:job>

  <bean id="runScheduler" class="com.myapp.Scheduler">
    <property name="jobLauncher" ref="jobLauncher" />
    <property name="job" ref="MyJob" />
  </bean>

  <task:scheduled-tasks>
    <task:scheduled ref="runScheduler" method="run" cron="*/30 * * * * *" />
  </task:scheduled-tasks>

Answer 1

I managed to solve the problem, which I published myself after I did not receive an answer to it for almost a week.

multiResourceReader StepListener <batch:tasklet>, StepExecutionListener. : , . StepListener filePattern, , multiResourceReader . beans:

  <batch:job id="MyJob">
    <batch:step id="readandstore">
      <batch:tasklet>
        <batch:chunk reader="multiResourceReader" writer="oracleItemWriter" commit-interval="10" />
        <batch:listeners>
          <batch:listener ref="StepListener" />
        </batch:listeners>
      </batch:tasklet>
    </batch:step>
  </batch:job>

  <bean id="multiResourceReader" class="org.springframework.batch.item.file.MultiResourceItemReader">
    <property name="delegate" ref="csvFileItemReader" />
  </bean>

  <bean id="StepListener" class="com.myapp.StepListener">
    <property name="filePattern" value="file:/opt/data/*.csv" />
  </bean>

com.myapp.StepListener:

package com.myapp;

import java.io.File;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.ExitStatus;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.core.StepExecutionListener;
import org.apache.commons.io.FileUtils;
import org.springframework.batch.item.file.MultiResourceItemReader;
import org.springframework.beans.BeansException;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.core.io.Resource;

public class StepListener implements StepExecutionListener, ApplicationContextAware {

  private static final Logger logger = Logger.getLogger(StepListener.class.getName());
  private Resource[] resources;
  private ApplicationContext applicationContext;
  private String filePattern;

  @Override
  public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
    this.applicationContext = applicationContext;
  }

  public void setFilePattern(String filePattern) {
    this.filePattern = filePattern;
  }

  @Override
  public void beforeStep(StepExecution stepExecution) {

    MultiResourceItemReader reader = (MultiResourceItemReader) applicationContext.getBean("multiResourceReader");
    try {
      resources = applicationContext.getResources(filePattern);
      reader.setResources(resources);
    } catch (IOException ex) {
      logger.log(Level.SEVERE, "Unable to set file resources to bean multiResourceItemReader", ex);
    }
  }

  @Override
  public ExitStatus afterStep(StepExecution stepExecution) {

    if (stepExecution.getExitStatus().equals(ExitStatus.COMPLETED)
            && stepExecution.getStatus().equals(BatchStatus.COMPLETED)
            && resources.length > 0) {

      for (Resource resource : resources) {
        try {
          File oldFile = new File(resource.getFile().getAbsolutePath());
          File newFile = new File(resource.getFile().getAbsolutePath() + ".processed");
          FileUtils.copyFile(oldFile, newFile);
          oldFile.delete();
        } catch (IOException ex) {
          logger.log(Level.SEVERE, "Encountered problem when trying to remove the processed file(s)", ex);
        }
      }
    }

    return stepExecution.getExitStatus();
  }
}

Answer 2

bean multiResourceReader singleton, 1. bean , .

bean multiResourceReader , . bean.

Answer 3

"step" bean "multiResourceReader":

  

Answer 4

I had the same problem, but I decided with a different approach, maybe it can be useful, I work with FlatFileItemReader, but the problem is the same.

The solution was to use the ref properties resource with a bean that handles jobParams. Thus, each time you restart the step, this is a new instance with a new value, the following code is how I do it:

<bean id="customFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
<property name="resource" ref="customFile"/><!-- avoid use directly jobParams here -->
<property name="linesToSkip" value="0" />
<property name="encoding" value="ISO-8859-1" />
<property name="comments" value="*, \u001A" />
<property name="lineMapper">
    <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
        <property name="lineTokenizer">
            <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
                <property name="delimiter" value=";" />
            </bean>
        </property>
        <property name="fieldSetMapper">
            <bean class="br.com.sample.batch.CustomFieldSetMapperSIAW"/>
        </property>
    </bean>
</property>

<bean id="customFile" class="org.springframework.core.io.FileSystemResource" scope="step">
<constructor-arg value="#{jobParameters[arquivoGSIN]}"></constructor-arg>