Monitoring a Directory in Spring

Overview

The idea for this article was to add a new feature to the customer application so that it can process files copied to a directory at any time. Monitoring a directory is a common task found in many computer systems and applications. There are different options available in the java ecosystem to perform this type of job. Below are listed just a few of the possible solutions:

1. Java WatchService API: It was introduced in Java 7 and it is low level.
2. Apache commons io: The package monitor provides a component for monitoring file system events.
3. Spring Integration's file support: This is part of the Spring integration project which supports a wide range of Enterprise Integration Patterns.
4. Filewatch package from Spring Boot developer tools: It allows to watch the local filesystem for changes.

We will look at option number 4 as it is straightforward to implement plus our project is already built on top of the Spring framework.

The use case

We would like to create new customers in the app by copying a csv file to a specific location. The file will be read once it is fully transferred. Then, the csv file will be validated, processed and moved to a destination directory. Below is a sample of the csv file:

name, email, dob, info, vip
James Hart,james.hart@gmail.com,12/05/2002,Information for James,No
Will Avery,will.avery@gmail.com,23/10/1991,Information for Will,Yes
Anne Williams,anne.williams@gmail.com,12/05/1975,Information for Anne,No
Julia Norton,julia.norton@gmail.com,23/10/1984,Information for Julia,Yes

The csv file will always contain the header row with 5 columns in that particular order.

Prerequisites

This project is built with the below technologies:

1. Java 21
2. Spring Boot 3.1.5
3. Maven 3.9.1

You will need at least java 17+, Spring Boot 3+ and Maven 3.8+ run the source code.

Dependencies

One new dependency has to be added to the pom.xml file to be able to import the necessary classes.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-devtools</artifactId>
</dependency> 

This is all we need to start coding in the next section.

Monitor Directories

The class to watch specific directories for file changes is FileSystemWatcher. It comes with three constructors but most likely you will use the one accepting three arguments

public FileSystemWatcher(boolean daemon,
                         Duration pollInterval,
                         Duration quietPeriod)

Let's have a look at each argument

1. deamon: if a deamon thread monitors the changes. Set it to true if you want the thread (the monitoring) to be killed when the jvm is stopped.
2. pollInterval: time to wait between checking again for changes.
3. quietPeriod: time to wait after a change is detected to ensure. If you transfer large files to the directory, this will have to be taken into account to avoid file corruption.

As we want all of the above argument values to be altered without affecting the source code, custom proepeties are added to the application.properties file

application.file.watch.daemon=true
application.file.watch.pollInterval=5
application.file.watch.quietPeriod=1
application.file.watch.directory=C:\\workspace\\files\\customer

The application will scan the directory for modifications every 5 mins. And the change will be trigger/propagated after 1 minute. This will suffice because the csv files are small (less than 1 MB).
And the corresponding java code to load the values in it. Spring Boot 3 supports records which keeps it brief and simple.

@ConfigurationProperties(prefix = "application.file.watch")
public record FileWatcherProperties(
    @NotBlank String directory,
    boolean daemon,
    @Positive Long pollInterval,
    @Positive Long quietPeriod
) {}

The next step is to define the configuration class with the bean type FileSystemWatch

@Configuration
@EnableConfigurationProperties(FileWatcherProperties.class)
public class CustomerFileWatcherConfig {
// ommiting class members, constructor and logger

    @Bean
    FileSystemWatcher fileSystemWatcher() {
        var fileSystemWatcher = new FileSystemWatcher(
                properties.daemon(),
                Duration.ofMinutes(properties.pollInterval()),
                Duration.ofMinutes(properties.quietPeriod()));
        fileSystemWatcher.addSourceDirectory( 
            Path.of(properties.directory()).toFile());
        fileSystemWatcher.addListener(
            new CustomerAddFileChangeListener(fileProcessor));
        fileSystemWatcher.setTriggerFilter(
            f -> f.toPath().endsWith(".csv"));        
        fileSystemWatcher.start();
        logger.info(String.format("FileSystemWatcher initialized. 
            Monitoring directory %s",properties.directory()));

        return fileSystemWatcher;
    }

    @PreDestroy
    public void onDestroy() throws Exception {
        logger.info("Shutting Down File System Watcher.");
        fileSystemWatcher().stop();
    }
}

Let's review the method fileSystemWatcher():

1. First an instance of fileSystemWatcher is created passing the values from the bean properties as arguments. The bean properties object is managed by the spring container and injected via constructor.
2. The addSourceDirectory method is invoked. It takes a File representing the directory to be monitored.
3. The addListener methods takes a listener for file change events. FileChangeListener is a functional interface and its method onChange will be called when files have been changed.
4. Optionally an FileFilter can be setup in the method setTriggerFilter to restrict the files that trigger a change. A lambda expression, evaluated to a boolean, is used to limit the files to csv.
5. The Last method is start to initiate monitoring the source directory for changes.

Notice that there a predestroy hookup method to shutdown gracefully the watcher when the jvm is stopped.

Adding the listener

The second step to implement a FileChangeListener where the files can be processed. This interface is functional too, hence a lambda expression or method reference can be used. In our case it will be better to place it in its own class as it will improve readability.

public class CustomerAddFileChangeListener implements 
    FileChangeListener {
    private static Logger logger = LoggerFactory.getLogger( 
        CustomerAddFileChangeListener.class);

    private CustomerCSVFileProcessor fileProcessor;

    public CustomerAddFileChangeListener(
        CustomerCSVFileProcessor fileProcessor) {
        this.fileProcessor = fileProcessor;
    }

    @Override
    public void onChange(Set<ChangedFiles> changeSet) {
        for(ChangedFiles files : changeSet)
            for(ChangedFile file: files.getFiles())
                if (file.getType().equals(ChangedFile.Type.ADD))
                    fileProcessor.process(file.getFile().toPath());
    }
}

As it has been mentioned earlier the onChange method is invoked when files have been changed. The parameter Set is a collection of files changed (since the poll interval started). Once iterating through the changeSet, the files can be obtained from each ChangedFiles object via the getFiles method. Therefore a nested loop will work to get to the individual files. The single file is of type ChangedFile which provides access to the File and Type of the change/event (This is a enum with three values ADD, DELETE and MODIFY).

Back to the code, the if statement checks the type to make sure only ADD events will handled. The others will be ingnored. The class CustomerCSVFileProcessor performs all the work. It is not in the scope of this article but it will be outlined briefly in the next section.

Processing the file

The businness logic to handle the file is found in the process method (declared in the FileProcessor interface). It needs the CustomerService to save the Customers in the database, thus it is constructor-injected. The class CustomerCSVFileProcessor is marked as a compoment because it is also injected in the Watcher.

@Component
public class CustomerCSVFileProcessor implements FileProcessor {
    public static final int NUMBER_OF_COLUMNS = 5;

    private static Logger logger = LoggerFactory.getLogger( 
        CustomerCSVFileProcessor.class);

     private CustomerService customerService;

     public CustomerCSVFileProcessor(
         CustomerService customerService) {
         this.customerService = customerService;
     }

     public void process(Path file) {
        logger.info(String.format(
            "Init processing file %s",file.getFileName()));
        var parser = CSVParser.parse(file);
        parser.getRecords().forEach(this::processRecord);
        moveFile(file);
    }

    private void processRecord(CSVRecord csvRecord) {
       if (csvRecord.size() < NUMBER_OF_COLUMNS) {
           logger.info(String.format(
              "Line %d skipped. Not enough values.", 
              csvRecord.lineNumber()));
           return;
       }

       Customer customer = customerMapper.mapToCustomer(csvRecord);
       customer.setStatus(Customer.Status.ACTIVATED);
       customerService.save(customer);
       logger.info(String.format(
           "Saved customer %s in line %d",
           customer.getName(), 
           csvRecord.lineNumber()));
    }

    private static void moveFile(Path file) {
        try {
            var destinationFolder = Path.of( 
                file.getParent().toString() + OUTPUT_FOLDER );
            Files.move(
                file, 
                destinationFolder.resolve(file.getFileName()), 
                REPLACE_EXISTING);
            logger.info(String.format(
                 "File %s has been moved to %s",file.getFileName(), 
                 destinationFolder));
        } catch (IOException e) {
            logger.error("Unable to move file "+ file, e);
        }
    }
}

The method parses the csv file with the help of a CSVParser. Then each cvs row (depicted as a CSVRecord) is mapped to a customer entity and saved it the db. Finally, the foreach loop is completed and the file is moved to a distinct location.

Testing the app

Re-running the application will start the file system watcher bean from CustomerFileWatcherConfig config class. As soon as the app is up, a csv file with a few customers is copied to the monitored directory. After 5 mins the below trace is generated

CustomerFileWatcherConfig : FileSystemWatcher initialized. Monitoring directory C:\workspace\files\customer
CustomerCSVFileProcessor   : Init processing file customerFile.csv
CustomerCSVFileProcessor   : Saved customer James Hart in line 1
CustomerCSVFileProcessor   : Saved customer Will Avery in line 2
CustomerCSVFileProcessor   : Saved customer Anne Williams in line 3
CustomerCSVFileProcessor   : Saved customer Julia Norton in line 4
CustomerCSVFileProcessor   : File customerFile.csv has been moved to C:\workspace\files\customer\output

Indeed, the data rows in the file where added as customer entities to the database and the file moved successfully.

Conclusion

In this short article we have demonstrated how to configure a Spring app to monitor local directories. This functionality comes in handy to perform automated tasks. For instance, in combination with other services as such an FTP server transferring files to a directory and then the file watcher picking it up for processing.

Hope you enjoyed the reading. As usual, source code can be found in the github project.