How classpath scanning works behind the scenes?

Today, I was playing around with some basic file scanning and classpath scanning and the result was great!

Good and tradititonal java frameworks, like Spring and Jakarta do a lot of classpath scanning, but, how do it works behind the scenes? How do it is implemented? Let's discuss a basic approach for this, step-by-step!

Step 1: getting classpath URL

First of all, every code that you write live in plain bytes file, and the case for java is not that different. But, every java program pass by a compile step, that converts java code to bytecode in a .class file. So, when it's get compiled, it should live anywhere in the filesystem, and, when java loads an entire program, it usually look for files in a path definition, called "classpath".
This classpath information can be acessible in runtime, by getting it from the SystemClassLoader and returns an Enum of URLs.

// GET ALL CLASSPATH LOCATIONS
        Enumeration<URL> resources =
ClassLoader.getSystemClassLoader().getResources("");
        Iterator<URL> urlIterator = resources.asIterator();

This Enum of URLs can be iterated by converting it to an iterator.

Step 2: Iterate over classpath

The classpath is a normal path, like any path in filesystem, so, what we must do is iterate over this classpath to discover plain files.

// ITERATE ON THESE LOCATIONS AND TAKE ONLY CLASS FILES AND PREFIX BEFORE PACKAGE DIR
        ArrayList<File> files = null;
        String dir = null;
        while(urlIterator.hasNext()){
            dir = urlIterator.next().getFile();
            File file1 = new File(dir);
            files = iteratorScan(file1);
        }
        final String dirFinal = dir;

However, we don't know how is the depth where our .class files are hidden, so, we need a method to be invoked recursively until we discover all the class files that exists.

public static ArrayList<File> iteratorScan(File dirOrF){
        ArrayList<File> files = new ArrayList<>();
        Arrays.stream(dirOrF.listFiles()).forEach(f -> {
            if (f.isDirectory()){
                iteratorScan(f).forEach(files::add);
            } else {
                files.add(f);
            }
        });
        return files;
    }

We collect the files and add them into an array, and additionally we take the base packages directory for the next step.

Step 3: Instantiate a class loader

For every execution on JVM, a class loader is instantiated and used for load the classes on classpath and imports on classes to run the program. Actually, the entry point for every Java program is the main function (psvm) , so, if a class is not invoked and used on this block of code, it cannot be visible by the conventional means. However, we can instantiate a class loader too to load dynamically classes that are "invisible" to the main function. I take an URLClassLoader and pass to it an URL Array, where we can load the classes.

Fun fact: the URLClassLoader was build for the purpose to load servlets. That way, we can load classes over other communication protocols, like http and ftp.

Further, we can experiment this with other protocols. The filesystem protocol is indicated with the prefix file:

// INSTANTIATE NEW CLASSLOADER, SPECIFICATING FILE PROTOCOL AND POINTING TO THE FOLDER WHERE PACKAGE
        // IS LOCATED
        URLClassLoader urlClassLoader = new URLClassLoader(new URL[]{
                new URL(dirFinal.startsWith("file:") ? dirFinal : "file:" + dirFinal)
        });

Step 4: Iterate over the files and convert the path to a Full Qualified Class Name

The next step is simple string manipulation. We already have an Array of files, now, what we must do is to get the absolute path of the file and remove the prefix of path before the package path, remove the .class extension and replace "/" with "." to convert it to a full qualified class name:

// ITERATE FOR ALL CLASSES AND DO THE OPERATIONS
        files.forEach(file ->{
            try {

                // GET THE ABSOLUTE PATH OF CLASS FILE AND REMOVE THE PATH BEFORE PACKAGE, THE ".class" EXTENSION AND
                // REPLACE "/" FOR "."
                String fullQualifiedName = file.getAbsolutePath()
                        .replace(dirFinal, "")
                        .replace(".class", "").replace("/", ".");

Step 5: load the class

Finally, we can load the class.

// LOAD THE CLASS 
                Class<?> aClass = urlClassLoader.loadClass(fullQualifiedName);

Testing a class

To see if this process gone all right, we can test loading a class, instatiating it and executing one of it's methods.

But, we cannot do it like we normally do. We need to do it using the Reflection API, because the class is stored on the heap of the program like a Class instance. So, we gonna manipulate it that way, and it's a bit more complicated, but not that hard.

I gonna do a simple class that displays a simple log to the console:

package org.eron;

import java.time.LocalDate;

public class Logger {

    public void log(String message){
        LocalDate now = LocalDate.now();
        System.out.println("[" + now + "] " + message);
    }

}

Allright.
I know beforehand the name of this class. When I get the name of the class reflectively, the JVM will return me the full qualified name. For this case, the full qualified name is org.eron.Logger.

So, what I need to do is check if the name of the class ends with Logger.

 // INSTANTIATE AND TEST CLASS
                if(aClass.getName().endsWith("Logger")){

After this, I prepare a placeholder for the object I gonna instantiate with a variable of type Object, and will look up in the class for a no-args constructor. When finding it, I gonna invoke this constructor and get an instance of the class.

Object[] test = new Object[1];
                    Arrays.stream(aClass.getConstructors()).forEach(constructor -> {
                        Parameter[] parameters = constructor.getParameters();
                        if(parameters.length == 0){
                            try{
                                test[0] = constructor.newInstance();
                            } catch (InvocationTargetException e) {
                                throw new RuntimeException(e);
                            } catch (InstantiationException e) {
                                throw new RuntimeException(e);
                            } catch (IllegalAccessException e) {
                                throw new RuntimeException(e);
                            }
                        }
});

After I have the instance, now, just do the type casting for the correct type and invoke the method to test it. I recommend wrap it on a try/catch block to ignore the NullPointerException (I don't know why I get this exception, but, it don't affects the test execution).

Logger log = (Logger) test[0];
                    try{
                        log.log("Testing");
                        log.log("Anyways again");
                    } catch(NullPointerException ex){}

Full Code and Testing result

The full code is available on this gist.

And the output of the test is as follows.

/home/eronads/.sdkman/candidates/java/22.3.r19-grl/bin/java -javaagent:/home/eronads/idea-IC-223.8214.52/lib/idea_rt.jar=43935:/home/eronads/idea-IC-223.8214.52/bin -Dfile.encoding=UTF-8 -Dsun.stdout.encoding=UTF-8 -Dsun.stderr.encoding=UTF-8 -classpath /home/eronads/repos/java/aoplogger/target/classes org.eron.Main
[2023-01-19] Testing
[2023-01-19] Anyways again

Process finished with exit code 0