A new Java ORM and why we need it

I've created a framework about immutable data and ORM for immutable data.

Project name: Jimmer

Project Home: https://babyfish-ct.github.io/jimmer-doc/

Overview

Jimmer is divided into two parts, jimmer-core and jimmer-sql.

1. jimmer-core: Immutable data
2. jimmer-sql: ORM based on jimmer-core

Their respective responsibilities are as follows

1. jimmer-core

   Porting a well-known project immer for Java, modifying immutable objects in the way of mutable objects.

   Jimmer can be used in any context where immutable data structures are required to replace java records. Immutable data structures allow for (effective) change detection: if the reference to the object hasn't changed, then neither has the object itself. Also, it makes cloning relatively cheap: unchanged parts of the data tree do not need to be copied and are shared in memory with older versions of the same state.

   In general, these benefits are achieved by ensuring that you never change any properties of an object or list, but always create a changed copy. In practice, this can lead to very cumbersome code to write, and it is easy to accidentally violate these constraints. Jimmer will help you follow the immutable data paradigm by addressing the following pain points:

   1. Jimmer will detect an unexpected mutation and throw an error.
   2. Jimmer will eliminate the need to create the typical boilerplate code required when doing deep updates to immutable objects: without Jimmer, you would need to manually make copies of objects at each level. Usually by using a lot of copy construction.
   3. When using JImmer, changes are made to the draft object, which records the changes and takes care of creating the necessary copies without affecting the original.

   When using Jimmer, you don't need to learn specialized APIs or data structures to benefit from paradigms.

   In addition, to support ORM, Jimmer adds object dynamics to immer. Any property of an object is allowed to be missing.

   • Missing properties cause exceptions when accessed directly by code
   • Missing properties are automatically ignored during Jackson serialization and will not cause an exception

2. jimmer-sql

   ORM based on jimmer-core dynamic immutable objects.

   In terms of implementation, jimmer-sql is incredibly lightweight, with no dependencies other than JDBC, not even some lightweight encapsulation for database connection like SqlSession of myBatis.

   Similar to QueryDsl, JOOQ, JPA Criteria, with strongly typed SQL DSLs, most SQL errors are reported at compile time rather than as runtime exceptions.

   However, strongly-typed SQL DSL does not conflict with Native SQL. Through elegant API, Native SQL is mixed into strongly-typed SQL DSL, and developers are encouraged to use features specific to specific database products, such as analytical functions and regularization.

   In addition to all ORM's must-have features, jimmer-sql provides 4 other features that far exceed other ORMs:

 - [Save command](https://babyfish-ct.github.io/jimmer-doc/docs/jimmer-sql/mutation/save-command)
 - [Object fetcher](https://babyfish-ct.github.io/jimmer-doc/docs/jimmer-sql/query/fetcher)
 - [Dynamic table joins](https://babyfish-ct.github.io/jimmer-doc/docs/jimmer-sql/table-join#dynamic-join)
 - [Smarter pagination queries](https://babyfish-ct.github.io/jimmer-doc/docs/jimmer-sql/query/pagination).

 These four powerful functions that are clearly different from other ORMs are the goals pursued by the ORM part of this framework.

1. Make User Bean powerful enough

1.1 Use immutable data, but support temporary mutable proxies.

@Immutable
public interface TreeNode {
    String name();
    TreeNode parent();
    List<TreeNode> childNodes();
}

The annotation processor will generate a mutable derived interface for the user: TreeNodeDraft. User can use it like this

// Step1: Create object from scratch
TreeNode oldTreeNode = TreeNodeDraft.$.produce(root ->  
    root
        .setName("Root")
        .addIntoChildNodes(child ->
            child.setName("Drinks")        
        )
        .addIntoChildNodes(child ->
            child.setName("Breads")        
        )
);

// Step2: Create object based on existing object
TreeNode newTreeNode = TreeNodeDraft.$.produce(
    oldTreeNode, // existing object
    root ->
      root.childNodes(false).get(0) // Get child proxy
          .setName("Dranks+"); // Change child proxy
);

System.out.println("Old tree node: ");
System.out.println(oldTreeNode);

System.out.println("New tree node: ");
System.out.println(newTreeNode);

The final print result is as follows

Old tree node: 
{"name": "Root", childNodes: [{"name": "Drinks"}, {"name": "Breads"}]}
New tree node: 
{"name": "Root", childNodes: [{"name": "`Drinks+`"}, {"name": "Breads"}]}

1.2 Dynamic object.

Any property of the data object can be unspecified.

1. Direct access to unspecified properties causes an exception.
2. Using Jackson serialization, Unspecified properties will be ignored, without exception throwing.

TreeNode current = TreeNodeDraft.$.produce(current ->
    node
        .setName("Current")
        .setParent(parent -> parent.setName("Father"))
        .addIntoChildNodes(child -> child.setName("Son"))
);

// You can access specified properties
System.out.println(current.name());
System.out.println(current.parent());
System.out.println(current.childNodes());

/*
 * But you cannot access unspecified fields, like this
 *
 * System.out.println(current.parent().parent());
 * System.out.println(
 *     current.childNodes().get(0).childNodes()
 * );
 *
 * , because direct access to unspecified 
 * properties causes an exception.
 */

/*
 * Finally You will get JSON string like this
 * 
 * {
 *     "name": "Current", 
 *     parent: {"name": "Father"},
 *     childNodes:[
 *         {"name": "Son"}
 *     ]
 * }
 *
 * , because unspecified will be ignored by 
 * jackson serialization, without exception throwing.
 */
String json = new ObjectMapper()
    .registerModule(new ImmutableModule())
    .writeValueAsString(current);

System.out.println(json);

Because entity objects are dynamic, users can build arbitrarily complex data structures. There are countless possibilities, such as

1. Lonely object, for example
   

   TreeNode lonelyObject = TreeNodeDraft.$.produce(draft ->
       draft.setName("Lonely object")
   );
   

2. Shallow object tree, for example
   

   TreeNode shallowTree = TreeNodeDraft.$.produce(draft ->
       draft
           .setName("Shallow Tree")
           .setParent(parent -> parent.setName("Father"))
           .addIntoChildNodes(child -> parent.setName("Son"))
   );
   

3. Deep object tree, for example
   

   TreeNode deepTree = TreeNodeDraft.$.produce(draft ->
       draft
           .setName("Deep Tree")
           .setParent(parent -> 
                parent
                    .setName("Father")
                    .setParent(deeperParent ->
                        deeperParent.setName("Grandfather")
                    )
           )
           .addIntoChildNodes(child -> 
               child
                   .setName("Son")
                   .addIntoChildNodes(deeperChild -> 
                       deeperChild.setName("Grandson");
                   )
           )
   );
   

This object dynamism, which includes countless possibilities, is the fundamental reason why jimmer's ORM can provide more powerful features.

2. ORM base on immutable object.

In jimmer's ORM, entities are also immutable interfaces

@Entity
public interface TreeNode {

    @Id
    @GeneratedValue(
        strategy = GenerationType.SEQUENCE,
        generator = "sequence:TREE_NODE_ID_SEQ"
    )
    long id();

    @Key // jimmer annotation, `name()` is business key,
    // business key will be used when `id` property is not specified
    String name();

    @Key // Business key too
    @ManyToOne
    @OnDelete(DeleteAction.DELETE)
    TreeNode parent();

    @OneToMany(mappedBy = "parent")
    List<TreeNode> childNodes();
}

Note!

Although jimmer uses some JPA annotations to complete the mapping between entities and tables, jimmer is not JPA.

2.1 Save arbitrarily complex object tree into database

1. Save lonely entity
   

   sqlClient.getEntities().save(
       TreeNode lonelyObject = TreeNodeDraft.$.produce(draft ->
           draft
               .setName("RootNode")
               .setParent((TreeNode)null)
       )
   );
   

2. Save shallow entity tree
   

   sqlClient.getEntities().save(
       TreeNode lonelyObject = TreeNodeDraft.$.produce(draft ->
           draft
               .setName("RootNode")
               .setParent(parent ->
                   parent.setId(100L)
               )
               .addIntoChildNodes(child ->
                   child.setId(101L)
               )
               .addIntoChildNodes(child ->
                   child.setId(102L)
               )
       )
   );
   

3. Save deep entity tree
   

   sqlClient.getEntities().saveCommand(
       TreeNode lonelyObject = TreeNodeDraft.$.produce(draft ->
           draft
               .setName("RootNode")
               .setParent(parent ->
                   parent
                       .setName("Parent")
                       .setParent(grandParent ->
                           grandParent.setName("Grand parent")
                       )
               )
               .addIntoChildNodes(child ->
                   child
                       .setName("Child-1")
                       .addIntoChildNodes(grandChild ->
                           grandChild.setName("Child-1-1")
                       )
                      .addIntoChildNodes(grandChild ->
                           grandChild.setName("Child-1-2")
                       )
               )
               .addIntoChildNodes(child ->
                   child
                       .setName("Child-2")
                       .addIntoChildNodes(grandChild ->
                           grandChild.setName("Child-2-1")
                       )
                       .addIntoChildNodes(grandChild ->
                           grandChild.setName("Child-2-2")
                       )
               )
       )
   ).configure(it ->
       // Auto insert associated objects 
       // if they do not exists in database
       it.setAutoAttachingAll()
   ).execute();
   

2.2 Query arbitrarily complex object trees from a database

1. Select root nodes from database (TreeNodeTable is a java class generated by annotation processor)
   

   List<TreeNode> rootNodes = sqlClient
       .createQuery(TreeNodeTable.class, (q, treeNode) -> {
           q.where(treeNode.parent().isNull()) // filter roots
           return q.select(treeNode);
       })
       .execute();
   

2. Select root nodes and their child nodes from database (TreeNodeFetcher is a java class generated by annotation processor)
   

   List<TreeNode> rootNodes = sqlClient
       .createQuery(TreeNodeTable.class, (q, treeNode) -> {
           q.where(treeNode.parent().isNull()) // filter roots
           return q.select(
               treeNode.fetch(
                   TreeNodeFetcher.$
                       .allScalarFields()
                       .childNodes(
                           TreeNodeFetcher.$
                               .allScalarFields()
                       )
               )
           );
       })
       .execute();
   

3. Query the root nodes, with two levels of child nodes

   You have two ways to do it

-   Specify a deeper tree format

    ```java
    List<TreeNode> rootNodes = sqlClient
    .createQuery(TreeNodeTable.class, (q, treeNode) -> {
        q.where(treeNode.parent().isNull()) // filter roots
        return q.select(
            treeNode.fetch(
                TreeNodeFetcher.$
                    .allScalarFields()
                    .childNodes( // level-1 child nodes
                        TreeNodeFetcher.$
                            .allScalarFields()
                            .childNodes( // level-2 child nodes
                                TreeNodeFetcher.$
                                    .allScalarFields()
                            )
                    )
            )
        );
    })
    .execute();
    ```

-   You can also specify depth for self-associative property, this is better way

    ```java
    List<TreeNode> rootNodes = sqlClient
    .createQuery(TreeNodeTable.class, (q, treeNode) -> {
        q.where(treeNode.parent().isNull()) // filter roots
        return q.select(
            treeNode.fetch(
                TreeNodeFetcher.$
                    .allScalarFields()
                    .childNodes(
                        TreeNodeFetcher.$
                            .allScalarFields(),
                        it -> it.depth(2) // Fetch 2 levels
                    )
            )
        );
    })
    .execute();
    ```

1. Query all root nodes, recursively get all child nodes, no matter how deep
   

   List<TreeNode> rootNodes = sqlClient
   .createQuery(TreeNodeTable.class, (q, treeNode) -> {
       q.where(treeNode.parent().isNull()) // filter roots
       return q.select(
           treeNode.fetch(
               TreeNodeFetcher.$
                   .allScalarFields()
                   .childNodes(
                       TreeNodeFetcher.$
                           .allScalarFields(),
   
                       // Recursively fetch all, 
                       // no matter how deep
                       it -> it.recursive() 
                   )
           )
       );
   })
   .execute();
   

2. Query all root nodes, it is up to the developer to control whether each node needs to recursively query child nodes
   

   List<TreeNode> rootNodes = sqlClient
   .createQuery(TreeNodeTable.class, (q, treeNode) -> {
       q.where(treeNode.parent().isNull()) // filter roots
       return q.select(
           treeNode.fetch(
               TreeNodeFetcher.$
                   .allScalarFields()
                   .childNodes(
                       TreeNodeFetcher.$
                           .allScalarFields(),
                       it -> it.recursive(args ->
                           // - If the node name starts with `Tmp_`, 
                           // do not recursively query child nodes.
                           //
                           // - Otherwise, 
                           // recursively query child nodes.
                           !args.getEntity().name().startsWith("Tmp_")
                       )
                   )
           )
       );
   })
   .execute();
   

2.3 Dynamic table joins.

In order to develop powerful dynamic queries, it is not enough to support dynamic where predicates, but dynamic table joins are required.

@Repository
public class TreeNodeRepository {

    private final SqlClient sqlClient;

    public TreeNodeRepository(SqlClient sqlClient) {
        this.sqlClient = sqlClient;
    }

    public List<TreeNode> findTreeNodes(
        @Nullable String name,
        @Nullable String parentName,
        @Nullable String grandParentName
    ) {
        return sqlClient
            .createQuery(TreeNodeTable.class, (q, treeNode) -> {
               if (name != null && !name.isEmpty()) {
                   q.where(treeNode.name().eq(name));
               }
               if (parentName != null && !parentName.isEmpty()) {
                   q.where(
                       treeNode
                       .parent() // Join: current -> parent
                       .name()
                       .eq(parentName)
                   );
               }
               if (grandParentName != null && !grandParentName.isEmpty()) {
                   q.where(
                       treeNode
                           .parent() // Join: current -> parent
                           .parent() // Join: parent -> grand parent
                           .name()
                           .eq(grandParentName)
                   );
               }
               return q.select(treeNode);
            })
            .execute();
    }
}

This dynamic query supports three nullable parameters.

1. When the parameter parentName is not null, the table join current -> parent is required
2. When the parameter grandParentName is not null, you need to join current -> parent -> grandParent

When the parameters parentName and grandParent are both specified, the table join paths current -> parent and current -> parent -> grandParent are both added to the query conditions. Among them, current->parent appears twice, jimmer will automatically merge the duplicate table joins.

This means

`current -> parent` 
+ 
`current -> parent -> grandParent` 
= 
--+-current
  |
  \--+-parent
     |
     \----grandParent

In the process of merging different table join paths into a join tree, duplicate table joins are removed.

The final SQL is

select 
    tb_1_.ID, tb_1_.NAME, tb_1_.PARENT_ID
from TREE_NODE as tb_1_

/* Two java joins are merged to one sql join*/
inner join TREE_NODE as tb_2_ 
    on tb_1_.PARENT_ID = tb_2_.ID

inner join TREE_NODE as tb_3_ 
    on tb_2_.PARENT_ID = tb_3_.ID
where
    tb_2_.NAME = ? /* parentName */
and
    tb_3_.NAME = ? /* grandParentName */

2.4 Automatically generate count-query by data-query.

Pagination query requires two SQL statements, one for querying the total row count of data, and the other one for querying data in one page, let's call them count-query and data-query.

Developers only need to focus on data-count, and count-query can be generated automatically.

// Developer create data-query
ConfigurableTypedRootQuery<TreeNodeTable, TreeNode> dataQuery = 
    sqlClient
        .createQuery(TreeNodeTable.class, (q, treeNode) -> {
            q
                .where(treeNode.parent().isNull())
                .orderBy(treeNode.name());
            return q.select(book);
        });

// Framework generates count-query
TypedRootQuery<Long> countQuery = dataQuery
    .reselect((oldQuery, book) ->
        oldQuery.select(book.count())
    )
    .withoutSortingAndPaging();

// Execute count-query
int rowCount = countQuery.execute().get(0).intValue();

// Execute data-query
List<TreeNode> someRootNodes = 
    dataQuery
        // limit(limit, offset), from 1/3 to 2/3
        .limit(rowCount / 3, rowCount / 3)
        .execute();