MATLAB Performance Benchmarking for CLI and GUI Performance

Although the MATLAB core has essentially stayed the same throughout
the years, the MATLAB language and runtime has with every release been
extended with more functionality as well as featured significant
improvements to the interpreter and Just-In-Time (JIT) compilation
engine. As developer of the
FEATool Multiphysics simulation toolbox for
MATLAB which features 10-years of both graphical user interface (GUI)
and command line interface (CLI) backwards compatibility (back to
R2009b), it would be very useful to be able to tell users with which
MATLAB version(s) is to be recommended for performance and/or other
reasons.

With this in mind, and as no official or comprehensive benchmarks of
MATLAB versions are available, both GUI and CLI script performance is
compared in the following for MATLAB versions R2011b (the subjectively
snappiest, most responsive, and currently preferred version in the
developers opinion) and R2019a (the latest version at the time of
writing). Timings and tests are performed with the FEATool
Multiphysics toolbox on a typical laptop with the Microsoft Windows 10
operating system and Intel CPU. The FEATool software is used for
benchmarking as the toolbox features a large number of pre-defined and
built-in validation test cases and fully automated (GUI) tutorials
which is used to automatically and consistently measure performance.

MATLAB GUI Performance

In the first step, the GUI model test suite is used for timing
runs. This test suite runs about 40 pre-defined multi-physics
simulation models (the test suite can be run by starting FEATool with
the featool testt command). The examples are completely automated so
that all GUI callbacks are automatically and consistently run just as
a user would manually do. The timing averages from five runs can be
seen in the table below.

From the results it is clear that MATLAB R2011b runs significantly
faster (around 25%) than the latest version R2019a even with the
improved JIT optimization as the subsequent MATLAB versions have. Note
that the UI layout toolkit used by FEATool has been specifically
developed and written to run fast as possible using procedural code.
The Mathworks GUI Layout Toolbox was tested in development and at the
time shown to have significant overhead and cost due to the object
oriented design, and just not performant enough for moderately complex
GUIs such as the ones use by FEATool. (Note that due to backwards
compatibility the FEATool GUI toolkit only uses get(h, ...) and set(h,
...) calls, and not struct call options introduced with handle
graphics hg2. Also these results do not apply to the newer web
graphics backend which was not tested.)

MATLAB CLI Performance

As these tests include everything from running the GUI to CAD geometry
and model pre-processing, mesh generation, linear equation solving,
plotting and visualization, we can not from the tests above
conclusively tell what is slower in newer versions of MATLAB. As
FEATool also is fully scriptable (one GUI action maps to an equivalent
MATLAB function) we can also run the tests without the GUI for
comparison, and the corresponding averaged results for 5 runs are
shown below (starting FEATool with featool testm runs all the
available script models in the
FEATool examples
directory).

Without using any GUI functionality the results are now reversed with
MATLAB R2019a being somewhat faster. As the FEATool code base is both
highly vectorized and optimized,
and since Finite Element Analysis (FEA) simulations makes significant
use of pre-compiled numerical libraries and sparse linear solvers such
as BLAS, LAPACK, Umfpack etc. which are not really influenced by JIT
improvements, a quite modest 10% speedup is maybe what one can expect
from any JIT improvements (instead of a major increase in speed).

All in all with these simple tests we can indeed see that current
MATLAB JIT optimizations have resulted in a modest speed increase on
average, however older versions of MATLAB are significantly more
responsive for using GUI functionality. To FEATool users one can
therefore suggest using version 2011b when using the GUI, and as
FEATool supports saving and exporting models to standard MATLAB
m-script files, subsequent large scale simulations can be run on the
command line in a later version of MATLAB to take advantage of any
speed increases.