(In a hurry? Jump to the conclusion!)

This is an attempt to measure how good are front-end frameworks in dealing with simple errors.

I took the implementations of TodoMVC made in 5 different frameworks: VanillaJS, React, Vue, Svelte and Elm.

Then I injected 7 different errors and I analyzed the behavior.

After injecting error in the 5 implementations, I measured the outcome using this scale (ordered from best to worse):

• A. Impossible to introduce such error
• B. Compiler error
• C.
  • Nothing render on the screen and there is an error in the console at start
  • The app render and there are no bugs
• D. The app render on the screen, it has a bug and an error in console at start
• E. The app render on the screen, it has a bug and an error in console during execution
• F. The app render on the screen but it has a bug, end there are no errors in the console

They are ordered in a way that I find more desirable. For example, having the app not showing at all (rank C) is better than having it appear on the screen but with bugs (ranks D, E, F) because in the first case it's easier to note that there is something wrong.

The best case is when errors are not even possible to be added at all (rank A) or are intercepted by the compiler (rank B). The compiler-error case is only possible for Elm, Svelte and React JSX. Everything else is not compiled.

The worst case is when the app renders on the screen and there are no errors in the console whatsoever (case F). It means that if we don't test our application carefully we are likely to deploy a bug in production.

I also rated the accuracy of the errors:

• Very accurate, errors that possibly suggest the solution to the problem (add a + to the score).
• Regular, errors that give a good explanation and position of the error (no changes to the score).
• Unclear, error with bad explanation or wrong indication about the error position, (add a - to the score).

Disclaimers

• My bias: I work primarily with Elm and I am an advocate for functional programming.
• I use the word "front-end frameworks" in a very loose way here, so that it includes all of them because some of them are languages with a compiler, others are just Javascript libraries.
• I changed the code with a plain text editor. No linters or any other plugin.
• I didn't run any automatic tests. A good development environment should be able to detect all these issues through the assistance of the IDE or the test suite. Here I am judging the naked framework.
• I didn't use Typescript or Babel to transpile the code. I only used the Svelte and Elm compiler as these are integrated parts of the language.
• Using the code of the TodoMVC application. The result may be affected by the quality of that particular implementation. Different implementations written using the same framework could get different scores.
• Sometimes it is not simple to introduce exactly the same error in different languages/implementations so sometimes I had to modify the code slightly.
• All the code modifications are available at github.com/lucamug/framework-resilience. Each of the seven errors has its own branch so you can verify how these errors were implemented.

Let's get started!

1. Misspelling one HTML element

• Code diff.

I changed the opening of a <span> element to <spam>, leaving the closure intact, in the footer of the application: <spam>...</span>

Outcome

Vanillajs

The application renders but the footer is not there. There is an error in the console: Cannot set property 'className' of null.

This is a typical example of an unhelpful error, the text does not contain any hint on what could be the cause and the file where the error happens (view.js) is different from where the error is (index.html).

Rank: D-

React

The application doesn't render and shows an error in the console at start:

Uncaught Error: Parse Error: Line 33: Expected corresponding JSX closing tag for spam

This would be intercepted by the JSX precompiler.

Rank: B

Vue

The application renders, but the footer has a wrong layout.

There is a clear error in the console:

vue.js:525 [Vue warn]: Unknown custom element: <spam> - did you register the component correctly?

Rank: D

Svelte

Nice compiler error

ParseError: </span> attempted to close an element that was not open

Rank: B+

Elm

Changing only one element is not possible in Elm. A span element, for example, is created with span [] [ text "content" ].

Rank: A

2. Misspelling two HTML elements

• Code diff.

This time we use spam both in the opening and in the closing element. This is a subtle mistake because we could have wanted to use a custom HTML type. But let's assume that is a mistake and see which framework detects it.

Details of code changes

Outcome

Elm in the only framework that has a compiler error:

In Elm, to create a spam element you need to write

node "spam" [] [ text "content" ]

instead of

spam [] [ text "content" ]

Other frameworks either just work or they have a warning:

This is the case of Vue that has this in the console:

vue.js:525 [Vue warn]: Unknown custom element: <spam> - did you register the component correctly? For recursive components, make sure to provide the "name" option.

• Rank Elm: B+
• Rank Vue: C+
• Everything else: C

3. Forgetting a space between a and href

• Code diff.

Just replace <a href="..."> with <ahref="...">. I did this in the footer button "Active" that displays the active todos.

Outcome

VanillaJS

No errors at all for VanillaJS, but the button "Active" doesn't work.

Rank: F

React

React return the error Unexpected token <ahref="#/" and it doesn't render the application. Another error that will be detected by the JSX compiler.

Rank: B

Vue

No errors at all, the same as VanillaJS, and the footer is also broken:

Rank: F

Svelte

Compiler error ParseError: Expected >

Rank: B

Elm

Is difficult to add this error as a and href are separated by a [ in Elm:

a [ href = "url" ] [ text = "label" ]

so I added the error this way

a [ ahref = "url" ] [ text = "label" ]

That generate this compiler error I cannot find a 'ahref' variable: These names seem close though: href

Rank: B+

4. Misspelling the content of a href

• Code diff.

Remove the content of href in the "Active" button. From href="#/active" to href="#/activa"

Outcome

Changing the content of a string, as expected, goes undetected by all frameworks except VanillaJS and makes the link "Active" not working anymore.

VanillaJS give an error at runtime:

controller.js:254 Uncaught TypeError: this[("show" + activeRoute)] is not a function.

Strangely enough, the Elm application kept working also with this modification!

I investigated the reason and I found out that

1. The Elm application apply changes "onClick" instead of waiting for the url changing
2. The Elm application store the entire model in the local storage while all other applications only store the list of todo items

As I consider this some kind of "cheating", I downgraded Elm to the same rank of all other applications.

This error is telling us that having strings in the code is usually a bad idea. This is why in Elm, and probably in other frameworks too, we usually write the routing differently.

Specifically

1. We create a custom type containing all possible routes
2. We create a function routeToString that convert such type into a string containing the url of the route

So, using the function routeToString as href when we create links, assure that this type of errors cannot happen. It also has the nice side effect of making the code more maintainable if, in the future, we decide to change the url format or name.

Rank: VanillaJS get an E-, all other applications get an F.

5. Corrupting the initial state

• Code diff.

We change the code where the state is initialized, changing the name of an object key or a variable.

Outcome

Vanillajs

There is an error in the console:

store.js:21 Uncaught ReferenceError: todos is not defined

and the app is not working. The error happens only if the local storage is missing.

Rank: D

React

Unclear error in the console:

app.jsx:96 Uncaught TypeError: Cannot read property 'filter' of undefined

but the file with the error is todoModel.js:18

The app doesn't render.

Rank: C-

Vue

Unclear error in the console:

vue.js:525 [Vue warn]: Property or method "todos" is not defined on the instance but referenced during render. Make sure to declare reactive data properties in the data option.

Rank: C-

Svelte

The compiler gives a warning

Plugin svelte: 'items' is not defined

but not an error, so it is still building the application.

The application doesn't render and in the console there is the same error Plugin svelte: 'items' is not defined

Rank: Between B- and C-. I will go with B- as maybe is possible to configure the compiler so that it will stop the compilation, instead of just giving a warning.

Elm

In Elm we change a key of a record from entries to entriez. This is the compiler error:

Rank: B+

6. Corrupting the function that add a new todo item

• Code diff.

Here I mostly change the variable name containing the title of the new todo.

The interesting part of this error is that it reveals itself only when users are adding a new item, so it is harder to detect compared to the others.

Outcome

Vanillajs

Error when adding a new todo:

controller.js:17 Uncaught ReferenceError: title is not defined

Rank: E

React

Error when adding a new todo:

todoModel.js:36 Uncaught ReferenceError: title is not defined

Rank: E

Vue

Changed "value" to "valua". Error when adding a new todo:

app.js:72 Uncaught ReferenceError: value is not defined

Rank: E

Svelte

Changed "description" to "descriptiom". Error when adding a new todo:

bundle.js:1031 Uncaught ReferenceError: description is not defined

Rank: E-

Elm

Changed "desc" to "descr". Compiler error:

I cannot find a 'descr' variable. These names seem close though: 'desc'

Rank B+

7. Forgetting to add the method .length to calculate the active todos

• Code diff.

Usually the total quantity of active todo is calculated by filtering all todos and counting the remaining items with length.

I remove ,length where possible. If the implementation was using a different system for counting active todos, I refacto that a bit so that I could introduce the same mistake.

Outcome

Vanillajs

The app output [object Object] instead of the number of completed todos:

Rank: F

React

The app output the entire list of items instead of the number of completed todos:

Rank: F

Vue

The app output the entire list of items instead of the number of completed todos:

Rank: F

Svelte

The app output [object Object] instead of the number of completed todos:

Rank: F

Elm

Changing from

entriesCompleted = List.length (List.filter .completed entries)

to

entriesCompleted = List.filter .completed entries

Generate this compiler error:

Rank: B

Conclusion

This is the summary

  Error n.   1   2   3   4   5   6   7
--------------------------------------
VanillaJS    D-  C   F   E-  D   E   F
    React    B   C   B   F   C-  E   F
      Vue    D   C+  F   F   C-  E   F
   Svelte    B+  C   B   F   B-  E-  F
      Elm    A   B+  B+  F   B+  B+  B

Let's try to quantify the result giving some number:

A: 10, B: 8, C: 6, D: 4, E: 2, F: 0

For the errors we add 1 when there is a + and subtract 1 when there is a -.

  Error n.   1   2   3   4   5   6   7
---------------------------------------------
      Elm   10   8+  8+  0   8+  8+  8 = 50++++ = 54
   Svelte    8+  6   8   0   8-  2-  0 = 32+--  = 31
    React    8   6   8   0   6-  2   0 = 30-    = 29
      Vue    4   6+  0   0   6-  2   0 = 18+-   = 18
VanillaJS    4-  6   0   2-  4   2   0 = 18--   = 16

It seems evident from the result that a compiler helps to detect these type of errors (Elm, Svelte and React).

Elm leads the rank mostly because it is strictly typed and the fact that everything in Elm is code, including HTML.

And also thanks to the high quality of its error messages.

Further developments

• Is it possible to have a more objective analysis?
• Was it fair to mix framework with compilers and framework without?
• How different would the result be with Babel and/or Typescript?

Thank you for reading and let me know what you think in the comments below.

Illustrations by Katerina Limpitsouni, from https://undraw.co/