ML Fundamentals in Javascript

I recently have been exploring the field of Machine Learning; in all honesty, I have had to relearn almost all of my math fundamentals. It has been a while since college and ML is based on a lot of Linear Algebra. In this blog, I plan to curate the fundamentals alongside my implementation of them in Javascript. I know Python's NumPy library is the industry standard, but I have always been curious about how the basic operations would translate in Javascript. Full disclosure, I am just learning this myself, so if there are any mistakes along the way, please help me out!

Topics include:

• Basics of Linear Algebra
• Matrix Operations
• Types of Matrices
• Complex Mathematical Expressions

I have also compiled a NodeJS App that contains a working version of all the examples in this blog: ML Fundamentals 1

Let's get started~!

Basics of Linear Algebra

Linear Algebra is a subset of algebra that deals with scalars, vectors, and matrices. In the simplest terms here's what they are:

Matrix

I like to think of a matrix as an array or an array of arrays in programming. Where m is the number of rows and n is the number of columns of a matrix[m][n]. If we were coding, it would look something like this:

const matrix = [
  [0, 1],
  [2, 3],
  [3, 4]
];

Vector

A vector is simply a type of matrix and specifically has only one column. Therefore, it would look something like this:

const vector = [
  [0],
  [1],
  [2],
];

Scalar

Probably, the simplest mathematical object in all of Linear Algebra is a scalar. It is just a number that's often used as a multiplier.

const scalar = 2;

Most matrices and vectors can be expressed with arrays in Javascript or any language. But what about matrices that are 3D or 4D or XD? Typically, most Linear Algebra courses state that a matrix can have x dimensions where x is a number greater than 0. This is where we begin to use the idea of tensors in programming where vectors are essentially ranked to correspond to the various dimensions. In fact, Javascript has a framework called Tensorflow.js that defines and runs computations using tensors. I will dive more into that in a future blog. For now, let's get back to the basics.

Matrix Operations

When you think about Matrix Operations, typically my mind jumps to loops. But using loops to iterate a matrix can start to become real ugly real quick. That's when I discovered the features of the Math.js library that provides JS and Node.js projects with powerful, optimized computations. This means that, instead of defining a matrix as an array of arrays, you could simply define a matrix and it would look something like this with Math.js:

const matrix = math.matrix([[0,1], [2,3], [4,5]])

Some useful methods include size() and valueOf():

math.size(matrix) //returns dimensions of matrix
//[3,2]
matrix.valueOf() //returns string representation of the matrix
//[[0,1], [2,3], [4,5]]

Let's explore examples of four main matrix operations such as addition, subtraction, multiplication, and division:

Matrix Addition

matA = math.matrix([[0, 1], [2, 3], [4, -5]]);
matB = math.matrix([[1, -1], [-2, 4], [-7, 4]]);

const matAdd = math.add(matA, matB);

console.log(matAdd.valueOf());
//[ [ 1, 0 ], [ 0, 7 ], [ -3, -1 ] ]

Matrix Subtraction

matA = math.matrix([[0, 1], [2, 3], [4, -5]]);
matB = math.matrix([[1, -1], [-2, 4], [-7, 4]]);

const matSub = math.subtract(matA, matB);

console.log(matSub.valueOf());
//[ [ -1, 2 ], [ 4, -1 ], [ 11, -9 ] ]

Matrix Multiplication

This is when things start to get interesting. Before I jump to coding examples, its important to understand these two properties of Matrix Multiplication: commutative and associative.

Commutative

Matrix multiplication is not commutative which simply means A x B != B x A. Let's test this out and check with MathJS's built in equal comparator:

matA = math.matrix([[0, 1], [2, 3]]);
matB = math.matrix([[2, 4], [6, 2]]);

const matAxB = math.multiply(matA, matB);
const matBxA = math.multiply(matB, matA);

console.log(math.equal(matAxB.valueOf(), matBxA.valueOf()));
// false

Associative

Matrix multiplication is associative which simply translates to A x (B x C) == (A x B) x C. Let's test this out as well:

const matA = math.matrix([[0, 1], [2, 3], [4, 5]]);
const matB = math.matrix([[2, 4], [6, 2]]);
const matC = math.matrix([[5, 2], [2, -2]]);

const matAxB_C = math.multiply(math.multiply(matA, matB), matC);
const matA_BxC = math.multiply(matA, math.multiply(matB, matC));

console.log(math.equal(matAxB_C.valueOf(), matA_BxC.valueOf()));
// true

Its really important to also note that in the case of math.js, the product of a matrix is not just a new matrix containing the product of the individual matrices - element-wise product (or Hardamard product). In fact, the product we see is a matrix product operation.

An example of a element-wise product is via matrix-scalar multiplication, A x 3, which is performed as such:

matA = math.matrix([[0, 1], [2, 3], [4, -5]]);
const scalar = 3;
const matAx3 = math.multiply(matA, scalar);

console.log(matAx3.valueOf());
//[ [ 0, 3 ], [ 6, 9 ], [ 12, -15 ] ]

Of course, since a vector is just a matrix it is also possible to perform a matrix-vector multiplication:

matA = math.matrix([[0, 1], [2, 3], [4, 5]]);
matB = math.matrix([[2], [1]]);

const matAxvB = math.multiply(matA, matB);

console.log(matAxvB.valueOf());
//[ [ 1 ], [ 7 ], [ 13 ] ]

Matrix Division

Matrix division is also possible to implement in Javascript. Note that in mathematically there is no "Matrix Divsion" as its defined as such:
A/B = AB^-1
However to save us from thinking about divisions and inverses we can implement matdix.divide() in js:

matA = math.matrix([[0, 2], [2, 4], [4, 6]]);
matB = math.matrix([[2, 1], [2, 2]]);

const matAB = math.divide(matA, matB);

console.log(matAB.valueOf());
// [ [ -2, 2 ], [ -2, 3 ], [ -2, 4 ] ]

After all, dealing with matrices in math.js isn't that difficult anymore! But you have to know the dimensions of each matrix in your operation, because not every matrix "operates" on another matrix!

Types of Matrices

There are a couple of matrix types in Linear Algebra that is important to understand as well.

Identity Matrix

The Identity (I) Matrix with the dimension i * j is defined as i-dimensional matrix where i == j. They are special and used widely because they are commutative; this means A x I === I x A. Here's an example of an Identity Matrix:

const matrix = [
  [1, 0, 0],
  [0, 1, 0],
  [0, 0, 1],
];

In math.js you can use the eye(i) method to quickly generate an identity matrix with dimension i:

const matI_3 = math.eye(3);
console.log(matA.valueOf());
// [ [ 1, 0, 0 ], [ 0, 1, 0 ], [ 0, 0, 1 ] ]

Transpose Matrix

In a transpose matrix the dimensions are flipped. Simply stated, the rows become columns and the columns become rows Here's an example of taking a vector and transposing it. The transposed matrix is referred to as a row vector:

const matV = math.matrix([[0], [1], [2]]);
const matV_T = math.transpose(matV);

console.log(matV_T.valueOf());
// [ [ 0, 1, 2 ] ]

Inverse Matrix

Of course, its important to discuss the inverse matrix. What's interesting about this is that matrices can have an inverse A^-1 but not all matrices (specifically singular or degenerate) have one. The formula to find the inverse of a matrix: A(A^-1) = (A^-1)A = I.
But Math.js gives us the inverse operation for free as math.inv() check it out:

matA = math.matrix([[0, 1], [2, 3]]);
const matA_1 = math.inv(matA);

console.log(matA_1.valueOf());
// [ [ -1.5, 0.5 ], [ 1, -0 ] ]

Complex Math Expressions

At some point, using built in math.js the proposed way doesn't scale anymore. Let's be honest, ML gets complicated real quick. Especially when you are trying to perform operations with multiple features and use a multivariate linear regression with gradient descent aka a function with multiple inputs. Take the following function of theta as an example:

theta = theta - ALPHA / m * ((X * theta - y)^T * X)^T

If you try to represent this out of the box in Javascript you will get a mess like this:

theta = math.subtract(
  theta,
  math.multiply(
    (ALPHA / m),
    math.transpose(
      math.multiply(
        math.transpose(
          math.subtract(
            math.multiply(
              X,
              theta
            ),
            y
          )
        ),
        X
      )
    )
  )
);

What a mess right? Luckily there is a concise and readable way to evaluate it using the eval function:

theta = math.eval(`theta - ALPHA / m * ((X * theta - y)' * X)'`, {
  theta,
  ALPHA,
  m,
  X,
  y,
});

Shocked that all of this is possible with Javascript? You are not alone! Regardless of the coding language you use today, you will surely find a powerful math library such as MathJS to enable matrix operations and other complex operations to get you started with ML fundamentals! I hope this was helpful.

If you want to experiment with the Math.js library on your own, do checkout the Github repository with all of these examples compiled in a NodeJS app:
https://github.com/mrinasugosh/ml-fundamentals-1

==== Follow me on Social Media(@mrinasugosh) ====
Dev.to: @mrinasugosh
Github: @mrinasugosh
Twitter: @mrinasugosh
LinkedIn: @mrinasugosh