DEV Community

Cover image for File Database in Node Js from scratch part 3: introducing types
Sk
Sk

Posted on • Edited on

File Database in Node Js from scratch part 3: introducing types

Introduction

At the end of this tut or article the aim is simple, having types for each column and a type checking system on insert, for now we will support three types and add more as they are needed.

In this tut, I've experimented a bit with a new type of writing as I am trying to improve my technical writing skills, I hope there is a notable difference, your feedback will be highly appreciated if you have any.

Add types in utils.js

export const types = {
     "String": 1,
     "Number": 2,
     "Object": 3,

}

// you will see how this object is useful in a sec


Enter fullscreen mode Exit fullscreen mode

database.js

now for types to work we have to enforce them early on, meaning a types option when creating a new db is not optional anymore, if it is not present we throw an error



let store = new db("test db", {timeStamp: true,
                   // mandatory
                   types: {name:"String", surname:"String", age:"Number"}}

                  )



Enter fullscreen mode Exit fullscreen mode

I could have used an array to manage types: ["String", "String", "Number"] which would be simpler: an index corresponds with a column, the problem is a document/object {name: "sk", surname: "mhlungu", age: 23} cannot be really trusted to maintain the order of columns, as objects are not by index but keys(for a large enough object the values MAY(i am not sure either and don't want to find out) change positions even if we use object.keys.

so that is why I am mapping column names to their types, which consequently adds a new feature: you cannot add a document with a column that does not exist

e.g {name: "sk", surname: "mhlungu", age: 23} correct
{name: "sk", surname: "mhlungu", age: 23, stack: "React, Ionic"} wrong: must throw an error

let's update database.js to handle this

 ... // three dots represent previous/existing code
 import {types} from "./utils.js"



function db(name,options) {
     // if options does not have the types property
    if(!options.types){

        throw new Error("db needs a types option to work")
    }

   // checking if the types are supported by the database
  const n = Object.keys(options.types).map((val, i)=> {

      return types[options.types[val]]

  })



    ...
}




Enter fullscreen mode Exit fullscreen mode

type support check breakdown


 const n = Object.keys(options.types).map((val, i)=> {

      return types[options.types[val]]

  })

 // this code is simple really 
 // for  {name:"String", surname:"String", age:"Number"}
 // the above loops over the object
 // the magic happens in the return, remember our types util:
  export const types = {
     "String": 1,
     "Number": 2,
     "Object": 3,

}

 // if the type in options.types is present in utils types
 // a number corresponding to that type is returned else undefined is returned
 // for {name:"String", surname:"String", age:"Number"} 
  // [1, 1, 2] will be returned 
  // for {name:"String", surname:"String", age:"Number", stack: "Array"}
  // [1, 1, 2, undefined] will be returned 
  // all we need to do now is check for undefined in the array if we find one
  // then we throw an error of unsupported type else we continue and create the db





Enter fullscreen mode Exit fullscreen mode

checking for undefined in database.js


function db(name,options) {


    ...

  if(n.indexOf(undefined) !== -1){ // if we have undefined 

     const m = Object.keys(options.types)[n.indexOf(undefined)]
     // show which column and type is unsupported
     throw new Error(`type of ${options.types[m]} for column ${m} does not exist`)

  }

    // if the above two if's are cleared then we can create the db 


       this.store = new Store(name, options)
}


Enter fullscreen mode Exit fullscreen mode

Goal one complete we have successfully introduced types, now we need to make sure on insert every document follows the same rules, insert a required type for a column, a column of type string cannot hold a number, that's an error

Store.js - enforcing types on insert

in store's setData we want to end up with something of sort


   set setData(data){
          // new code
       // check if the document has required columns
         if(!checkColumns(data, this.#meta.options.types)){

               throw new Error(`db expected a document with these columns: ${Object.keys(this.#meta.options.types)},

                                          but got ${Object.keys(data)} for this document ${JSON.stringify(data)}`)

         }

         // check if the document has correct types
         if(!checkTypes(data, this.#meta.options.types)){

         throw new Error(`db expected a document with these types: ${Object.values(this.#meta.options.types)},

                                          but got ${Object.values(data)} for this document ${JSON.stringify(data)}`)

         }


       // new code ends
      data._id = this.#meta.length

      if(this.#meta.options && this.#meta.options.timeStamp && this.#meta.options.timeStamp){

           data.timeStamp = Date.now()



      }

      this.#data[this.#meta.length] = data

      this.#meta.length++

      // console.log('data', this.#data)

   }




Enter fullscreen mode Exit fullscreen mode

before we write checkColumns and types we need a few utils

in utils.js add :


// return booleans        
// () =>  👈 functions of sort are called immediate return functions
// they have no {}, they return their values after runnig
export const isStr = (val) => typeof val === "string"

export const isNumb = (val) => typeof val === "number"

export const isObj = (val) => typeof val === "object"

Enter fullscreen mode Exit fullscreen mode

back to Store.js

CheckColumns function

place these func's on top of the class



function checkColumns(doc, types){

  let checkOut = true  // state -> the most important value here 
                       // if true everything is correct else not

   // yes you can definetley use forEach below instead of map(will change it too) 
  // react.js habits cause me to use map everywhere 😂😂 i just noticed writing the article 
   Object.keys(types).map((key, i)=> {

      if(!checkOut) return checkOut;

      if(doc[key] === undefined){

        console.log(key, "is missing in this document")

        checkOut = false

      }

   })

   if(Object.keys(types).length !== Object.keys(doc).length) checkOut = false

   return checkOut




}


Enter fullscreen mode Exit fullscreen mode

explanation:



 Object.keys(types).map((key, i)=> {

      if(!checkOut) return checkOut;  // break out of map early if we already have a
                                      // a column problem

      if(doc[key] === undefined){ // if the document has a missing column

        console.log(key, "is missing in this document")

        checkOut = false

      }

   })


Enter fullscreen mode Exit fullscreen mode

to notice in the above is that the code will pass even if we have an extra column that does not exist in types Object.keys(types) as we checking columns in types against doc

example:

{name:"String", surname:"String", age:"Number"}
{name: "sk", surname: "mhlungu", age: 23, stack: "React"}

// stack is extra
// the above map will pass cause doc has all types column, the extra will be ignored 
// which is wrong, hence the below code to handle this and make sure 
// columns are of the same size and we have no extra column


Enter fullscreen mode Exit fullscreen mode

checking for extra columns


 if(Object.keys(types).length !== Object.keys(doc).length) checkOut = false

Enter fullscreen mode Exit fullscreen mode

if we found an extra column we return false then insert won't run but throw an error


 if(!checkColumns(data, this.#meta.options.types)){

               throw new Error(`db expected a document with these columns: ${Object.keys(this.#meta.options.types)},

                                          but got ${Object.keys(data)} for this document ${JSON.stringify(data)}`)

         }


Enter fullscreen mode Exit fullscreen mode

if the column check passes then we can check for types

CheckTypes function

import {isStr, isNumb, isObj} from "./utils.js" // typecheck helpers 



// basically this function is the same as columns check 


function checkTypes(doc, types){

    let checkOut = true  // state


     // map again 🤦‍♂️, change to forEach please
    Object.keys(doc).map((key,i)=> { // looping over the doc keys {name: "sk", surname: "mhlungu", age: 23}

        if(!checkOut) return checkOut; // early break

       if(types[key] === "String"){ // if the column in question expects a string

           if(!isStr(doc[key])) checkOut = false // and the value in doc is not a string throw an error(checkout = false)

       }else if(types[key] === "Number"){

          if(!isNumb(doc[key])) checkOut = false

       }else if(types[key] === "Object"){

          if(!isObj(doc[key])) checkOut = false

       }

    })




    return checkOut

}




Enter fullscreen mode Exit fullscreen mode

same thing happens here also if the check types fail insert breaks without inserting, I am one to admit for now error handling is horrible , we cannot just break(which is an assumption the developer is using try catch, which is very rare), I am thinking of a dedicated article to handle errors better maybe returning an object with status, and what happened etc

this will checktypes before running insert code


 if(!checkTypes(data, this.#meta.options.types)){

         throw new Error(`db expected a document with these types: ${Object.values(this.#meta.options.types)},

                                          but got ${Object.values(data)} for this document ${JSON.stringify(data)}`)

         }


Enter fullscreen mode Exit fullscreen mode

what I am noticing so far in these three articles is the vortex abstract API thing we been following is kinda working, look we added a bunch of code, done a lot of refactoring without touching the end point and changing much of previous code, that is indeed a victory 🍾👌🎉, our end point is still clean in index.js no plumbing yet:


import db from "./database.js"





export default db


Enter fullscreen mode Exit fullscreen mode

no shade to plumbers by the way, plumbing or plumber is a football(soccer) slang from my country, meaning a coach who looks promising but is doing absolute rubbish in tactics and formation while having a quality team, which is losing by the way, by plumbing code I mean something similar.

we have basically achieved both goals we set out in the beginning, but remember the main goal was to assist the where function from the previous article with transforming age > 23 string commands to proper values without trying much

let's do that now,

Select.js

Remember our vortex analogy, code that does not concern itself with certain data or state or does not need or require directly must ask the responsible end point for it, so here Select will need types so select must ask Store for them meaning we need a function to return types from the store.

in store.js


// add under set setData inside the class
  get getTypes(){

         return this.#meta.options.types

      }



Enter fullscreen mode Exit fullscreen mode

our proto to get types




Store.prototype.types = function(){

    return this.getTypes



}

Enter fullscreen mode Exit fullscreen mode

back to select, because types will be used by the entire channel(possibly in the future), we can add them in the tracker for each channel, this will make it so on channel destruction the types are destroyed also(saving memory)

update beginQuery with code followed by new code comment


 this.beginQuery = (channelName = "") => {

                   // prepare

                   console.log("creating channel", channelName)

                     if(tracker[this.id] && tracker[this.id].beganQ){

                                    console.warn('please close the previous query');

                                    return

                      }

                     // keys = this.store.allKeys()

              this.id = tracker.id

               tracker[this.id] = {

              filtered: [],

              types: {}, // new code

              beganQ: false,

              cName : channelName === "" ? this.id : channelName

             }

            tracker.id++

                    tracker[this.id].filtered = Object.values(store.getAll())

                    tracker[this.id].types = store.types() // new code

                    tracker[this.id].beganQ = true

                    console.log('opening channel: ', tracker[this.id].cName)

                    // console.log('tracker obj', tracker)

                   };




Enter fullscreen mode Exit fullscreen mode

update where also to pass the types to search, we can pass the id but it's not that necessary if we can pass the types directly




//update where 
     // now search takes three arguments
     // command, data and types
    let f = search(str, tracker[this.id].filtered, tracker[this.id].types)

Enter fullscreen mode Exit fullscreen mode

next we need to update search, for now all we need to know in search is does the command have a number and convert that str number to an actual number, solving our previous problems 23 === "23" // returning false


const search = function(comm, data, types){

  let split = comm.split(" ")

    
     // new
     if(types[split[0]] === 'Number'){

        split[2] = +split[2]  // converting "23" to 23 (number)

     }
      // Problems split[0] is a column
      // if the column does not exist cause in where you can pass 
      // a column that does not exist e.g "aged > 23" 
      // we need to handle that gracefully
      // for now it will fail silently because of the way `where` works
      // and return unfiltered data

    ...



}

Enter fullscreen mode Exit fullscreen mode

that is it for this article you can experiment with test.js, with that we have types finally, and things are getting exciting honestly, I am thinking on moving to dumping the data to a file next. to fulfill the file part in file database, we will deal with other CRUD functions later

conclusion

If you want a programming buddy I will be happy to connect on twitter , or you or you know someone who is hiring for a front-end(react or ionic) developer or just a JS developer(modules, scripting etc) I am looking for a job or gig please contact me: mhlungusk@gmail.com, twitter will also do

Thank you for your time, enjoy your day or night. until next time

Top comments (0)