How I Copied About 3968 Records From a Website Using JavaScript

So I was looking for a way to pull up thousands of business categories to save me the time of having to type each of them, and I figured out that this Great website Dalton Luka had it all listed out with nearly 4000 categories.

If you tried to open the page, you would notice that there's a download button attached just above the records, but I decided to scrap out the data with JavaScript.

I then tasked myself to find an easy way to copy all those categories using JavaScript and use them in my project.

How I Started

I used the inspect element to look at how this data was organized and I saw that the Data was organized in tabular form which made the process a whole lot easier for me.

At this point, all I needed to do was to find a way to get all table cell elements with a category, loop through them, and store their inner HTML in an array.

This was the code I used.

//retrieve all table cell elements
const cats = document.querySelectorAll('tbody > tr > td');
//create array
let categories = [];
//loop through table cell elements
cats.forEach(c => {
    //store the innerHTML of each element in the array
    categories.push(c.innerHTML);
})

So when you log the categories to the console, you will see that there are about 3968 records

How then do I copy the categories?

In order to copy the categories to the clipboard, we will use navigator.clipboard.writeText(). But there's a catch.

If you try to copy the code directly from the console, you'll run into the error below

In order to maneuver this error, we need to set a timeout that will copy the categories for us to the clipboard after 5 seconds, during which we will click on anywhere on the web page for it to be focused.

Since we're copying an array of categories to the clipboard, we need to convert it to a string using JSON.stringify().

//click on anywhere on the web page
setTimeout( async() =>{
    await(navigator.clipboard.writeText(JSON.stringify(categories)));
}, 5000)

Here's the full code

//retrieve all table cell elements
const cats = document.querySelectorAll('tbody > tr > td');
//create array
let categories = [];
//loop through table cell elements
cats.forEach(c => {
    //store the innerHTML of each element in the array
    categories.push(c.innerHTML);
})

//click on anywhere on the web page
setTimeout( async() =>{
    await(navigator.clipboard.writeText(JSON.stringify(categories)));
}, 5000)

If you did not receive an error, then it must have been copied successfully.

To confirm, use Command + V. You should see the data below

What makes a good developer good, is his ability to use his knowledge to find code solutions to problems ~ Simon Ugorji.

There you have it! That was how I easily copied 3968 records from a website using JavaScript.

Image credit: unsplash.com