What is Data?

There are three categories of data: structured, semi-structured, and unstructured. Each type of data has its own characteristics and use cases, understanding the differences between them is crucial for effective data management and analysis.

Structured data is organized and easily searchable. It is typically stored in relational databases, and its format is well-defined with pre-determined columns, data types, and relationships.

Examples include data from enterprise resource planning (ERP) systems, customer relationship management (CRM) databases, and financial records. Structured data can be easily queried and analyzed using SQL and other database tools. An example of Structured Data:

import sqlite3

# Connect to the database
conn = sqlite3.connect('example.db')

# Create a table with structured data
conn.execute('''CREATE TABLE employees
             (id INT PRIMARY KEY NOT NULL,
             name TEXT NOT NULL,
             age INT NOT NULL);''')

# Insert data into the table
conn.execute("INSERT INTO employees (id, name, age) VALUES (1, 'John Doe', 25)")
conn.execute("INSERT INTO employees (id, name, age) VALUES (2, 'Jane Smith', 30)")

# Query the data from the table
cursor = conn.execute("SELECT * FROM employees")
for row in cursor:
    print("ID = ", row[0])
    print("Name = ", row[1])
    print("Age = ", row[2])

# Close the database connection
conn.close()

A structured database table with pre-defined columns for id, name, and age. We insert data into the table and query it using SQL.

Semi-structured data falls somewhere between structured and unstructured data. It has a defined structure, but it's not as rigid as structured data.

Semi-structured data often includes metadata and tags that provide additional context. Examples include XML and JSON files, which are commonly used to exchange data between web applications.

import json

# Define a JSON object with semi-structured data
employee = {
  "id": 1,
  "name": "John Doe",
  "age": 25,
  "department": {
    "name": "Engineering",
    "manager": "Jane Smith"
  }
}

# Convert the JSON object to a string
employee_json = json.dumps(employee)

# Print the JSON string
print(employee_json)

# Convert the JSON string back to a Python object
employee_dict = json.loads(employee_json)

# Access the data in the Python object
print(employee_dict['id'])
print(employee_dict['name'])
print(employee_dict['age'])
print(employee_dict['department']['name'])
print(employee_dict['department']['manager'])

we define a JSON object with semi-structured data that includes a nested department object. We convert the object to a JSON string and back to a Python object, accessing the data using dictionary keys.

Unstructured data lacks any predefined structure. It's the most challenging type of data to work with because it includes text, images, and multimedia files that don't fit neatly into a database schema.

Examples include emails, social media posts, images, and videos. Unstructured data can be challenging to analyze using traditional data analysis tools, but advancements in natural language processing NLP and machine learning algorithms are making it easier to derive insights from unstructured data.

import pytesseract
from PIL import Image

# Open an image file with unstructured data
image = Image.open('example.png')

# Use Tesseract OCR to extract text from the image
text = pytesseract.image_to_string(image)

# Print the extracted text
print(text)

we open an image file with unstructured data and use Tesseract OCR to extract text from the image. The extracted text doesn't have a pre-defined structure and is challenging to analyze without advanced NLP techniques.

Each datatype have their unique characteristics and uses. Programmers and organizations can better manage and analyze their data by understanding the usecases, costs and benefits of each datatype, leading to more informed decision-making and better business outcomes.

A great resource to explore for learning about working with within the Azure ecosystem is the Microsoft Certificate: Azure Data fundamentals. It looks like this course is even freely available for students.

Something I happened across today to add context to how the industry is thinking about data and datatypes in the ongoing conversation around AI