Open Architecture Logging with SenseDeep

All enterprise logging solutions ship your log data into their proprietary log storage mechanisms. And thereafter, your log data is accessible only via their interfaces. That is great if the logging solution does 100% of what you need today and tomorrow. But if you need to get insights that their interface does not permit -- you are stuck.

SenseDeep is different. Your log data never leaves your account and is stored in an open architecture database in your account, over which you have full control.

You can read, transform and analyze your log data in any way you want. You are not limited by SenseDeep in what you can do with your serverless and log data.

SenseDeep Watcher and Log Capture

SenseDeep captures serverless and log data via a small Lambda function called the SenseDeepWatcher which receives log data and stores it in a DynamoDB database.

The Watcher and the DynamoDB table run inside your AWS account. This offers the highest level of security as your log data never leaves your account. Log performance is enhanced and latencies are greatly reduced because log data does not have far to travel.

Open Access

While the SenseDeep UI provides a fast, powerful and flexible log viewer, there are many possible needs that cannot be foreseen. So SenseDeep provides open access to your log database and publishes the log data schema so you create your own log analysis capabilities. This is an open, transparent architecture for your logging needs.

SenseDeep uses best-practices via a DynamoDB single-table design and uses the OneTable access library for easy access and manipulation of log data.

The schema for the DynamoDB SenseDeep table is defined at:

• https://github.com/sensedeep/sensedeep/blob/master/schema/index.js

Example

Here is a quick sample that demonstrates access to your log data.

import DynamoDB from 'aws-sdk/clients/dynamodb.js'
import { Table } from 'dynamodb-onetable'
import Schema from './schema.js'

const client = new DynamoDB.DocumentClient({params})

const table = new Table({
    client,
    delimiter: ':',
    hidden: false,
    name: `SenseDeep`,
    schema: Schema
})

const Log = table.getModel('Log')
const Event = table.getModel('Event')

async function main() {

    let logs = await Log.find()
    console.log('Logs', JSON.stringify(logs, null, 4))

    // Retrieve log events for /aws/lambda/HelloWorld between Jun 10 and July 10 2021

    let events = await Event.find({
        pk: '/aws/lambda/HelloWorld',
        sk: {between: [
            '2021-06-10T00:00:00.000Z',
            '2021-07-10T00:00:00.000Z',
        ]},
    }, {limit: 100})
    console.log('Events', JSON.stringify(events, null, 4))

    // Get the most recent 10 events, most recent first

    events = await Event.find({
        pk: '/aws/lambda/HelloWorld',
        sk: {between: [
            '2021',
            new Date().toISOString(),
        ]},
    }, {limit: 10, reverse: true})
    console.log('Most recent Events', JSON.stringify(events, null, 4))

    //  Get log events that match a pattern
    //  Get recent events that match the pattern "ERROR"

    events = await Event.find({
        pk: '/aws/lambda/HelloWorld',
        sk: {between: [ '2021', new Date().toISOString() ]},
    }, {
        maxPages: 100,
        limit: 100,
        where: `contains(\${message}, {"ERROR"})`,
        reverse: true,
    })

}
main()

Scalable

The SenseDeep logging architecture is 100% serverless. As your log volume increases, AWS will scale the Watcher and DynamoDB table as required to capture all your log data -- regardless of how big. As your logging load decreases, the Lambda/DynamoDB services automatically adjust and scale down as required.

This enables SenseDeep to offer pricing plans that have no log ingestion limits. You are not capped on the volume of logs or on the amount of log data captured. As your log load increases, the Watcher Lambda and SenseDeep DynamoDB table will scale predictably with a very low cost basis.

About SenseDeep

SenseDeep provides AWS developers with critical tools to create, debug, deliver and maintain serverless applications. It helps developers pin-point serverless errors via an integrated developer studio with invocation traces, metrics, logs, alarms, alerts and notifications. It watches over your services 24x7.

SenseDeep is the only open architecture solution where your serverless and log data never leaves your account and you have full control over your logged data.

Getting Started

There is nothing to install. Just navigate your browser to: https://app.sensedeep.com.

To learn more about SenseDeep please read more at: https://www.sensedeep.com/product.

Please let us know what you think, we thrive on feedback. dev@sensedeep.com.

Links

• SenseDeep Web Site
• SenseDeep App
• SenseDeep Features
• SenseDeep Documentation