p2p Databases — Hyperbee — workshop

🎓 Workshop: What you will learn

* Some background on peer to peer and decentralized web

* How to get started using hyperspace cli

* How to create a p2p database using the hyp cli

* How to perform basic database functions using p2p db

* How to connect to a p2p database from a remote system

* How to query a remote p2p database and show results

Hyperbee is a p2p database built using b-tree structures and distributed using append-only log structures called hypercore. The folks at hyper protocol have created some tooling to work with p2p databases called hyperspace. This workshop will learn how to create a p2p database, interact with a p2p database, and connect to the database from a remote server.

Prerequisites

What do I need to know for this workshop?

• Javascript — https://developer.mozilla.org/en-US/docs/Web/Javascript
• NodeJS v14+ — https://nodejs.org
• Some understanding of NodeJS Streams — https://nodejs.dev/learn/nodejs-streams
• ExpressJS Basics — https://expressjs.com/

What is p2p?

p2p or peer to peer is a technology where there is no dependency on centralized servers to share information. Some call p2p the decentralized web, but I think the decentralized web may be more broad. p2p enables computers to talk without having to depend on a centralized system coordinating the discussion. Some popular peer to peer technologies are bittorrent and webrtc. BitTorrent lets you download large files in pieces from different servers, which results in a much faster download time. WebRTC enables browsers to share video, audio, and data between themselves without having to go through a central service.

What is a p2p database?

A p2p database is a database that is accessible via a p2p network so that applications can connect to the database and access the data. You may be a bit confused, why should I be interested in a p2p database? With a traditional database, you have a gateway and that gateway controls the access of creating the data and querying the data. So for everyone to use that database to create content they must connect to the database and store their information, to read the content they must connect to the database and query information. This works well in a lot of cases.

Hypothetical use case

A p2p social app, where every user created their own local database and provided the identifier of that database to a feed aggregator application, that would consume all of those user databases and merge them into one large database showing everyone’s posts. The benefit of such a system is that the creator of the content would still have ownership of their content, and the feed aggregator only has control of the whole feed, but could never delete the content of a client.

Getting started

Let’s use the terminal to install the hyperspace cli.

npm install --global @hyperspace/cli

Let’s use the terminal to create a new folder and initialize a new NodeJS application:

mkdir p2pDb
cd p2pDb
npm init -y

Install some dependencies from npm.

npm install express hyperspace hyperbee

Copy the server.js file example server code:

p2p Database workshop initial server code

p2p Database workshop initial server code. GitHub Gist: instantly share code, notes, and snippets.

262588213843476Gist

Creating a hyperbee database using hyp

To create a hyperbee database we need to start the hyperspace daemon then call the create command with the cli. This will give us a hyper:// URL, this URL will be our access identifier to our hyperbee database.

hyp daemon start
hyp create bee

Setting up the DB in our server.js

There are two node modules we need to connect to our hyperbee database, hyperspace and hyperbee, let's create a new file in our project called db.js in this file, we will keep all of our DB commands.

db.js

const { Client } = require('hyperspace')
const Hyperbee = require('hyperbee')
const cuid = require('cuid')

const client = new Client()
const store = client.corestore()
const core = store.get('ee26ef1982c9f3bb3ce49adc46cbc55467ecb88779a629234af702da9965758e')
const db = new Hyperbee(core, {
  keyEncoding: 'utf-8',
  valueEncoding: 'json'
})

// add movie to db
exports.add = async function (movie) {
  return db.put(cuid(), movie)
}

// list movies
exports.list = async function () {
  var movies = []
  await new Promise(r => {
    db.createReadStream()
      .on('data', entry => movies.push(entry.value))
      .on('end', r)
  })
  return movies
}

In this db.js file, we are creating two functions, add and list the add function will add data to the hyperbee database and the list function will stream all of the documents from the hyperbee database. To connect to the database, we need to instantiate a new client to hyperspace and then connect to the hyperspace store, then get a hypercore from the store, using that core, we will instantiate a hyperbee database.

Adding a movie from our web app

To add a movie from our web app, we need to bring our db.js module into the server js module:

const { add, list } = require('./db') // at the top of serverjs

In our app.post('/') handler, we can convert to an async function and call add(req.body).

app.post('/', express.json(), async (req, res) => {
  await add(req.body)
  res.setHeader('content-type', 'text/html')
  res.send(`<li>${req.body.title}</li>`)
})

list movies from hypebee

On our get /movies endpoint, let's call the list method from the DB module and map over the results to create some

app.get('/movies', async (req, res) => {
  const s = await list()
  res.setHeader('content-type', 'text/html')
  res.send(s.map(m => `<li>${m.title}</li>`).join(''))
})

app.post('/', express.json(), async (req, res) => {
  await add(req.body)
  const s = await list()
  res.setHeader('content-type', 'text/html')
  res.send(s.map(m => `<li>${m.title}</li>`).join(''))
})

npm install --global @hyperspace/cli

mkdir p2pClient
cd p2pClient

npm init -y
npm install express hyperspace hyperbee

npx nodemon server2.js

db.ready()
  .then(() => client.replicate(core))