What is Bottle๐ค?
๐จ๐ปโ๐ป: Bottle is a lightweight kv storage engine based on LSM-TREE.
Project Home pages: https://bottle.ibyte.me/
Project Introduction
The first thing to note is that Bottle is a KV embedded storage engine, not a KV database.
Other
I am a novice, self-taught go language programming, and then realized this project through the bitcask paper, I hope you can give me a star, thank you, my English is not very good.
The function implementation of this project is completely based on the
bitcask
paper.
Interested friends can go and see this course. If you think it is good, you can give me a small star โญ Thank you.
Install Module๐
You just need to install the Bottle module in your project to use:
go get -u github.com/auula/bottle
Bottle Source code๐ฒ
.
โโโ LICENSE
โโโ README.md
โโโ bottle-logo.svg
โโโ bottle.go # Main File
โโโ bottle_test.go
โโโ encoding.go # Data encoding module
โโโ encrypted.go # Data encrypted module
โโโ encrypted_test.go
โโโ example # Exmaple code
โย ย โโโ config.yaml
โย ย โโโ demo_exchange.go
โย ย โโโ demo_put_get.go
โโโ go.mod
โโโ go.sum
โโโ gopher-bottle.svg
โโโ hashed.go # Hashed function
โโโ item.go # Data log Entry
โโโ option.go # Bottle Configure
1 directory, 17 files
Data Directory ๐
Since the bottle design is based on a single-process program, each storage instance corresponds to a data
directory, data
is the log merge structure data directory, and index
is the index
data version.
Log consolving structural data The current version is merged every time the data is started. The default is that all data files under the DATA data folder occupies the total and more than 1GB
of more than 1GB
will trigger a merge, and the data that is not used after the merger is discarded.
Of course, if the dirty data merge requirements are not reached, the data file is archived in the size of the configured, each data has a version number, and is set to read-only mount, the process work directory structure is as follows:
./testdata
โโโ data
โ โโโ 1.data
โโโ index
โโโ 1646378326.index
โโโ 1646378328.index
2 directories, 3 files
When the storage engine starts working, the folder and files can only be operated by this process to ensure data security.
Basic API โ
Below is a usage example of a basic data manipulation I wrote:
package main
import (
"fmt"
"github.com/auula/bottle"
)
func init() {
// Open a storage instance by default configuration
err := bottle.Open(bottle.DefaultOption)
// And handle possible errors
if err != nil {
panic(err)
}
}
// Userinfo test data structure
type Userinfo struct {
Name string
Age uint8
Skill []string
}
func main() {
// PUT Data
bottle.Put([]byte("foo"), []byte("66.6"))
// If it is converted to string, it is a string
fmt.Println(bottle.Get([]byte("foo")).String())
// If there is no default, it is 0
fmt.Println(bottle.Get([]byte("foo")).Int())
// If it is not success, it is false.
fmt.Println(bottle.Get([]byte("foo")).Bool())
// If it is not success, it is 0.0
fmt.Println(bottle.Get([]byte("foo")).Float())
user := Userinfo{
Name: "Leon Ding",
Age: 22,
Skill: []string{"Java", "Go", "Rust"},
}
var u Userinfo
// Save the data object by BSON, and set the timeout for 5 seconds
// TTL timeout can not set demand
bottle.Put([]byte("user"), bottle.Bson(&user), bottle.TTL(5))
// Analysis of structures by unwrap
bottle.Get([]byte("user")).Unwrap(&u)
// Print value
fmt.Println(u)
// Remove key 'foo'
bottle.Remove([]byte("foo"))
// Close handleable error that may happen
if err := bottle.Close(); err != nil {
fmt.Println(err)
}
}
The following is a separate variable assignment and catch handling errors:
data := bottle.Get([]byte("user"))
if data.IsError() {
fmt.Println(data.Err)
} else {
fmt.Println(data.Value)
}
Encryptor๐
The data encryptor
records the value of the data, that is, the block encryption at the field level, instead of encrypting the entire file once, which will cause performance consumption, so the block data segment encryption method is adopted.
The following example uses the bottle.SetEncryptor(Encryptor,[]byte)
function to set the data encryptor and configure the 16-bit data encryption key.
func init() {
bottle.SetEncryptor(bottle.AES(), []byte("1234567890123456"))
}
You can also customize the interface to implement the data encryptor:
// SourceData for encryption and decryption
type SourceData struct {
Data []byte
Secret []byte
}
// Encryptor used for data encryption and decryption operation
type Encryptor interface {
Encode(sd *SourceData) error
Decode(sd *SourceData) error
}
The following code is the implementation code of the built-in AES
encryptor, just implement the bottle.Encryptor
interface, and the data source is the bottle.SourceData
structure field:
// AESEncryptor Implement the Encryptor interface
type AESEncryptor struct{}
// Encode source data encode
func (AESEncryptor) Encode(sd *SourceData) error {
sd.Data = aesEncrypt(sd.Data, sd.Secret)
return nil
}
// Decode source data decode
func (AESEncryptor) Decode(sd *SourceData) error {
sd.Data = aesDecrypt(sd.Data, sd.Secret)
return nil
}
The specific encryptor implementation code can be viewed in encrypted.go
Hashed ๐งฐ
If you need to customize the salad function, you can implement the bottle.Hashed interface:
type Hashed interface {
Sum64([]byte) uint64
}
Then complete your hash function configuration by built-in bottle.SetHashFunc(hash Hashed)
settings.
Index Size ๐
The index pre-set size will greatly affect your program acception and read data. If you can expect the index size required to run during initialization, and in initialization, you can decreaseThe performance issues that run data migration and expansion brought by the program during operation.
func init() {
// setting indexes size
bottle.SetIndexSize(1000)
}
Configuration Information
You can also use the default configuration, you can initialize your storage engine with the structure of the built-in bottle.Option
, and the configuration instance is as follows:
func init() {
// Custom configuration information
option := bottle.Option{
// Working directory
Directory: "./data",
// Algorithm opens encryption
Enable: true,
// Customize the secret, you can use a built-in secret
Secret: bottle.Secret,
// Custom data size, storage unit is KB
DataFileMaxSize: 1048576,
}
// Custom configuration information
bottle.Open(option)
}
Of course, you can also use the built-in bottle.Load(path string)
function to load the configuration file to start Bottle, the configuration file format is YAML
, the configurable items are as follows:
# Bottle config options
Enable: TRUE
Secret: "1234567890123456"
Directory: "./testdata"
DataFileMaxSize: 536870912
It is important to note that the key implemented by the built-in encryptor must be 16
bits, if you are a custom implementation encrypler to set your custom encryprators through bottle.SetEncryptor(Encryptor,[]byte)
, then this secretThe number of digits will not be restricted.
Vision ๐
If you have any questions about this project, you can contact me from the GitHub home page, you can also initiate a pull request
, after your code is merged, you will appear in the following list~
Top comments (0)