The Structured project started as a Go conversion of Instructor,
but it is a more general-purpose library. It is designed to be extremely easy to use and set up.
It also features a language-agnostic HTTP server that you can set up in front of llama.cpp.
Same features, Go-like API. Model agnostic - maps data from arbitrary JSON schema to arbitrary Go struct (or just plain JSON).
It is focused on llama.cpp. Support for other vendor APIs (like OpenAI or Anthropic) might be added in the future.
HTTP API
Start a server and point it to your local llama.cpp instance:
./structured \
--llamacpp-host 127.0.0.1 \
--llamacpp-port 8081 \
--port 8080
Structured server connects to llama.cpp to extract the data.
Now, you can issue requests. Include schema and data in your POST body.
The server will respond with JSON matching your schema:
POST http://127.0.0.1:8080/extract/entity
{
"schema": {
"type": "object",
"properties": {
"hello": {
"type": "string"
}
},
"required": ["hello"]
},
"data": "Say 'world'"
}
Response:
{
"hello": "world"
}
Programmatic Usage (Optional)
API can change with time until all features are implemented.
Initializing the Mapper
Point it to your local llama.cpp instance:
import (
"fmt"
"net/http"
"testing"
"github.com/distantmagic/structured/structured"
"github.com/distantmagic/paddler/llamacpp"
"github.com/distantmagic/paddler/netcfg"
)
var entityExtractor *EntityExtractor = &structured.EntityExtractor{
LlamaCppClient: &llamacpp.LlamaCppClient{
HttpClient: http.DefaultClient,
LlamaCppConfiguration: &llamacpp.LlamaCppConfiguration{
HttpAddress: &netcfg.HttpAddressConfiguration{
Host: "127.0.0.1",
Port: 8081,
Scheme: "http",
},
},
},
MaxRetries: 3,
}
Extracting Structured Data from String
import "github.com/distantmagic/structured/structured"
responseChannel := make(chan structured.EntityExtractorResult)
go entityExtractor.ExtractFromString(
responseChannel,
map[string]any{
"type": "object",
"properties": map[string]any{
"name": map[string]string{
"type": "string",
},
"surname": map[string]string{
"type": "string",
},
"age": map[string]string{
"description": "Age in years.",
"type": "integer",
},
},
},
"I am John Doe - living for 40 years and I still like to play chess.",
)
for result := range responseChannel {
if result.Error != nil {
panic(result.Error)
}
// map[name:John, surname:Doe, age:40]
fmt.Print(result.Result)
}
Mapping Extracted Result onto an Arbitrary Struct
Once you obtain the result:
import "github.com/distantmagic/structured/structured"
type myTestPerson struct {
Name string `json:"name"`
Surname string `json:"surname"`
Age int `json:"age"`
}
func DoUnmarshalsToStruct(result structured.EntityExtractorResult) {
var person myTestPerson
err := structured.UnmarshalToStruct(result, &person)
if nil != err {
panic(err)
}
person.Name // John
person.Surname // Doe
}
Summary
That's it! :) You can use it as a language-agnostic server. Visit the repository and leave a star to show your support and get notified about new developments.
Top comments (0)