Adding API Key Management for Non-LLM AI Providers in ChatCraft

Introduction

In this blog post, I'll discuss my recent contribution to ChatCraft.org, where I implemented support for non-language model AI providers, specifically focusing on Jina.ai integration for PDF to Markdown conversion. This work was done as part of PR #740.

The Challenge

ChatCraft already supported API key management for LLM providers like OpenAI, but there was a need to extend this functionality to other AI services. The primary goals were:

1. Separate LLM providers from other AI services in the UI
2. Implement API key management for Jina.ai
3. Create a foundation for supporting additional document processing providers

Implementation Process

1. UI Modifications

First, I modified the settings dialog to clearly distinguish between different types of providers:

• Changed "Providers" label to "LLM Providers"
• Added a new "Other AI Providers" section
• Implemented a consistent table layout matching the existing providers section

2. Provider Architecture

I created an abstraction for non-LLM providers:

export abstract class NonLLMProviders {
  id: string;
  name: string;
  apiUrl: string;
  apiKey?: string;
}

3. Jina.ai Integration

Implemented the JinaAIProvider class:

export class JinaAIProvider extends NonLLMProviders {
  constructor(apiKey?: string) {
    super(JINA_AI, JINA_API_URL, apiKey);
  }

  static fromSettings(): JinaAIProvider {
    const settings = getSettings();
    const provider = settings.nonLLMProviders["Jina AI"];

    // case 1 when the provider already exists in the settings
    if (provider instanceof JinaAIProvider) {
      return provider;
    }
    // case 2 when the provider doesn't exist but the api key is set
    if (provider?.apiKey) {
      return new JinaAIProvider(provider.apiKey);
    }

    // case 3 when the provider doesn't exist and the api key is not set
    return new JinaAIProvider();
  }

Key Learnings

1. Provider Abstraction: Initially, I created separate implementations for each provider type. However, after code review feedback, I realized there was a significant overlap with ChatCraftProvider. This led to discussions about future refactoring to create a more generic provider system.

2. Error Handling: Working with Jina.ai's API taught me the importance of graceful degradation. When users hit free-tier limits, we needed to provide clear guidance about API key usage.

3. Document Processing Power: During testing, I discovered Jina.ai's impressive capabilities - it successfully converted an 85-page PowerPoint PDF to markdown with high accuracy, opening possibilities for future document processing pipelines.

Challenges and Solutions

1. UI Consistency: Initially, the new provider section had different sizing from the existing table. This was standardized after code review feedback.

2. Code Organization: The original implementation had some redundancy in provider management. This was resolved by better encapsulation and static utility functions.

Future Improvements

1. Better error handling for free-tier limitations
2. Refactoring provider classes for better code reuse
3. Support for additional document processing services

Conclusion

This repository was too good to work on, in fact some of the knowledge that I have gained from here would be used in my own project DialectMorph, because both of the projects are utilising different AI clients but are encapsulating logic in a way which makes it easier to be integrated into any app