Chi-Sheng Liu

Posted on Sep 11 • Originally published at chishengliu.com

GraphRAG Local Setup via Ollama: Pitfalls Prevention Guide

#ai #llm #rag #graphrag

TL;DR

If you just want to see the conclusion, skip to the Summary. Please note that the content discussed here is only guaranteed to work for version 0.3.2. Other versions have not been tested.

Introduction

GraphRAG is a new Retrieval-Augmented Generation (RAG) technology released by Microsoft. Unlike the usual RAG approach, which breaks text into chunks, vectorizes them, and stores them in a vector store, and then compares the similarity by vectorizing the user's query, GraphRAG extracts information from text and builds a Knowledge Graph. This approach has the advantage of better understanding the complex relationships between different documents.

As of the time of writing, the latest official version is 0.3.2, and here is the GitHub repository link for version 0.3.2. While the content discussed here might work for other versions, it is not guaranteed.

If you've read the official tutorial, you'll notice that the config for version 0.3.2 only supports OpenAI and Azure OpenAI. You need to put your API Key in the config file to use it, which works and is the simplest approach. However, GraphRAG requires frequent LLM calls during indexing, which makes it costly, and we cannnot use our own custom fine-tuned model.

So, is it possible to use an open-source model running locally, such as Llama 3.1? The answer is yes, but you'll need to overcome several challenges, as the official implementation doesn't support local models, and you'll need to modify some things manually.

You might wonder why not use LLMGraphTransformer from LangChain instead? I can only say that, as of now, it is still an experimental feature and differs slightly from Microsoft's official implementation. If you're interested, you can check out the following links:

Prerequisites

Install Ollama and have it running.
An LLM model, such as ollama pull llama3.1:8b.
A Text Embedding model, such as ollama pull nomic-embed-text.

Troubleshooting Process

When I first opened the Get Started page, I was excited to see how short it was and thought I could get it working quickly. Looking back, I was overly optimistic...

The first few steps were straightforward and did not cause issues:

pip install graphrag
mkdir -p ./ragtest/input
curl https://www.gutenberg.org/cache/epub/24022/pg24022.txt > ./ragtest/input/book.txt
python -m graphrag.index --init --root ./ragtest

Next, I needed to edit settings.yaml, and that’s where the problems began. How should I configure the model?

First, delete .env because we don’t need the API key.

Then, open settings.yaml and input the following:

The differences compared to the original settings.yaml are:

llm
- api_key: Set it to anything. It’s not important since we won’t be using an API. I set it to NONE.
- model: Set this to your local LLM model, such as llama3.1:8b.
- max_tokens: Set this to a number smaller than the maximum token limit for your LLM model.
- api_base: Set this to http://localhost:11434/v1.
embeddings
- llm
  - api_key: Again, set it to anything since we won’t use the API. I set it to NONE.
- model: Set this to your local Text Embedding model, such as nomic-embed-text.
- api_base: Set this to http://localhost:11434/api.
- batch_max_tokens: Set this to a number smaller than the maximum token limit for your Text Embedding model.

chunks

size: Lower this to 300. If not adjusted, you’ll encounter the following error. More details in the related issue: https://github.com/microsoft/graphrag/issues/362

09:44:07,997 datashaper.workflow.workflow ERROR Error executing verb "cluster_graph" in create_base_entity_graph: Columns must be same length as key
Traceback (most recent call last):
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/datashaper/workflow/workflow.py", line 410, in _execute_verb
    result = node.verb.func(**verb_args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/verbs/graph/clustering/cluster_graph.py", line 102, in cluster_graph
    output_df[[level_to, to]] = pd.DataFrame(
    ~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/pandas/core/frame.py", line 4299, in __setitem__
    self._setitem_array(key, value)
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/pandas/core/frame.py", line 4341, in _setitem_array
    check_key_length(self.columns, key, value)
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/pandas/core/indexers/utils.py", line 390, in check_key_length
    raise ValueError("Columns must be same length as key")
ValueError: Columns must be same length as key
16:25:46,215 graphrag.index.reporting.file_workflow_callbacks INFO Error executing verb "cluster_graph" in create_base_entity_graph: Columns must be same length as key details=None
16:25:46,216 graphrag.index.run ERROR error running workflow create_base_entity_graph
Traceback (most recent call last):
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/run.py", line 325, in run_pipeline
    result = await workflow.run(context, callbacks)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/datashaper/workflow/workflow.py", line 369, in run
    timing = await self._execute_verb(node, context, callbacks)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/datashaper/workflow/workflow.py", line 410, in _execute_verb
    result = node.verb.func(**verb_args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/verbs/graph/clustering/cluster_graph.py", line 102, in cluster_graph
    output_df[[level_to, to]] = pd.DataFrame(
    ~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/pandas/core/frame.py", line 4299, in __setitem__
    self._setitem_array(key, value)
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/pandas/core/frame.py", line 4341, in _setitem_array
    check_key_length(self.columns, key, value)
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/pandas/core/indexers/utils.py", line 390, in check_key_length
    raise ValueError("Columns must be same length as key")
ValueError: Columns must be same length as key

After changing the settings, excitedly run python -m graphrag.index --init --root ./ragtest, and you'll get the following error:

09:19:54,508 datashaper.workflow.workflow ERROR Error executing verb "text_embed" in create_final_entities: 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/datashaper/workflow/workflow.py", line 415, in _execute_verb
    result = await result
             ^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/verbs/text/embed/text_embed.py", line 105, in text_embed
    return await _text_embed_in_memory(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/verbs/text/embed/text_embed.py", line 130, in _text_embed_in_memory
    result = await strategy_exec(texts, callbacks, cache, strategy_args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/verbs/text/embed/strategies/openai.py", line 62, in run
    embeddings = await _execute(llm, text_batches, ticker, semaphore)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/verbs/text/embed/strategies/openai.py", line 106, in _execute
    results = await asyncio.gather(*futures)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/verbs/text/embed/strategies/openai.py", line 100, in embed
    chunk_embeddings = await llm(chunk)
                       ^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/caching_llm.py", line 96, in __call__
    result = await self._delegate(input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 177, in __call__
    result, start = await execute_with_retry()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 159, in execute_with_retry
    async for attempt in retryer:
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 166, in __anext__
    do = await self.iter(retry_state=self._retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 165, in execute_with_retry
    return await do_attempt(), start
           ^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 147, in do_attempt
    return await self._delegate(input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/base_llm.py", line 49, in __call__
    return await self._invoke(input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/base_llm.py", line 53, in _invoke
    output = await self._execute_llm(input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/openai/openai_embeddings_llm.py", line 36, in _execute_llm
    embedding = await self.client.embeddings.create(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/resources/embeddings.py", line 237, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/_base_client.py", line 1816, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/_base_client.py", line 1510, in request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/_base_client.py", line 1613, in _request
    return await self._process_response(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/_base_client.py", line 1710, in _process_response
    return await api_response.parse()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/_response.py", line 420, in parse
    parsed = self._options.post_parser(parsed)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/resources/embeddings.py", line 225, in parser
    for embedding in obj.data:
TypeError: 'NoneType' object is not iterable
09:19:54,533 graphrag.index.reporting.file_workflow_callbacks INFO Error executing verb "text_embed" in create_final_entities: 'NoneType' object is not iterable details=None
09:19:54,541 graphrag.index.run ERROR error running workflow create_final_entities
Traceback (most recent call last):
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/run.py", line 325, in run_pipeline
    result = await workflow.run(context, callbacks)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/datashaper/workflow/workflow.py", line 369, in run
    timing = await self._execute_verb(node, context, callbacks)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/datashaper/workflow/workflow.py", line 415, in _execute_verb
    result = await result
             ^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/verbs/text/embed/text_embed.py", line 105, in text_embed
    return await _text_embed_in_memory(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/verbs/text/embed/text_embed.py", line 130, in _text_embed_in_memory
    result = await strategy_exec(texts, callbacks, cache, strategy_args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/verbs/text/embed/strategies/openai.py", line 62, in run
    embeddings = await _execute(llm, text_batches, ticker, semaphore)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/verbs/text/embed/strategies/openai.py", line 106, in _execute
    results = await asyncio.gather(*futures)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/index/verbs/text/embed/strategies/openai.py", line 100, in embed
    chunk_embeddings = await llm(chunk)
                       ^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/caching_llm.py", line 96, in __call__
    result = await self._delegate(input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 177, in __call__
    result, start = await execute_with_retry()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 159, in execute_with_retry
    async for attempt in retryer:
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 166, in __anext__
    do = await self.iter(retry_state=self._retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 165, in execute_with_retry
    return await do_attempt(), start
           ^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 147, in do_attempt
    return await self._delegate(input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/base_llm.py", line 49, in __call__
    return await self._invoke(input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/base/base_llm.py", line 53, in _invoke
    output = await self._execute_llm(input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/openai/openai_embeddings_llm.py", line 36, in _execute_llm
    embedding = await self.client.embeddings.create(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/resources/embeddings.py", line 237, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/_base_client.py", line 1816, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/_base_client.py", line 1510, in request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/_base_client.py", line 1613, in _request
    return await self._process_response(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/_base_client.py", line 1710, in _process_response
    return await api_response.parse()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/_response.py", line 420, in parse
    parsed = self._options.post_parser(parsed)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/openai/resources/embeddings.py", line 225, in parser
    for embedding in obj.data:
TypeError: 'NoneType' object is not iterable

The solution comes from this GitHub comment.

The comment suggests modifying a file’s content, but I don’t think that’s a good idea, as it might cause issues when using the package's original functionality later. So, I opted for an external monkey patch approach. First, create a file called monkey_patch.py and input the following content:

def patch_openai_embeddings_llm():
    from graphrag.llm.openai.openai_embeddings_llm import OpenAIEmbeddingsLLM
    import ollama

    async def _execute_llm(self, input, **kwargs):
        embedding_list = []
        for inp in input:
            embedding = ollama.embeddings(model=self.configuration.model, prompt=inp)
            embedding_list.append(embedding["embedding"])
        return embedding_list

    OpenAIEmbeddingsLLM._execute_llm = _execute_llm

Make sure to install ollama by running pip install ollama.

We download the graphrag.index CLI content and monkey-patch it.

First, run

wget https://raw.githubusercontent.com/microsoft/graphrag/v0.3.2/graphrag/index/__main__.py -O index.py

Then, replace line 8 of index.py:

from .cli import index_cli

with the following:

from graphrag.index.cli import index_cli
from monkey_patch import patch_openai_embeddings_llm
patch_openai_embeddings_llm()

Now, run python index.py --root ./ragtest. Finally, no more errors!

Try executing a Global Query:

python -m graphrag.query \
--root ./ragtest \
--method global \
"What are the top themes in this story?"

This error occurs:

Traceback (most recent call last):
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/openai/utils.py", line 130, in try_parse_json_object
    result = json.loads(input)
             ^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
not expected dict type. type=<class 'str'>:
Traceback (most recent call last):
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/openai/utils.py", line 130, in try_parse_json_object
    result = json.loads(input)
             ^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
not expected dict type. type=<class 'str'>:
Traceback (most recent call last):
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/openai/utils.py", line 130, in try_parse_json_object
    result = json.loads(input)
             ^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
not expected dict type. type=<class 'str'>:
Traceback (most recent call last):
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/openai/utils.py", line 130, in try_parse_json_object
    result = json.loads(input)
             ^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
not expected dict type. type=<class 'str'>:
Traceback (most recent call last):
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/openai/utils.py", line 130, in try_parse_json_object
    result = json.loads(input)
             ^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
not expected dict type. type=<class 'str'>:
Traceback (most recent call last):
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/llm/openai/utils.py", line 130, in try_parse_json_object
    result = json.loads(input)
             ^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Try executing a Local Query:

python -m graphrag.query \
--root ./ragtest \
--method local \
"Who is Scrooge, and what are his main relationships?"

This error occurs:

Error embedding chunk {'OpenAIEmbedding': "'NoneType' object is not iterable"}
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/query/__main__.py", line 92, in <module>
    run_local_search(
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/query/cli.py", line 164, in run_local_search
    response, context_data = asyncio.run(
                             ^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/query/api.py", line 222, in local_search
    result: SearchResult = await search_engine.asearch(query=query)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/query/structured_search/local_search/search.py", line 67, in asearch
    context_text, context_records = self.context_builder.build_context(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/query/structured_search/local_search/mixed_context.py", line 139, in build_context
    selected_entities = map_query_to_entities(
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/query/context_builder/entity_extraction.py", line 55, in map_query_to_entities
    search_results = text_embedding_vectorstore.similarity_search_by_text(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/vector_stores/lancedb.py", line 118, in similarity_search_by_text
    query_embedding = text_embedder(text)
                      ^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/query/context_builder/entity_extraction.py", line 57, in <lambda>
    text_embedder=lambda t: text_embedder.embed(t),
                            ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/query/llm/oai/embedding.py", line 96, in embed
    chunk_embeddings = np.average(chunk_embeddings, axis=0, weights=chunk_lens)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/numpy/lib/function_base.py", line 550, in average
    raise ZeroDivisionError(
ZeroDivisionError: Weights sum to zero, can't be normalized

So, I had to monkey-patch the query CLI as well. Related issues and comments:

Add the following two functions to monkey_patch.py:

def patch_query_embedding():
    from graphrag.query.llm.oai.embedding import OpenAIEmbedding
    import ollama
    from tenacity import (
        AsyncRetrying,
        RetryError,
        Retrying,
        retry_if_exception_type,
        stop_after_attempt,
        wait_exponential_jitter,
    )

    def _embed_with_retry(self, text, **kwargs):
        try:
            retryer = Retrying(
                stop=stop_after_attempt(self.max_retries),
                wait=wait_exponential_jitter(max=10),
                reraise=True,
                retry=retry_if_exception_type(self.retry_error_types),
            )
            for attempt in retryer:
                with attempt:
                    embedding = (ollama.embeddings(model=self.model, prompt=text)["embedding"] or [])
                    return (embedding, len(text))
        except RetryError as e:
            self._reporter.error(
                message="Error at embed_with_retry()",
                details={self.__class__.__name__: str(e)},
            )
            return ([], 0)
        else:
            # TODO: why not just throw in this case?
            return ([], 0)

    async def _aembed_with_retry(self, text, **kwargs):
        try:
            retryer = AsyncRetrying(
                stop=stop_after_attempt(self.max_retries),
                wait=wait_exponential_jitter(max=10),
                reraise=True,
                retry=retry_if_exception_type(self.retry_error_types),
            )
            async for attempt in retryer:
                with attempt:
                    embedding = (ollama.embeddings(model=self.model, prompt=text)["embedding"] or [])
                    return (embedding, len(text))
        except RetryError as e:
            self._reporter.error(
                message="Error at embed_with_retry()",
                details={self.__class__.__name__: str(e)},
            )
            return ([], 0)
        else:
            # TODO: why not just throw in this case?
            return ([], 0)

    OpenAIEmbedding._embed_with_retry = _embed_with_retry
    OpenAIEmbedding._aembed_with_retry = _aembed_with_retry

def patch_global_search():
    from graphrag.query.structured_search.global_search.search import GlobalSearch
    import logging
    import time
    from graphrag.query.llm.text_utils import num_tokens
    from graphrag.query.structured_search.base import SearchResult
    log = logging.getLogger(__name__)

    async def _map_response_single_batch(self, context_data, query, **llm_kwargs):
        """Generate answer for a single chunk of community reports."""
        start_time = time.time()
        search_prompt = ""
        try:
            search_prompt = self.map_system_prompt.format(context_data=context_data)
            search_messages = [ {"role": "user", "content": search_prompt + "\n\n### USER QUESTION ### \n\n" + query} ]
            async with self.semaphore:
                search_response = await self.llm.agenerate(
                    messages=search_messages, streaming=False, **llm_kwargs
                )
                log.info("Map response: %s", search_response)
            try:
                # parse search response json
                processed_response = self.parse_search_response(search_response)
            except ValueError:
                # Clean up and retry parse
                try:
                    # parse search response json
                    processed_response = self.parse_search_response(search_response)
                except ValueError:
                    log.warning(
                        "Warning: Error parsing search response json - skipping this batch"
                    )
                    processed_response = []

            return SearchResult(
                response=processed_response,
                context_data=context_data,
                context_text=context_data,
                completion_time=time.time() - start_time,
                llm_calls=1,
                prompt_tokens=num_tokens(search_prompt, self.token_encoder),
            )

        except Exception:
            log.exception("Exception in _map_response_single_batch")
            return SearchResult(
                response=[{"answer": "", "score": 0}],
                context_data=context_data,
                context_text=context_data,
                completion_time=time.time() - start_time,
                llm_calls=1,
                prompt_tokens=num_tokens(search_prompt, self.token_encoder),
            )

    GlobalSearch._map_response_single_batch = _map_response_single_batch

Run

wget https://raw.githubusercontent.com/microsoft/graphrag/v0.3.2/graphrag/query/__main__.py -O query.py

Replace line 9 of query.py:

from .cli import run_global_search, run_local_search

with the following:

from graphrag.query.cli import run_global_search, run_local_search
from monkey_patch import patch_query_embedding, patch_global_search
patch_query_embedding()
patch_global_search()

Then, run the Global and Local Search queries. Local Search looks good, while Global Search still needs parameter adjustments. But at least there are no more errors! Hooray!

$ python query.py --root ./ragtest --method global "What are the top themes in this story?"

INFO: Reading settings from ragtest/settings.yaml
/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/query/indexer_adapters.py:71: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  entity_df["community"] = entity_df["community"].fillna(-1)
/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/query/indexer_adapters.py:72: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  entity_df["community"] = entity_df["community"].astype(int)
creating llm client with {'api_key': 'REDACTED,len=4', 'type': "openai_chat", 'model': 'llama3.1:8b', 'max_tokens': 8191, 'temperature': 0.0, 'top_p': 1.0, 'n': 1, 'request_timeout': 180.0, 'api_base': 'http://localhost:11434/v1', 'api_version': None, 'organization': None, 'proxy': None, 'cognitive_services_endpoint': None, 'deployment_name': None, 'model_supports_json': True, 'tokens_per_minute': 0, 'requests_per_minute': 0, 'max_retries': 10, 'max_retry_wait': 10.0, 'sleep_on_rate_limit_recommendation': True, 'concurrent_requests': 25}

SUCCESS: Global Search Response:
You didn't provide a story. Please share the text, and I'll be happy to help you identify the top themes.

$ python query.py --root ./ragtest --method local "Who is Scrooge, and what are his main relationships?"

INFO: Reading settings from ragtest/settings.yaml

INFO: Vector Store Args: {}
/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/query/indexer_adapters.py:71: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  entity_df["community"] = entity_df["community"].fillna(-1)
/home/chishengliu/miniforge3/envs/graphrag/lib/python3.11/site-packages/graphrag/query/indexer_adapters.py:72: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  entity_df["community"] = entity_df["community"].astype(int)
creating llm client with {'api_key': 'REDACTED,len=4', 'type': "openai_chat", 'model': 'llama3.1:8b', 'max_tokens': 8191, 'temperature': 0.0, 'top_p': 1.0, 'n': 1, 'request_timeout': 180.0, 'api_base': 'http://localhost:11434/v1', 'api_version': None, 'organization': None, 'proxy': None, 'cognitive_services_endpoint': None, 'deployment_name': None, 'model_supports_json': True, 'tokens_per_minute': 0, 'requests_per_minute': 0, 'max_retries': 10, 'max_retry_wait': 10.0, 'sleep_on_rate_limit_recommendation': True, 'concurrent_requests': 25}
creating embedding llm client with {'api_key': 'REDACTED,len=4', 'type': "openai_embedding", 'model': 'nomic-embed-text', 'max_tokens': 4000, 'temperature': 0, 'top_p': 1, 'n': 1, 'request_timeout': 180.0, 'api_base': 'http://localhost:11434/api', 'api_version': None, 'organization': None, 'proxy': None, 'cognitive_services_endpoint': None, 'deployment_name': None, 'model_supports_json': None, 'tokens_per_minute': 0, 'requests_per_minute': 0, 'max_retries': 10, 'max_retry_wait': 10.0, 'sleep_on_rate_limit_recommendation': True, 'concurrent_requests': 25}

SUCCESS: Local Search Response:
A classic character!

Ebenezer Scrooge is the main protagonist of Charles Dickens' novella "A Christmas Carol". He is a miserly and bitter old man who lives in London during the Victorian era. Scrooge is known for his extreme love of money, his disdain for charity and kindness, and his general grumpiness.

Scrooge's main relationships are:

1. **Jacob Marley**: The ghost of his deceased business partner, who appears to Scrooge on Christmas Eve to warn him about the consequences of his selfish ways.
2. **Bob Cratchit**: His underpaid and overworked clerk, who is struggling to provide for his large family during the holiday season. Scrooge's treatment of Bob is particularly cruel, as he pays him a meager salary and expects him to work long hours without complaint.
3. **Tiny Tim**: The youngest son of Bob Cratchit, who suffers from illness and poverty. Scrooge's heartlessness towards Tiny Tim serves as a catalyst for his transformation in the story.
4. **His nephew, Fred**: A kind and generous young man who invites Scrooge to join him for Christmas dinner, but is rebuffed by his miserly uncle.
5. **The Ghosts of Christmas Past, Present, and Yet to Come**: Three supernatural visitations that haunt Scrooge on Christmas Eve, forcing him to confront the consequences of his actions and the possibility of a better future.

Through these relationships, Dickens explores themes of redemption, kindness, and the importance of human connection in "A Christmas Carol".

Summary

All the code from this article is in this Gist. Here's a quick summary of the steps:

Follow the official Get Started guide and run python -m graphrag.index --init --root ./ragtest.
Replace settings.yaml with the one in the Gist.

Run

wget https://raw.githubusercontent.com/microsoft/graphrag/v0.3.2/graphrag/index/__main__.py -O index.py

Run

wget https://raw.githubusercontent.com/microsoft/graphrag/v0.3.2/graphrag/query/__main__.py -O query.py

Download the graphrag_monkey_patch.py from the Gist.

Replace line 8 of index.py with:

from graphrag.index.cli import index_cli
from graphrag_monkey_patch import patch_all
patch_all()

Replace line 9 of query.py with:

from graphrag.query.cli import run_global_search, run_local_search
from graphrag_monkey_patch import patch_all
patch_all()

Whenever you need to use python -m graphrag.index, switch to using python index.py instead. Similarly, for python -m graphrag.query, switch to python query.py.

Top comments (1)

Dan Shalev • Dec 10

very cool, thanks for sharing! If you're exploring GraphRAG setups, you might find GraphRAG-SDK useful for managing structured queries and optimizing retrieval efficiency.

Full disclosure: I'm affiliated with FalkorDB

DEV Community

GraphRAG Local Setup via Ollama: Pitfalls Prevention Guide

TL;DR

Introduction

Prerequisites

Troubleshooting Process

Summary

Top comments (1)

Read next

VoiceScribe: Elevating Transcriptions with AssemblyAI's Universal-2 Model

Negative Eigenvalues Boost Neural Networks' Memory and Pattern Recognition Abilities

Day 40: Constrained Decoding with LLMs

From a Content Creator to an AI Tool Maker