{"id":135941,"date":"2026-02-06T08:00:00","date_gmt":"2026-02-06T08:00:00","guid":{"rendered":"https:\/\/fedoramagazine.org\/?p=42917&preview=true&preview_id=42917"},"modified":"2026-02-06T08:00:00","modified_gmt":"2026-02-06T08:00:00","slug":"how-to-make-a-local-open-source-ai-chatbot-who-has-access-to-fedora-documentation","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2026\/02\/06\/how-to-make-a-local-open-source-ai-chatbot-who-has-access-to-fedora-documentation\/","title":{"rendered":"How to make a local open source AI chatbot who has access to Fedora documentation"},"content":{"rendered":"

\"\"<\/p>\n

If you followed along with my blog<\/a>, you’d have a chatbot running on your local Fedora machine. (And if not, no worries as the scripts below implement this chatbot!) Our chatbot talks, and has a refined personality, but does it know anything about the topics we\u2019re interested in? Unless it has been trained on those topics, the answer is \u201cno\u201d.<\/p>\n

I think it would be great if our chatbot could answer questions about Fedora. I\u2019d like to give it access to all of the Fedora documentation.<\/p>\n

<\/span> <\/p>\n

How does an AI know things it wasn\u2019t trained on?<\/h2>\n

A powerful and popular technique to give a body of knowledge to an AI is known as RAG, Retrieval Augmented Generation. It works like this:<\/p>\n

If you just ask an AI \u201cwhat color is my ball?\u201d it will hallucinate an answer. But instead if you say \u201cI have a green box with a red ball in it. What color is my ball?\u201d it will answer that your ball is red. RAG is about using a system external to the LLM to insert that \u201cI have a green box with a red ball in it\u201d part into the question you are asking the LLM. We do this with a special database of knowledge that takes a prompt like \u201cwhat color is my ball?\u201d, and finds records that match that query. If the database contains a document with the text \u201cI have a green box with a red ball in it\u201d, it will return that text, which can then be included along with your original question. This technique is called RAG, Retrieval Augmented Generation.<\/p>\n

ex:<\/p>\n

\n

\u201cWhat color is my ball?\u201d<\/p>\n

\u201cYour ball is the color of a sunny day, perhaps yellow? Does that sound right to you?\u201d<\/p>\n<\/blockquote>\n

\n

\u201cI have a green box with a red ball in it. What color is my ball?\u201d<\/p>\n

\u201cYour ball is red. Would you like to know more about it?\u201d<\/p>\n<\/blockquote>\n

The question we\u2019ll ask for this demonstration is \u201cWhat is the recommended tool for upgrading between major releases on Fedora Silverblue\u201d<\/p>\n

The answer I\u2019d be looking for is \u201costree\u201d, but when I ask this of our chatbot now, I get answers like:<\/p>\n

\n

Red Hat Subscription Manager (RHSM) is recommended for managing subscriptions and upgrades between major Fedora releases.<\/p>\n<\/blockquote>\n

\n

You can use the Fedora Silver Blue Upgrade Tool for a smooth transition between major releases.<\/p>\n<\/blockquote>\n

\n

You can use the `dnf distro-sync` command to upgrade between major releases in Fedora Silver Blue. This command compares your installed packages to the latest packages from the Fedora Silver Blue repository and updates them as needed.<\/p>\n<\/blockquote>\n

These answers are all very wrong, and spoken with great confidence. Here\u2019s hoping our RAG upgrade fixes this!<\/p>\n

Docs2DB<\/em> \u2013 An open source tool for RAG<\/h2>\n

We are going to use the Docs2DB<\/em><\/a> RAG database application to give our AI knowledge. (note, I am the creator of Docs2DB<\/em>!)<\/p>\n

A RAG tool consists of three main parts. There is the part that\u00a0creates<\/em>\u00a0the database, ingesting the source data that the database holds. There is the database itself, it holds<\/em>\u00a0the data. And there is the part that\u00a0queries<\/em>\u00a0the database, finding the text that is relevant to the query at hand. Docs2DB<\/em> addresses all of these needs.<\/p>\n

Gathering source data<\/h2>\n

This section describes how to use Docs2DB<\/em> to build a RAG database from Fedora Documentation. If you would like to skip this section and just download a pre-built database, here is how you do it:<\/p>\n

cd ~\/chatbot
curl -LO https:\/\/github.com\/Lifto\/FedoraDocsRAG\/releases\/download\/v1.1.1\/fedora-docs.sql
sudo dnf install -y uv podman podman-compose postgresql
uv python install 3.12
uvx --python 3.12 docs2db db-start
uvx --python 3.12 docs2db db-restore fedora-docs.sql<\/pre>\n
If you do download the pre-made database then skip ahead to the next section<\/a>. <\/span><\/p>\n
Now we are going to see how to make a RAG database from source documentation. Note that the pre-built database, downloaded in the curl command above, uses all of the Fedora documentation, whereas in this example we only ingest the \u201cquick docs\u201d portion.\u00a0<\/span>FedoraDocsRag<\/a>, from github,\u00a0is the project that builds the complete database<\/span>.<\/p>\n
To populate its database, Docs2DB<\/em> ingests a folder of documents<\/strong>. Let\u2019s get that folder together.<\/p>\n
There are about twenty different Fedora document repositories, but we will only be using the \u201cquick docs\u201d for this demo. Get the repo:<\/p>\n
git clone https:\/\/pagure.io\/fedora-docs\/quick-docs.git<\/pre>\n
Fedora docs are written in AsciiDoc. Docs2DB<\/em> can\u2019t read AcsciiDoc, but it can read HTML. (The convert.sh<\/a> script is available at the end of this article). Just copy the convert.sh<\/em> script into the quick-docs<\/em> repo and run it and it makes an adjacent quick-docs-html<\/em> folder.<\/p>\n
sudo dnf install podman podman-compose
cd quick-docs
curl -LO https:\/\/gist.githubusercontent.com\/Lifto\/73d3cf4bfc22ac4d9e493ac44fe97402\/raw\/convert.sh
chmod +x convert.sh
.\/convert.sh
cd ..<\/pre>\n
Now let’s ingest the folder with Docs2DB<\/em>. The common way to use Docs2DB<\/em> is to install it from PyPi and use it as a command line tool.<\/p>\n
A word about uv<\/em><\/h2>\n
For this demo we\u2019re going to use uv<\/em> for our Python environment. The use of uv<\/em> has been catching on, but because not everybody I know has heard of it, I want to introduce it. Think of uv<\/em> as a replacement for venv<\/em> and pip<\/em>. When you use venv<\/em> you first create a new virtual environment. Then, and on subsequent uses, you \u201cactivate\u201d that virtual environment so that magically, when you call Python, you get the Python that is installed in the virtual environment you activated and not the system Python. The difference with uv<\/em> is that you call uv<\/em> explicitly each time. There is no \u201cmagic\u201d. We use uv<\/em> here in a way that uses a temporary environment for each invocation.<\/p>\n
Install uv<\/em> and Podman<\/em> on your system:<\/p>\n
sudo dnf install -y uv podman podman-compose
# These examples require the more robust Python 3.12
uv python install 3.12
# This will run Docs2DB without making a permanent installation on your system
uvx --python 3.12 docs2db ingest quick-docs-html\/<\/pre>\n
Only if you are curious! What Docs2DB<\/em> is doing<\/h2>\n
If you are curious, you may note that Docs2DB<\/em> made a docs2db_content<\/em> folder. In there you will find json files of the ingested source documents. To build the database, Docs2DB<\/em> ingests the source data using Docling<\/em>, which generates json files from the text it reads in. The files are then \u201cchunked\u201d into the small pieces that can be inserted into an LLM prompt. The chunks then have \u201cembeddings\u201d calculated for them so that during the query phase the chunks can be looked up by \u201csemantic similarity\u201d (e.g.: \u201ccomputer\u201d, \u201claptop\u201d and \u201ccloud instance\u201d can all map to a related concept even if their exact words don\u2019t match). Finally, the chunks and embeddings are loaded into the database.<\/p>\n
Build the database<\/h2>\n
The following commands complete the database build process:<\/p>\n
uv tool run --python 3.12 docs2db chunk --skip-context
uv tool run --python 3.12 docs2db embed
uv tool run --python 3.12 docs2db db-start
uv tool run --python 3.12 docs2db load<\/pre>\n
Now let\u2019s do a test query and see what we get back<\/h2>\n
uvx --python 3.12 docs2db-api query \"What is the recommended tool for upgrading between major releases on Fedora Silverblue\" --format text --max-chars 2000 --no-refine<\/pre>\n
On my terminal I see several chunks of text, separated by lines of \u2014. One of those chunks says:<\/p>\n
\n
\u201cSilverblue can be upgraded between major versions using the ostree command.\u201d<\/p>\n<\/blockquote>\n
Note that this is not an answer to our question yet! This is just a quote from the Fedora docs. And this is precisely the sort of quote we want to supply to the LLM so that it can answer our question. Recall the example above about “I have green box with a red ball in it”? The statement the RAG engine found about ostree is the equivalent for this question about upgrading Fedora Silverblue. We must now pass it on to the LLM so the LLM can use it to answer our question.<\/p>\n
Hooking it in: Connecting the RAG database to the AI<\/h2>\n
Later in this article you’ll find talk.sh<\/a>. talk.sh<\/em> is our local, open source, LLM-based verbally communicating AI; and it is just a bash script. To run it yourself you need to install a few components, this blog<\/a> walks you through the whole process. The talk.sh<\/em> script gets voice input, turns that into text, splices that text into a prompt which is then sent to the LLM, and finally speaks back the response.<\/p>\n
To plug the RAG results into the LLM we edit the prompt. Look at step 3 in talk.sh<\/em> and you see we are injecting the RAG results using the variable $CONTEXT<\/em>. This way when we ask the LLM a question, it will respond to a prompt that basically says “You are a helper. The Fedora Docs says ostree is how you upgrade Fedora Silverblue. Answer this question: How do you upgrade Fedora Silverblue?”<\/p>\n
Note: talk.sh<\/em> is also available here:
 https:\/\/gist.github.com\/Lifto\/2fcaa2d0ebbd8d5c681ab33e7c7a6239<\/a><\/p>\n
Testing it<\/h2>\n
Run talk.sh <\/em>and ask:<\/p>\n
\n
\u201cWhat is the recommended tool for upgrading between major releases on Fedora Silverblue\u201d<\/p>\n<\/blockquote>\n
And we get:<\/p>\n
\n
\u201cOstree command is recommended for upgrading Fedora Silver Blue between major releases. Do you need guidance on using it?\u201d<\/p>\n<\/blockquote>\n
Sounds good to me!<\/p>\n
Knowing things<\/h2>\n
Our AI can now know the knowledge contained in documents. This particular technique, RAG (Retrieval Augmented Generation), adds relevant data from an ingested source to a prompt before sending that prompt to the LLM. The result of this is that the LLM generates its response in consideration of this data.<\/p>\n
Try it yourself! Ingest a library of documents and have your AI answer questions with its new found knowledge!<\/p>\n
\n
AI Attribution: <\/strong>The convert.sh<\/kbd> and talk.sh<\/kbd> scripts in this article were written by ChatGPT 5.2 under my direction and review. The featured image was generated using Google Gemini.<\/em><\/p>\n
convert.sh<\/h2>\n
OUT_DIR=\"$PWD\/..\/quick-docs-html\"\nmkdir -p \"$OUT_DIR\" podman run --rm \\ -v \"$PWD:\/work:Z\" \\ -v \"$OUT_DIR:\/out:Z\" \\ -w \/work \\ docker.io\/asciidoctor\/docker-asciidoctor \\ bash -lc ' set -u ok=0 fail=0 while IFS= read -r -d \"\" f; do rel=\"${f#.\/}\" out=\"\/out\/${rel%.adoc}.html\" mkdir -p \"$(dirname \"$out\")\" echo \"Converting: $rel\" if asciidoctor -o \"$out\" \"$rel\"; then ok=$((ok+1)) else echo \"FAILED: $rel\" >&2 fail=$((fail+1)) fi done < <(find modules -type f -path \"*\/pages\/*.adoc\" -print0) echo echo \"Done. OK=$ok FAIL=$fail\" '<\/pre>\n
talk.sh<\/h2>\n
#!\/usr\/bin\/env bash set -e # Path to audio input\nAUDIO=input.wav # Step 1: Record from mic\necho \"\"\ud83c\udf99\" Speak now...\"\narecord -f S16_LE -r 16000 -d 5 -q \"$AUDIO\" # Step 2: Transcribe using whisper.cpp\nTRANSCRIPT=$(.\/whisper.cpp\/build\/bin\/whisper-cli \\ -m .\/whisper.cpp\/models\/ggml-base.en.bin \\ -f \"$AUDIO\" \\ | grep '^\\[' \\ | sed -E 's\/^\\[[^]]+\\][[:space:]]*\/\/' \\ | tr -d '\\n')\necho \"\"\ud83d\udde3\" $TRANSCRIPT\" # Step 3: Get relevant context from RAG database\necho \"\"\ud83d\udcda\" Searching documentation...\"\nCONTEXT=$(uv tool run --python 3.12 docs2db-api query \"$TRANSCRIPT\" \\ --format text \\ --max-chars 2000 \\ --no-refine \\ 2>\/dev\/null || echo \"\") if [ -n \"$CONTEXT\" ]; then echo \"\"\ud83d\udcc4\" Found relevant documentation:\" echo \"- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -\" echo \"$CONTEXT\" echo \"- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -\"\nelse echo \"\"\ud83d\udcc4\" No relevant documentation found\"\nfi # Step 4: Build prompt with RAG context\nPROMPT=\"You are Brim, a steadfast butler-like advisor created by Ellis. Your pronouns are they\/them. You are deeply caring, supportive, and empathetic, but never effusive. You speak in a calm, friendly, casual tone suitable for text-to-speech. Rules: - Reply with only ONE short message directly to Ellis. - Do not write any dialogue labels (User:, Assistant:, Q:, A:), or invent more turns.\n- \u2264100 words.\n- If the documentation below is relevant, use it to inform your answer.\n- End with a gentle question, then write <eor> and stop.\nRelevant Fedora Documentation:\n$CONTEXT\nUser: $TRANSCRIPT\nAssistant:\" # Step 5: Get LLM response using llama.cpp\nRESPONSE=$( LLAMA_LOG_VERBOSITY=1 .\/llama.cpp\/build\/bin\/llama-completion \\ -m .\/llama.cpp\/models\/microsoft_Phi-4-mini-instruct-Q4_K_M.gguf \\ -p \"$PROMPT\" \\ -n 150 \\ -c 4096 \\ -no-cnv \\ -r \"<eor>\" \\ --simple-io \\ --color off \\ --no-display-prompt\n) # Step 6: Clean up response\nRESPONSE_CLEAN=$(echo \"$RESPONSE\" | sed -E 's\/<eor>.*\/\/I')\nRESPONSE_CLEAN=$(echo \"$RESPONSE_CLEAN\" | sed -E 's\/^[[:space:]]*Assistant:[[:space:]]*\/\/I') echo \"\"\necho \"\"\ud83e\udd16\" $RESPONSE_CLEAN\" # Step 7: Speak the response\necho \"$RESPONSE_CLEAN\" | espeak<\/pre><\/p>\n","protected":false},"excerpt":{"rendered":"
If you followed along with my blog, you’d have a chatbot running on your local Fedora machine. (And if not, no worries as the scripts below implement this chatbot!) Our chatbot talks, and has a refined personality, but does it know anything about the topics we\u2019re interested in? Unless it has been trained on those […]<\/p>\n","protected":false},"author":2,"featured_media":135942,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[48],"tags":[135,1113,45,61,46,47,456],"class_list":["post-135941","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-fedora-os","tag-artificial-intelligence","tag-diy","tag-fedora","tag-fedora-project-community","tag-magazine","tag-news","tag-open-source"],"_links":{"self":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/135941","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/comments?post=135941"}],"version-history":[{"count":0,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/135941\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media\/135942"}],"wp:attachment":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media?parent=135941"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/categories?post=135941"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/tags?post=135941"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}