Build your own documentation chatbot with ChatGPT and EdgeDB​

How it works​

tl;dr- Training a language model is hard, but using embeddings to give it access to information beyond what it’s trained on is easy… so we will do that! Now, skip ahead to get started building or read on for more detail.

Our chatbot is backed by OpenAI’s ChatGPT. ChatGPT is an advanced large language model (LLM) that uses machine learning algorithms to generate human-like responses based on the input it’s given.

There are two options when integrating ChatGPT and language models in general: fine-tuning the model or using embeddings. Fine-tuning produces the best result, but it needs more of everything: more money, more time, more resources, and more training data. That’s why many people and businesses use embeddings instead to provide additional context to an existing language model.

Embeddings are a way to convert words, phrases, or other types of data into a numerical form that a computer can do math with. All of this is built on top of the foundation of natural language processing (NLP) which allows computers to fake an understanding of human language. In the context of NLP, word embeddings are used to transform words into vectors. These vectors define a word’s position in space where the computer sorts them based on their syntactic and semantic similarity. For instance, synonyms are closer to each other, and words that often appear in similar contexts are grouped together.

When using embeddings we are not training the language model. Instead we’re creating embedding vectors for every piece of documentation which will later help us find which piece of documentation likely answers a user’s question. When a user asks a question, we create a new embedding for that question and compare it against the embeddings generated from our documentation to find the most similar embeddings. The answer is generated using the content that corresponds to these similar embeddings.

With that out of the way, let’s walk through how the pieces fit together.

Prerequisites​

This tutorial assumes you have Node.js installed. If you don’t, please install it before continuing.

The build requires other software too, but we’ll help you install it as part of the tutorial.

Initial setup​

Let’s start by scaffolding our app with the Next.js create-next-app tool. Run this wherever you would like to create the new directory for this project.

$ 

npx create-next-app --typescript docs-chatbot

Need to install the following packages:
  create-next-app@13.4.12
Ok to proceed? (y) y
✔ Would you like to use ESLint? … No / Yes
✔ Would you like to use Tailwind CSS? … No / Yes
✔ Would you like to use `src/` directory? … No / Yes
✔ Would you like to use App Router? (recommended) … No / Yes
✔ Would you like to customize the default import alias? … No / Yes
Creating a new Next.js app in /<path>/<to>/<project>/docs-chatbot.

Choose “Yes” for all questions except “Would you like to use `src/` directory?” and “Would you like to customize the default import alias?”

Once bootstrapping is complete, you should see a success message:

Success! Created docs-chatbot at
/<path>/<to>/<project>/docs-chatbot

Change into the new directory so we can get started!

$ 

cd docs-chatbot

Let’s make two changes to the tsconfig.json generated by create-next-app. Change the target to "es6" because we will use some data structures that are only available in ES6. Update the compilerOptions object by setting the baseUrl property to the root with "baseUrl": ".". Later when we add modules to the root of the project, this will make it easier to import them.

{
  "compilerOptions": {
    "target": "es5",
    "target": "es6",
    "lib": ["dom", "dom.iterable", "esnext"],
    "allowJs": true,
    "skipLibCheck": true,
    "strict": true,
    "forceConsistentCasingInFileNames": true,
    "noEmit": true,
    "esModuleInterop": true,
    "module": "esnext",
    "moduleResolution": "bundler",
    "resolveJsonModule": true,
    "isolatedModules": true,
    "jsx": "preserve",
    "incremental": true,
    "plugins": [
      {
        "name": "next"
      }
    ],
    "paths": {
      "@/*": ["./*"]
    }
    },
    "baseUrl": "."
  },
  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
  "exclude": ["node_modules"]
}

Now, we’ll create an instance of EdgeDB for our project, but first, we need to install EdgeDB!

$ 

curl --proto '=https' --tlsv1.2 -sSf https://sh.edgedb.com | sh

PS> 

iwr https://ps1.edgedb.com -useb | iex

$ 

edgedb project init

No `edgedb.toml` found in `/<path>/<to>/<project>/docs-chatbot`
or above

Do you want to initialize a new project? [Y/n]
> Y

Specify the name of EdgeDB instance to use with this project
[default: docs_chatbot]:
> docs_chatbot

Checking EdgeDB versions...
Specify the version of EdgeDB to use with this project
[default: 3.2]:
> 3.2

$ 

touch .env.local

$ 

edgedb instance credentials --insecure-dsn

EDGEDB_DSN=<your-dsn>
EDGEDB_CLIENT_TLS_SECURITY="insecure"

EDGEDB_DSN=<your-dsn>
EDGEDB_CLIENT_TLS_SECURITY="insecure"
OPENAI_API_KEY="<your-openai-api-key>"

import OpenAI from "openai";

export function initOpenAIClient() {
  if (!process.env.OPENAI_API_KEY)
    throw new Error("Missing environment variable OPENAI_API_KEY");

  return new OpenAI({
    apiKey: process.env.OPENAI_API_KEY,
  });
}

export const errors = {
  flagged: `OpenAI has declined to answer your question due to their
          [usage-policies](https://openai.com/policies/usage-policies).
          Please try another question.`,
  default: "There was an error processing your request. Please try again.",
};

Put the documentation in place​

For this project, we will be using documentation written as Markdown files since they are straightforward for OpenAI’s language models to use.

Create a docs folder in the root of the project. Here we will place our Markdown documentation files. You can grab the files we use from the example project’s GitHub repo or add your own. (If you use your own, you may also want to adjust the system message we send to OpenAI later.)

Create the schema to store embeddings​

To be able to store data in the database, we have to create its schema first. We want to make the schema as simple as possible and store only the relevant data. We need to store the section’s embeddings, content, and the number of tokens. The embeddings allow us to match content to questions. The content gives us context to feed to the LLM. We will need the token count later when calculating how many related sections fit inside the prompt context while staying under the model’s token limit.

Open the empty schema file that was generated when we initialized the EdgeDB project (located at dbschema/default.esdl from the project directory). We’ll walk through what we’ll add to it, one step at a time. First, add this at the top of the file (above module default {):

using extension pgvector;
module default {
  # Schema will go here
}

We are able to store embeddings and find similar embeddings in the EdgeDB database because of the pgvector extension. In order to use it in our schema, we have to activate the ext::pgvector module with using extension pgvector at the beginning of the schema file. This module gives us access to the ext::pgvector::vector data type as well as few similarity functions and indexes we can use later to retrieve embeddings. Read our pgvector documentation for more details on the extension.

Just below that, we can start building our module by creating a new scalar type.

using extension pgvector;
module default {
  scalar type OpenAIEmbedding extending
    ext::pgvector::vector<1536>;

  type Section {
    # We will build this out next
  }
}

With the extension active, we may now add properties to our object types using the included ext::pgvector::vector data type. However, in order to be able to use indexes, the vectors in question need to be a of a fixed length. This can be achieved by creating a custom scalar extending the vector and specifying the desired length. OpenAI embeddings have length of 1,536, so that’s what we use in our schema for this custom scalar.

Now, the Section type:

using extension pgvector;
module default {
  scalar type OpenAIEmbedding extending
    ext::pgvector::vector<1536>;

  type Section {
    required content: str;
    required tokens: int16;
    required embedding: OpenAIEmbedding;

    index ext::pgvector::ivfflat_cosine(lists := 1)
      on (.embedding);
  }
}

The Section contains properties to store the content, a count of tokens, and the embedding, which is of the custom scalar type we created in the previous step.

We’ve also added an index inside the Section type to speed up queries. In order for this to work properly, the index should correspond to the cosine_similarity function we’re going to use to find sections related to the user’s question. That corresponding index is ivfflat_cosine.

We are using the value 1 for the lists parameter because we will have very few items in our database — three, to be exact 😅. Best practice is to use the number of objects divided by 1,000 for up to 1,000,000 objects.

In our case indexing does not have much impact, but if you plan to store and query a large number of entries, you’ll see performance gains by adding this index.

Put that all together, and your entire schema file should look like this:

using extension pgvector;

module default {
  scalar type OpenAIEmbedding extending
    ext::pgvector::vector<1536>;

  type Section {
    required content: str;
    required tokens: int16;
    required embedding: OpenAIEmbedding;

    index ext::pgvector::ivfflat_cosine(lists := 1)
      on (.embedding);
  }
}

We apply this schema by creating and running a migration.

$ 

edgedb migration create

$ 

edgedb migrate

type Section {
  required path: str {
    constraint exclusive;
  }
  required checksum: str;
  # The rest of the Section type
}

Create and store embeddings​

Before we can script the creation of embeddings, we need to install some libraries that will help us.

$ 

npm install openai edgedb

$ 
  
  
  
  
  

npm install \
  @edgedb/generate \
  gpt-tokenizer \
  dotenv \
  tsx \
  --save-dev

The @edgedb/generate package provides a set of code generation tools that are useful when developing an EdgeDB-backed applications with TypeScript/JavaScript. We’re going to write queries using our query builder, but before we can, we need to run the query builder generator.

$ 

npx @edgedb/generate edgeql-js

Answer “y” when asked about adding the query builder to .gitignore.

This generator gives us a code-first way to write fully-typed EdgeQL queries with TypeScript. After running the generator, you should see a new edgeql-js folder inside dbschema.

Finally, we’re ready to create embeddings for all sections and store them in the database we created earlier. Let’s make a generate-embeddings.ts file inside the project root.

$ 

touch generate-embeddings.ts

Let’s look at the script’s skeleton and get an understanding of the flow of tasks we need to perform.

import { promises as fs } from "fs";
import { join } from "path";
import dotenv from "dotenv";
import { encode } from "gpt-tokenizer";
import * as edgedb from "edgedb";
import e from "dbschema/edgeql-js";
import { initOpenAIClient } from "./utils";

dotenv.config({ path: ".env.local" });

const openai = initOpenAIClient();

interface Section {
  id?: string;
  tokens: number;
  content: string;
  embedding: number[];
}

async function walk(dir: string): Promise<string[]> {
  // …
}

async function prepareSectionsData(
  sectionPaths: string[]
): Promise<Section[]> {
  // …
}


async function storeEmbeddings() {
  // …
}

(async function main() {
  await storeEmbeddings();
})();

At the top are all imports we will need throughout the file. The second to last import is the query builder we generated earlier, and the last one is the function that initializes our OpenAI API client.

After the imports, we use the dotenv library to import environment variables from the .env.local file.

Then, we initialize our OpenAI API client by calling initOpenAIClient.

Next, we define a Section TypeScript interface that corresponds to the Section type we have defined in the schema.

Then we have a few function definitions:

• walk and prepareSectionsData will be called from inside storeEmbeddings. walk returns an array of all documentation page paths relative to the project root. prepareSectionsData takes care of preparing the Section objects we will insert into the database and returns those as an array.

• storeEmbeddings coordinates everything.

To finish the script, we await a call to our coordinating function which kicks off everything else as needed.

// …
async function walk(dir: string): Promise<string[]> {
  const entries = await fs.readdir(dir, { withFileTypes: true });

  return (
    await Promise.all(
      entries.map((entry) => {
        const path = join(dir, entry.name);
        if (entry.isFile()) return [path];
        else if (entry.isDirectory()) return walk(path);
        return [];
      })
    )
  ).flat();
}
// …

[
  'docs/edgeql/design-goals.md',
  'docs/edgeql/overview.md',
  'docs/edgeql/try-edgeql.md',
]

// …
async function prepareSectionsData(
  sectionPaths: string[]
): Promise<Section[]> {
  const contents: string[] = [];
  const sections: Section[] = [];

  for (const path of sectionPaths) {
    const content = await fs.readFile(path, "utf8");
    // OpenAI recommends replacing newlines with spaces for best results
    // when generating embeddings
    const contentTrimmed = content.replace(/\n/g, " ");
    contents.push(contentTrimmed);
    sections.push({
      content,
      tokens: encode(content).length,
      embedding: [],
    });
  }
  // The rest of the function
}
// …

// …
async function prepareSectionsData(
  sectionPaths: string[]
): Promise<Section[]> {
  // Part we just talked about

  const embeddingResponse = await openai.embeddings.create({
    model: "text-embedding-ada-002",
    input: contents,
  });

  // The rest
}
// …

// …
async function prepareSectionsData(
  sectionPaths: string[]
): Promise<Section[]> {
  // The stuff we already talked about

  embeddingResponse.data.forEach((item, i) => {
    sections[i].embedding = item.embedding;
  });

  return sections;
}
// …

// …
async function prepareSectionsData(
  sectionPaths: string[]
): Promise<Section[]> {
  const contents: string[] = [];
  const sections: Section[] = [];

  for (const path of sectionPaths) {
    const content = await fs.readFile(path, "utf8");
    // OpenAI recommends replacing newlines with spaces for best results
    // when generating embeddings
    const contentTrimmed = content.replace(/\n/g, " ");
    contents.push(contentTrimmed);
    sections.push({
      content,
      tokens: encode(content).length,
      embedding: [],
    });
  }

  const embeddingResponse = await openai.embeddings.create({
    model: "text-embedding-ada-002",
    input: contents,
  });

  embeddingResponse.data.forEach((item, i) => {
    sections[i].embedding = item.embedding;
  });

  return sections;
}
// …

// …
async function storeEmbeddings() {
  const client = edgedb.createClient();

  const sectionPaths = await walk("docs");

  console.log(`Discovered ${sectionPaths.length} sections`);

  const sections = await prepareSectionsData(sectionPaths);

  // The rest of the function
}
// …

// …
async function storeEmbeddings() {
  // The parts we just talked about

  // Delete old data from the DB.
  await e.delete(e.Section).run(client);

  // Bulk-insert all data into EdgeDB database.
  const query = e.params({ sections: e.json }, ({ sections }) => {
    return e.for(e.json_array_unpack(sections), (section) => {
      return e.insert(e.Section, {
        content: e.cast(e.str, section.content!),
        tokens: e.cast(e.int16, section.tokens!),
        embedding: e.cast(e.OpenAIEmbedding, section.embedding!),
      });
    });
  });

  await query.run(client, { sections });
  console.log("Embedding generation complete");
}
// …

// …
async function storeEmbeddings() {
  const client = edgedb.createClient();

  const sectionPaths = await walk("docs");

  console.log(`Discovered ${sectionPaths.length} sections`);

  const sections = await prepareSectionsData(sectionPaths);

  // Delete old data from the DB.
  await e.delete(e.Section).run(client);

  // Bulk-insert all data into EdgeDB database.
  const query = e.params({ sections: e.json }, ({ sections }) => {
    return e.for(e.json_array_unpack(sections), (section) => {
      return e.insert(e.Section, {
        content: e.cast(e.str, section.content!),
        tokens: e.cast(e.int16, section.tokens!),
        embedding: e.cast(e.OpenAIEmbedding, section.embedding!),
      });
    });
  });

  await query.run(client, { sections });
  console.log("Embedding generation complete");
}
// …

import { promises as fs } from "fs";
import { join } from "path";
import dotenv from "dotenv";
import { encode } from "gpt-tokenizer";
import * as edgedb from "edgedb";
import e from "dbschema/edgeql-js";
import { initOpenAIClient } from "@/utils";

dotenv.config({ path: ".env.local" });

const openai = initOpenAIClient();

interface Section {
  id?: string;
  tokens: number;
  content: string;
  embedding: number[];
}

async function walk(dir: string): Promise<string[]> {
  const entries = await fs.readdir(dir, { withFileTypes: true });

  return (
    await Promise.all(
      entries.map((entry) => {
        const path = join(dir, entry.name);
        if (entry.isFile()) return [path];
        else if (entry.isDirectory()) return walk(path);
        return [];
      })
    )
  ).flat();
}

async function prepareSectionsData(
  sectionPaths: string[]
): Promise<Section[]> {
  const contents: string[] = [];
  const sections: Section[] = [];

  for (const path of sectionPaths) {
    const content = await fs.readFile(path, "utf8");
    // OpenAI recommends replacing newlines with spaces for best results
    // when generating embeddings
    const contentTrimmed = content.replace(/\n/g, " ");
    contents.push(contentTrimmed);
    sections.push({
      content,
      tokens: encode(content).length,
      embedding: [],
    });
  }

  const embeddingResponse = await openai.embeddings.create({
    model: "text-embedding-ada-002",
    input: contents,
  });

  embeddingResponse.data.forEach((item, i) => {
    sections[i].embedding = item.embedding;
  });

  return sections;
}

async function storeEmbeddings() {
  const client = edgedb.createClient();

  const sectionPaths = await walk("docs");

  console.log(`Discovered ${sectionPaths.length} sections`);

  const sections = await prepareSectionsData(sectionPaths);

  // Delete old data from the DB.
  await e.delete(e.Section).run(client);

  // Bulk-insert all data into EdgeDB database.
  const query = e.params({ sections: e.json }, ({ sections }) => {
    return e.for(e.json_array_unpack(sections), (section) => {
      return e.insert(e.Section, {
        content: e.cast(e.str, section.content!),
        tokens: e.cast(e.int16, section.tokens!),
        embedding: e.cast(e.OpenAIEmbedding, section.embedding!),
      });
    });
  });

  await query.run(client, { sections });
  console.log("Embedding generation complete");
}

(async function main() {
  await storeEmbeddings();
})();

{
  "name": "docs-chatbot",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
    "lint": "next lint",
    "embeddings": "tsx generate-embeddings.ts"
  },
  "dependencies": {
    "edgedb": "^1.3.5",
    "next": "^13.4.19",
    "openai": "^4.0.1",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "typescript": "5.1.6"
  },
  "devDependencies": {
    "@edgedb/generate": "^0.3.3",
    "@types/node": "20.4.8",
    "@types/react": "18.2.18",
    "@types/react-dom": "18.2.7",
    "autoprefixer": "10.4.14",
    "dotenv": "^16.3.1",
    "eslint": "8.46.0",
    "eslint-config-next": "13.4.13",
    "gpt-tokenizer": "^2.1.1",
    "postcss": "8.4.27",
    "tailwindcss": "3.3.3",
    "tsx": "^3.12.7"
  }
}

$ 

npm run embeddings

$ 

edgedb ui

Answering user questions​

Now that we have the content’s embeddings stored, we can start working on the handler for user questions. The user will submit a question to our server, and the handler will send them an answer back. We will define a route and an HTTP request handler for this task. Thanks to the power of Next.js, we can do all of this within our project using a route handler.

As we write our handler, one important consideration is that answers can be quite long. We could wait on the server side to get the whole answer from OpenAI and then send it to the client, but that would feel slow to the user. OpenAI supports streaming, so instead we can send the answer to the client in chunks, as they arrive to the server. With this approach, the user doesn’t have to wait for the entire response before they start getting feedback and our API seems faster.

In order to stream responses, we will use the browser’s server-sent events (SSE) API. Server-sent events enable a client to receive automatic updates from a server via an HTTP connection, and describes how the server maintains data transmissions to a client once an initial client connection has been established. The client sends a request and with that request initiates a connection with the server. The server then sends data back to the client in chunks until all of the data is sent, at which point it closes the connection.

$ 

mkdir app/api && cd app/api

$ 

mkdir generate-answer && touch generate-answer/route.ts

$ 

npm install common-tags

$ 

npm install @types/common-tags --save-dev

import { stripIndents, oneLineTrim } from "common-tags";
import * as edgedb from "edgedb";
import e from "dbschema/edgeql-js";
import { errors } from "../../constants";
import { initOpenAIClient } from "@/utils";

export const runtime = "edge";

const openai = initOpenAIClient();

const client = edgedb.createHttpClient();

export async function POST(req: Request) {
    // …
}

// other functions that are called inside POST handler

// …

export async function POST(req: Request) {
  try {
    const { query } = await req.json();
    const sanitizedQuery = query.trim();

    const flagged = await isQueryFlagged(query);

    if (flagged) throw new Error(errors.flagged);

    const embedding = await getEmbedding(query);

    const context = await getContext(embedding);

    const prompt = createFullPrompt(sanitizedQuery, context);

    const answer = await getOpenAiAnswer(prompt);

    return new Response(answer.body, {
      headers: {
        "Content-Type": "text/event-stream",
      },
    });
  } catch (error: any) {
    console.error(error);

    const uiError = error.message || errors.default;

    return new Response(uiError, {
      status: 500,
      headers: { "Content-Type": "application/json" },
    });
  }
}

async function isQueryFlagged(query: string) {
  const moderation = await openai.moderations.create({
    input: query,
  });

  const [{ flagged }] = moderation.results;

  return flagged;
}

async function getEmbedding(query: string) {
  const embeddingResponse = await openai.embeddings.create({
    model: "text-embedding-ada-002",
    input: query.replaceAll("\n", " "),
  });

  const [{ embedding }] = embeddingResponse.data;

  return embedding;
}

const getSectionsQuery = e.params(
    {
        target: e.OpenAIEmbedding,
        matchThreshold: e.float64,
        matchCount: e.int16,
        minContentLength: e.int16,
    },
    (params) => {
        return e.select(e.Section, (section) => {
        const dist = e.ext.pgvector.cosine_distance(
            section.embedding,
            params.target
        );
        return {
            content: true,
            tokens: true,
            dist,
            filter: e.op(
                e.op(
                  e.len(section.content),
                  ">",
                  params.minContentLength
                ),
                "and",
                e.op(dist, "<", params.matchThreshold)
            ),
            order_by: {
                expression: dist,
                empty: e.EMPTY_LAST,
            },
            limit: params.matchCount,
        };
        });
    }
);

async function getContext(embedding: number[]) {
    const sections = await getSectionsQuery.run(client, {
        target: embedding,
        matchThreshold: 0.3,
        matchCount: 8,
        minContentLength: 20,
    });

    let tokenCount = 0;
    let context = "";

    for (let i = 0; i < sections.length; i++) {
        const section = sections[i];
        const content = section.content;
        tokenCount += section.tokens;

        if (tokenCount >= 1500) {
            tokenCount -= section.tokens;
            break;
        }

        context += `${content.trim()}\n---\n`;
    }

    return context;
}

function createFullPrompt(query: string, context: string) {
    const systemMessage = `
        As an enthusiastic EdgeDB expert keen to assist,
        respond to questions referencing the given EdgeDB
        sections.

        If unable to help based on documentation, respond
        with: "Sorry, I don't know how to help with that."`;

    return stripIndents`
        ${oneLineTrim`${systemMessage}`}

        EdgeDB sections: """
        ${context}
        """

        Question: """
        ${query}
        """`;
}

async function getOpenAiAnswer(prompt: string) {
  const completion = await openai.chat.completions
    .create({
      model: "gpt-3.5-turbo",
      messages: [{ role: "user", content: prompt }],
      max_tokens: 1024,
      temperature: 0.1,
      stream: true,
    })
    .asResponse();

  return completion;
}

import { stripIndents, oneLineTrim } from "common-tags";
import * as edgedb from "edgedb";
import e from "dbschema/edgeql-js";
import { errors } from "../../constants";
import { initOpenAIClient } from "@/utils";

export const runtime = "edge";

const openai = initOpenAIClient();

const client = edgedb.createHttpClient();

export async function POST(req: Request) {
  try {
    const { query } = await req.json();
    const sanitizedQuery = query.trim();

    const flagged = await isQueryFlagged(query);

    if (flagged) throw new Error(errors.flagged);

    const embedding = await getEmbedding(query);

    const context = await getContext(embedding);

    const prompt = createFullPrompt(sanitizedQuery, context);

    const answer = await getOpenAiAnswer(prompt);

    return new Response(answer.body, {
      headers: {
        "Content-Type": "text/event-stream",
      },
    });
  } catch (error: any) {
    console.error(error);

    const uiError = error.message || errors.default;

    return new Response(uiError, {
      status: 500,
      headers: { "Content-Type": "application/json" },
    });
  }
}

async function isQueryFlagged(query: string) {
  const moderation = await openai.moderations.create({
    input: query,
  });

  const [{ flagged }] = moderation.results;

  return flagged;
}

async function getEmbedding(query: string) {
  const embeddingResponse = await openai.embeddings.create({
    model: "text-embedding-ada-002",
    input: query.replaceAll("\n", " "),
  });

  const [{ embedding }] = embeddingResponse.data;

  return embedding;
}

const getSectionsQuery = e.params(
  {
    target: e.OpenAIEmbedding,
    matchThreshold: e.float64,
    matchCount: e.int16,
    minContentLength: e.int16,
  },
  (params) => {
    return e.select(e.Section, (section) => {
      const dist = e.ext.pgvector.cosine_distance(
        section.embedding,
        params.target
      );
      return {
        content: true,
        tokens: true,
        dist,
        filter: e.op(
          e.op(
            e.len(section.content),
            ">",
            params.minContentLength
          ),
          "and",
          e.op(dist, "<", params.matchThreshold)
        ),
        order_by: {
          expression: dist,
          empty: e.EMPTY_LAST,
        },
        limit: params.matchCount,
      };
    });
  }
);

async function getContext(embedding: number[]) {
  const sections = await getSectionsQuery.run(client, {
    target: embedding,
    matchThreshold: 0.3,
    matchCount: 8,
    minContentLength: 20,
  });

  let tokenCount = 0;
  let context = "";

  for (let i = 0; i < sections.length; i++) {
    const section = sections[i];
    const content = section.content;
    tokenCount += section.tokens;

    if (tokenCount >= 1500) {
      tokenCount -= section.tokens;
      break;
    }

    context += `${content.trim()}\n---\n`;
  }

  return context;
}

function createFullPrompt(query: string, context: string) {
  const systemMessage = `
        As an enthusiastic EdgeDB expert keen to assist,
        respond to questions referencing the given EdgeDB
        sections.

        If unable to help based on documentation, respond
        with: "Sorry, I don't know how to help with that."`;

  return stripIndents`
        ${oneLineTrim`${systemMessage}`}

        EdgeDB sections: """
        ${context}
        """

        Question: """
        ${query}
        """`;
}

async function getOpenAiAnswer(prompt: string) {
  const completion = await openai.chat.completions
    .create({
      model: "gpt-3.5-turbo",
      messages: [{ role: "user", content: prompt }],
      max_tokens: 1024,
      temperature: 0.1,
      stream: true,
    })
    .asResponse();

  return completion;
}

Building the UI​

To make things as simple as possible, we will just update the Home component that’s inside app/page.tsx file. By default all components inside the App Router are server components, but we want to have client-side interactivity and dynamic updates. In order to do that we have to use a client component for our Home component. The way to accomplish that is to convert the page.tsx file to use the client component. We do that by adding the use client directive to the top of the file.

"use client";

Now we build a simple UI for the chatbot.

import { useState } from "react";
import { errors } from "./constants";

export default function Home() {
    const [prompt, setPrompt] = useState("");
    const [question, setQuestion] = useState("");
    const [answer, setAnswer] = useState<string>("");
    const [isLoading, setIsLoading] = useState(false);
    const [error, setError] = useState<string | undefined>(undefined);

    const handleSubmit = () => {};

    return (
    <main className="w-screen h-screen flex items-center justify-center bg-[#2e2e2e]">
        <form className="bg-[#2e2e2e] w-[540px] relative">
        <input
            className={`py-5 pl-6 pr-[40px] rounded-md bg-[#1f1f1f] w-full
            outline-[#1f1f1f] focus:outline outline-offset-2 text-[#b3b3b3]
            mb-8 placeholder-[#4d4d4d]`}
            placeholder="Ask a question..."
            value={prompt}
            onChange={(e) => {
              setPrompt(e.target.value);
            }}
        ></input>
        <button
            onClick={handleSubmit}
            className="absolute top-[25px] right-4"
            disabled={!prompt}
        >
            <ReturnIcon
            className={`${!prompt ? "fill-[#4d4d4d]" : "fill-[#1b9873]"}`}
            />
        </button>
        <div className="h-96 px-6">
            {question && (
            <p className="text-[#b3b3b3] pb-4 mb-8 border-b border-[#525252] ">
                {question}
            </p>
            )}
            {(isLoading && <LoadingDots />) ||
            (error && <p className="text-[#b3b3b3]">{error}</p>) ||
            (answer && <p className="text-[#b3b3b3]">{answer}</p>)}
        </div>
        </form>
    </main>
    );
}

function ReturnIcon({ className }: { className?: string }) {
    return (
        <svg
            width="20"
            height="12"
            viewBox="0 0 20 12"
            fill="none"
            xmlns="http://www.w3.org/2000/svg"
            className={className}
        >
            <path
            fillRule="evenodd"
            clipRule="evenodd"
            d={`M12 0C11.4477 0 11 0.447715 11 1C11 1.55228 11.4477 2 12
            2H17C17.5523 2 18 2.44771 18 3V6C18 6.55229 17.5523 7 17
            7H3.41436L4.70726 5.70711C5.09778 5.31658 5.09778 4.68342 4.70726
            4.29289C4.31673 3.90237 3.68357 3.90237 3.29304 4.29289L0.306297
            7.27964L0.292893 7.2928C0.18663 7.39906 0.109281 7.52329 0.0608469
            7.65571C0.0214847 7.76305 0 7.87902 0 8C0 8.23166 0.078771 8.44492
            0.210989 8.61445C0.23874 8.65004 0.268845 8.68369 0.30107
            8.71519L3.29289 11.707C3.68342 12.0975 4.31658 12.0975 4.70711
            11.707C5.09763 11.3165 5.09763 10.6833 4.70711 10.2928L3.41431
            9H17C18.6568 9 20 7.65685 20 6V3C20 1.34315 18.6568 0 17 0H12Z`}
            />
        </svg>
    );
}

function LoadingDots() {
    return (
        <div className="grid gap-2">
            <div className="flex items-center space-x-2 animate-pulse">
            <div className="w-1 h-1 bg-[#b3b3b3] rounded-full"></div>
            <div className="w-1 h-1 bg-[#b3b3b3] rounded-full"></div>
            <div className="w-1 h-1 bg-[#b3b3b3] rounded-full"></div>
            </div>
        </div>
    );
}

We have created an input field where the user can enter a question. The text the user types in the input field is captured as prompt. question is the submitted prompt that we show under the input when user submits their question. We clear the input and delete the prompt when user submits it, but keep the question value so the user can reference it.

Let’s look at the fleshed-out form submission handler function that we stubbed in earlier:

const handleSubmit = (
  e: KeyboardEvent | React.MouseEvent<HTMLButtonElement>
) => {
  e.preventDefault();

  setIsLoading(true);
  setQuestion(prompt);
  setAnswer("");
  setPrompt("");
  generateAnswer(prompt);
};

When the user submits a question, we set the isLoading state to true and show the loading indicator. We clear the prompt state and set the question state. We also clear the answer state because the answer may hold an answer to a previous question, but we want to start with an empty answer.

At this point we want to create a server-sent event and send a request to our api/generate-answer route. We will do this inside the generateAnswer function.

The browser-native SSE API doesn’t allow the client to send a payload to the server; the client is only able to open a connection to the server to begin receiving events from it via a GET request. In order for the client to be able to send a payload via a POST request to open the SSE connection, we will use the sse.js package, so let’s install it.

$ 

npm install sse.js

This package doesn’t have a corresponding types package, so we need to add them manually. Let’s create a new folder named types in the project root and an sse.d.ts file inside it.

$ 

mkdir types && touch types/sse.d.ts

Open sse.d.ts and add this code:

type SSEOptions = EventSourceInit & {
    payload?: string;
};

declare module "sse.js" {
    class SSE extends EventSource {
        constructor(url: string | URL, sseOptions?: SSEOptions);
        stream(): void;
    }
}

This extends the native EventStream by adding a payload to the constructor. We also added the stream function to it which is used to activate the stream in the sse.js library.

This allows us to import SSE in page.tsx and use it to open a connection to our handler route while also sending the user’s query.

"use client";

import { useState } from "react";
import { useState, useRef } from "react";
import { SSE } from "sse.js";
import { errors } from "./constants";

export default function Home() {
    const eventSourceRef = useRef<SSE>();

    const [prompt, setPrompt] = useState("");
    const [question, setQuestion] = useState("");
    const [answer, setAnswer] = useState<string>("");
    const [isLoading, setIsLoading] = useState(false);
    const [error, setError] = useState<string | undefined>(undefined);

    const handleSubmit = () => {};

    const generateAnswer = async (query: string) => {
        if (eventSourceRef.current) eventSourceRef.current.close();

        const eventSource = new SSE(`api/generate-answer`, {
            payload: JSON.stringify({ query }),
        });
        eventSourceRef.current = eventSource;

        eventSource.onerror = handleError;
        eventSource.onmessage = handleMessage;
        eventSource.stream();
    };

    handleError() { /* … */ }
    handleMessage() { /* … */ }
// …

Note that we save a reference to the eventSource object. We need this in case a user submits a new question while answer to the previous one is still assembling on the client. If we don’t close the existing connection to the server before opening the new one, this could cause problems since two connections will be open and trying to receive data.

We opened a connection to the server, and we are now ready to receive events from it. We just need to write handlers for those events so the UI knows what to do with them. We will get the answer as part of a message event, and if an error is returned, the server will send an error event to the client.

Let’s break down these handlers.

// …

function handleError(err: any) {
    setIsLoading(false);

    const errMessage =
    err.data === errors.flagged ? errors.flagged : errors.default;

    setError(errMessage);
}


function handleMessage(e: MessageEvent<any>) {
    try {
        setIsLoading(false);
        if (e.data === "[DONE]") return;

        const chunkResponse = JSON.parse(e.data);
        const chunk = chunkResponse.choices[0].delta?.content || "";
        setAnswer((answer) => answer + chunk);
    } catch (err) {
        handleError(err);
    }
}

When we get the message event, we extract the data from it and add it to the answer state until we receive all chunks. This is indicated when the data is equal to [DONE], meaning the whole answer has been received and the connection to the server will be closed. There is no data to be parsed in this case, so we return instead of trying to parse it. (An error will be thrown if we try to parse it in this case.)

"use client";

import { useState, useRef } from "react";
import { SSE } from "sse.js";
import { errors } from "./constants";

export default function Home() {
  const eventSourceRef = useRef<SSE>();

  const [prompt, setPrompt] = useState("");
  const [question, setQuestion] = useState("");
  const [answer, setAnswer] = useState<string>("");
  const [isLoading, setIsLoading] = useState(false);
  const [error, setError] = useState<string | undefined>(undefined);

  const handleSubmit = (
    e: KeyboardEvent | React.MouseEvent<HTMLButtonElement>
  ) => {
    e.preventDefault();

    setIsLoading(true);
    setQuestion(prompt);
    setAnswer("");
    setPrompt("");
    generateAnswer(prompt);
  };

  const generateAnswer = async (query: string) => {
    if (eventSourceRef.current) eventSourceRef.current.close();

    const eventSource = new SSE(`api/generate-answer`, {
      payload: JSON.stringify({ query }),
    });
    eventSourceRef.current = eventSource;

    eventSource.onerror = handleError;
    eventSource.onmessage = handleMessage;
    eventSource.stream();
  };

  function handleError(err: any) {
    setIsLoading(false);

    const errMessage =
      err.data === errors.flagged ? errors.flagged : errors.default;

    setError(errMessage);
  }

  function handleMessage(e: MessageEvent<any>) {
    try {
      setIsLoading(false);
      if (e.data === "[DONE]") return;

      const chunkResponse = JSON.parse(e.data);
      const chunk = chunkResponse.choices[0].delta?.content || "";
      setAnswer((answer) => answer + chunk);
    } catch (err) {
      handleError(err);
    }
  }

  return (
    <main className="w-screen h-screen flex items-center justify-center bg-[#2e2e2e]">
      <form className="bg-[#2e2e2e] w-[540px] relative">
        <input
          className={`py-5 pl-6 pr-[40px] rounded-md bg-[#1f1f1f] w-full
            outline-[#1f1f1f] focus:outline outline-offset-2 text-[#b3b3b3]
            mb-8 placeholder-[#4d4d4d]`}
          placeholder="Ask a question..."
          value={prompt}
          onChange={(e) => {
            setPrompt(e.target.value);
          }}
        ></input>
        <button
          onClick={handleSubmit}
          className="absolute top-[25px] right-4"
          disabled={!prompt}
        >
          <ReturnIcon
            className={`${!prompt ? "fill-[#4d4d4d]" : "fill-[#1b9873]"}`}
          />
        </button>
        <div className="h-96 px-6">
          {question && (
            <p className="text-[#b3b3b3] pb-4 mb-8 border-b border-[#525252] ">
              {question}
            </p>
          )}
          {(isLoading && <LoadingDots />) ||
            (error && <p className="text-[#b3b3b3]">{error}</p>) ||
            (answer && <p className="text-[#b3b3b3]">{answer}</p>)}
        </div>
      </form>
    </main>
  );
}

function ReturnIcon({ className }: { className?: string }) {
  return (
    <svg
      width="20"
      height="12"
      viewBox="0 0 20 12"
      fill="none"
      xmlns="http://www.w3.org/2000/svg"
      className={className}
    >
      <path
        fillRule="evenodd"
        clipRule="evenodd"
        d={`M12 0C11.4477 0 11 0.447715 11 1C11 1.55228 11.4477 2 12
            2H17C17.5523 2 18 2.44771 18 3V6C18 6.55229 17.5523 7 17
            7H3.41436L4.70726 5.70711C5.09778 5.31658 5.09778 4.68342 4.70726
            4.29289C4.31673 3.90237 3.68357 3.90237 3.29304 4.29289L0.306297
            7.27964L0.292893 7.2928C0.18663 7.39906 0.109281 7.52329 0.0608469
            7.65571C0.0214847 7.76305 0 7.87902 0 8C0 8.23166 0.078771 8.44492
            0.210989 8.61445C0.23874 8.65004 0.268845 8.68369 0.30107
            8.71519L3.29289 11.707C3.68342 12.0975 4.31658 12.0975 4.70711
            11.707C5.09763 11.3165 5.09763 10.6833 4.70711 10.2928L3.41431
            9H17C18.6568 9 20 7.65685 20 6V3C20 1.34315 18.6568 0 17 0H12Z`}
      />
    </svg>
  );
}

function LoadingDots() {
  return (
    <div className="grid gap-2">
      <div className="flex items-center space-x-2 animate-pulse">
        <div className="w-1 h-1 bg-[#b3b3b3] rounded-full"></div>
        <div className="w-1 h-1 bg-[#b3b3b3] rounded-full"></div>
        <div className="w-1 h-1 bg-[#b3b3b3] rounded-full"></div>
      </div>
    </div>
  );
}

Testing it out​

You should now be able to run the project to test it.

$ 

npm run dev

If you used our example documentation, the chatbot will know a few things about EdgeQL along with whatever it was trained on.

Some questions you might try:

• “What is EdgeQL?”

• “Who is EdgeQL for?”

• “How should I get started with EdgeQL?”

If you don’t like the responses you’re getting, here are a few things you might try tweaking:

• systemMessage in the createFullPrompt function in app/api/generate-answer/route.ts

• temperature in the getOpenAiAnswer in app/api/generate-answer/route.ts

• the matchThreshold value passed to the query from the getContext function in app/api/generate-answer/route.ts

You can see the finished source code for this build in our examples repo on GitHub. You might also find our actual implementation interesting. You’ll find it in our website repo. Pay close attention to the contents of buildTools/gpt, where the embedding generation happens and components/gpt, which contains most of the UI for our chatbot.

If you have trouble with the build or just want to hang out with other EdgeDB users, please join our awesome community on Discord!