LLM Index Document

The LLM Index Document task is used to index a document into a vector database for efficient search, retrieval, and processing at a later stage.

The task uses a large language model (LLM) to create embeddings of the indexed document text, which are then stored in the vector database for later retrieval.

Prerequisites

• Integrate the required AI model with Orkes Conductor.
• Integrate the required vector database with Orkes Conductor.

Task parameters

Configure these parameters for the LLM Index Document task.

Parameter	Description	Required/ Optional
inputParameters.vectorDB	The vector database to store the data. Note: If you haven’t configured the vector database on your Orkes Conductor cluster, navigate to the Integrations tab and configure your required provider. Refer to the documentation on how to integrate Vector Databases with Orkes console.	Required.
inputParameters.index	The index in your vector database where the text or data will be stored. The terminology of the index field varies depending on the integration: For Weaviate, the index field indicates the class name. For other integrations, it denotes the index name.	Required.
inputParameters.namespace	Namespaces are separate isolated environments within the database to manage and organize vector data effectively. Choose from the available namespace configured within the chosen vector database. The usage and terminology of the namespace field vary depending on the integration: For Pinecone, the namespace field is applicable. For Weaviate, the namespace field is not applicable. For MongoDB, the namespace field is referred to as “Collection” in MongoDB. For Postgres, the namespace field is referred to as “Table” in Postgres.	Required.
inputParameters.embeddingModelProvider	The LLM provider for generating the embeddings. Note: If you haven’t configured your AI/LLM provider on your Orkes console, navigate to the Integrations tab and configure your required provider. Refer to the documentation on how to integrate the LLM providers with Orkes Conductor.	Required.
inputParameters.embeddingModel	The embedding model provided by the selected LLM provider to generate the embeddings.	Required.
inputParameters.url	The URL of the file to be indexed.	Required.
inputParameters.mediaType	The media type of the file to be indexed. Supported media types: application/pdf text/html text/plain application/json	Optional.
inputParameters.chunkSize	The length of each input text segment when divided for processing by the LLM. For example, if the document contains 2,000 words and the chunk size is set to 500, the document is divided into four chunks for processing.	Optional.
inputParameters.chunkOverlap	The overlap between adjacent chunks. For example, if the chunk overlap is specified as 100, then the first 100 words of each chunk would overlap with the last 100 words of the previous chunk.	Optional.
inputParameters.dimensions	The size of the vector, which is the number of elements in the vector.	Optional.

Task configuration

This is the task configuration for an LLM Index Document task.

{
    "name": "llm_index_document_task",
    "taskReferenceName": "llm_index_document_task_ref",
    "inputParameters": {
        "vectorDB": "pineconedb",
        "namespace": "myNewModel",
        "index": "test",
        "embeddingModelProvider": "azure_openai",
        "embeddingModel": "text-davinci-003",
        "url": "${workflow.input.url}",
        "mediaType": "application/xhtml+xml",
        "chunkSize": 500,
        "chunkOverlap": 100,
        "dimensions": "${workflow.input.dimensions}"
    },
    "type": "LLM_INDEX_DOCUMENT"
}

Task output

There is no output. The LLM Index Document task will store the indexed data in the specified vector database.

Adding an LLM Index Document task in UI

To add an LLM Index Document task:

1. In your workflow, select the (+) icon and add an LLM Index Document task.
2. Choose the Vector database, Index, Namespace, Embedding model provider, and Embedding model.
3. Enter the URL of the document to be indexed.
4. Choose the Media type, and enter the Chunk Size and Chunk Overlap.