search

III Search Agent

A. Introduction

Search Agent

Private Search Agent that can cite data sources

Explore the Search Agent SDK documentation. This handbook is designed to assist you in merging a variety of data origins into your search agent, enriching its conversational functionalities. Topics include initial setup, integration of different data streams, and tailoring your agent with universal parameters. Let's delve into it!

Introduction

The Search Agent SDK offers a robust set of tools for crafting advanced search functionalities within your chat or search engines. Through the integration of diverse data sources and harnessing cutting-edge machine learning models, you can develop an agent capable of comprehending and delivering responses to queries with exceptional relevance and precision.

AI-Horizon’s Search Agent integrates all the building blocks of a Search

What methods and arguments are available for passing to the AI-Horizon’s Search class?

Methods

MethodWhat it does?
add_pdftrain and search across PDF documents
add_websiteautomatically scrape the website content, train and search across website data
add_docxtrain and search across Microsoft Word documents
add_txttrain and search across flat files
add_youtubetrain and search across YouTube content (must have transcriptions)
add_webpageautomatically scrape the webpage content, train and search across webpage data

B. Usage

a) Quick Start

Before diving into the SearchAgent SDK, it's essential to confirm that your environment includes all required dependencies. Although exact installation commands aren't outlined here, you can utilize standard Python package management tools such as pip to handle your environment setup.

Key Components

The SDK consists of several essential components, each designed to manage various types of content:

  • PDF Files
  • DOCX Files
  • Text Files
  • Webpages
  • Websites
  • YouTube Videos

These components enable the SearchAgent to train, process, and index content from these sources, rendering it searchable within your Search Agent.

Using the SearchAgent SDK

The following sections provide a step-by-step guide to using the SearchAgent SDK in your projects.

Initializing the SearchAgent

To use the SDK, start by importing and initializing the SearchAgent class:

from <<package name>> import SearchAgent 
 
# Initialize the SearchAgent
search_agent = SearchAgent()

Setup Vector Database

First install Qdrant Vector Database with pip install qdrant_client

import qdrant_client
 
QDRANT_CLIENT_URL = "<YOUR-QDRANT-CLIENT-URL>"
QDRANT_API_KEY = "<YOUR-QDRANT-API-KEY>"
 
client = qdrant_client.QdrantClient(
    url=QDRANT_CLIENT_URL,
    api_key=QDRANT_CLIENT_URL,
)
 
vector_store_params = {
    "vector_store_type": "QdrantVectorStore",
    "client": client,
    "collection_name": "some_name"
}

Search Agent supports a variety of Vector Databases

Setup Large Language Model

import os
import openai
 
os.environ["OPENAI_API_KEY"] = "<YOUR-OPENAI-API-KEY>"
 
llm_params = {
    "model": "gpt-4-turbo-preview",
    "api_key": "<YOUR-OPENAI-API-KEY>"
}

Search Agent supports a variety of Large Language Models

Training Search Agent

With the SearchAgent initialized, you can begin adding content from various sources. The SDK provides methods for integrating different content types, as outlined below.

Adding PDF Files

search_agent.add_pdf(
    input_files=["path-to-file"],
    vector_store_params=vector_store_params,
    llm_params=llm_params
)

Adding DOCX Files

search_agent.add_docx(
    input_files=["path-to-file"],
    vector_store_params=vector_store_params,
    llm_params=llm_params
)

Adding Text Files

search_agent.add_text(
    input_files=["path-to-file"],
    vector_store_params=vector_store_params,
    llm_params=llm_params
)

Adding Webpages

search_agent.add_webpage(
    url="https://example.com",
    vector_store_params=vector_store_params,
    llm_params=llm_params
)

Adding Websites

search_agent.add_website(
    url="https://sample.com",
    vector_store_params=vector_store_params,
    llm_params=llm_params
)

Example Usage

Adding Text Files from a Directory

This snippet configures the chat agent to ingest text files from the specified directory and its subdirectories.

chat_agent.txt_chat(
    input_dir="/path/to/your_text/files",
    recursive=True
)

Let’s Search

Ask a question

response = agent.query("<you-question>")

Sources

for n, source in enumerate(response.source_nodes):
  print(source.text)
  print(source.score)
  print(source.metadata)

Advanced Configuration

Each method provides a variety of parameters, allowing for precise customization of content processing and indexing. These parameters include options for handling hidden files, controlling recursive directory processing, and specifying file extensions.

C. Data Sources

a) Overview

  • Add PDF
  • Add Docx
  • Add Text
  • Add YouTube
  • Add Webpage
  • Add Website

b) Data Types

i. PDF

Add PDF Files to Your Search Agent

Incorporating PDF documents into your search agent vastly improves its capacity to furnish thorough search outcomes by indexing the content within these documents. The add_pdf method streamlines the inclusion of PDF files into your search agent, utilizing a range of parameters for tailoring and refining the process.

Function Signature:

The add_pdf function is crafted with versatility in mind, capable of accommodating different scenarios, ranging from adding a single PDF file to incorporating entire directories of PDF documents.

Parameters:

  • input_dir (Optional[str]): The directory path containing PDF files to be added. If specified, the method scans this directory for PDF files.
  • input_files (Optional[List]): A list of paths to individual PDF files to be added. If provided, input_dir is ignored.
  • exclude_hidden (bool): When set to True, hidden files or files starting with a dot (.) in input_dir are excluded.
  • filename_as_id (bool): If True, uses the filename as the unique identifier for each PDF document in the database.
  • recursive (bool): If set to True, the method also searches subdirectories within input_dir for PDF files.
  • required_exts (Optional[List[str]]): Specifies file extensions to include. Defaults to [".pdf"] to target PDF files.
  • system_prompt (str): An optional prompt to guide the system in processing PDF content.
  • query_wrapper_prompt (str): An optional prompt that wraps user queries, enhancing the relevance of search results.
  • embed_model (Union[str, EmbedType]): Specifies the embedding model for text extraction and embedding. The default setting uses the predefined model.
  • llm_params (dict): Parameters for configuring the integration with Large Language Models, enhancing content understanding and query processing.
  • vector_store_params (dict): Configuration for the vector store, defining how and where the extracted embeddings are stored.
  • service_context_params (dict): Additional parameters for customizing the service context.
  • query_engine_params (dict): Parameters for customizing the query engine’s behavior.
  • retriever_params (dict): Configuration for the retriever component, affecting how documents are retrieved based on queries.

Example Usage:

Adding a Directory of PDF Files

search_agent.add_pdf(
    input_dir="/path/to/your_pdf/documents",
    recursive=True
)

This example scans the specified directory (and its subdirectories) for PDF files, adding them to the search agent’s database.

Adding Specific PDF Files

search_agent.add_pdf(
    input_files=["/path/to/your/document1.pdf", "/path/to/your/document2.pdf"],
)

ii. Docx file

Add DOCX Files to Your Search Agent

Integrating DOCX (Microsoft Word) documents into your search agent enables it to index and search through a diverse array of structured text content, enhancing the agent’s capacity to comprehend and address queries with pertinent information. The add_docx method is engineered to simplify the process of integrating DOCX files into your search agent.

Function Signature

Parameters

  • input_dir (Optional[str]): Directory containing DOCX files. If specified, the function searches this directory for files to add.
  • input_files (Optional[List]): A specific list of DOCX file paths to add. If provided, input_dir is ignored.
  • exclude_hidden (bool): If True, hidden files or files starting with a dot (.) are excluded from input_dir.
  • filename_as_id (bool): Uses the filename as the document’s unique identifier if set to True.
  • recursive (bool): Searches subdirectories within input_dir for DOCX files if True.
  • required_exts (Optional[List[str]]): File extensions to include. Defaults to targeting DOCX files.
  • system_prompt (str): Optional prompt guiding the system in processing DOCX content.
  • query_wrapper_prompt (str): Optional prompt enhancing query relevance by wrapping user queries.
  • embed_model (Union[str, EmbedType]): Embedding model for text extraction and embedding. Defaults to a predefined model.
  • llm_params (dict): Configuration parameters for integrating Large Language Models.
  • vector_store_params (dict): Configuration for vector storage, defining embedding storage and retrieval.
  • service_context_params (dict): Additional service context configuration.
  • query_engine_params (dict): Customization parameters for the query engine.
  • retriever_params (dict): Configuration for the document retriever, affecting document retrieval strategies.

Example Usage

Adding DOCX Files from a Directory

search_agent.add_docx(
    input_dir="/path/to/your/docx/documents",
    recursive=True
)

This code snippet adds DOCX files from the specified directory and its subdirectories.

Adding Specific DOCX Files

search_agent.add_docx(
    input_files=["/path/to/your/file1.docx", "/path/to/your/file2.docx"],
)

iii. Text

Add Text Files to Your Search Agent

Including plain text (.txt) documents in your search agent is vital for expanding its knowledge repository, empowering it to gather information from a vast array of text-based documents. The add_text method streamlines this procedure, facilitating the seamless integration of text content into the agent’s searchable data.

Function Signature

The add_text function is tailored for adaptability, enabling the inclusion of text files either from a designated directory or as an inventory of individual files.

Parameters

  • input_dir (Optional[str]): Directory path containing text files to be added. If specified, the function searches this directory for eligible files.
  • input_files (Optional[List]): A list of specific text file paths to add. Takes precedence over input_dir if provided.
  • exclude_hidden (bool): If True, ignores hidden files or files starting with a dot (.) within input_dir.
  • filename_as_id (bool): If True, uses the filename as the unique identifier for each document.
  • recursive (bool): If True, includes files from subdirectories within input_dir.
  • required_exts (Optional[List[str]]): Specifies the file extensions to include, defaulting to text files.
  • system_prompt (str): An optional prompt to guide the system in processing text content.
  • query_wrapper_prompt (str): An optional prompt to enhance the relevance of user queries by wrapping them.
  • embed_model (Union[str, EmbedType]): The embedding model used for text extraction and embedding, defaulting to a standard model.
  • llm_params (dict): Configuration parameters for integrating Large Language Models, if needed.
  • vector_store_params (dict): Configuration for the vector storage, detailing how and where embeddings are stored.
  • service_context_params (dict): Additional parameters for customizing the service context.
  • query_engine_params (dict): Parameters to customize the behavior of the query engine.
  • retriever_params (dict): Configuration for the document retriever, influencing how documents are retrieved based on queries.

Example Usage

Adding Text Files from a Directory

search_agent.add_text(
    input_dir="/path/to/text/files",
    recursive=True
)

This snippet scans the specified directory (and subdirectories, if recursive is True) for text files, adding them to the search agent’s database.

Adding Specific Text Files

search_agent.add_text(
    input_files=["/path/to/specific_file1.txt", "/path/to/specific_file2.txt"],
)

Integrating YouTube Content into Your Search Agent

Broadening the scope of your search agent to encompass content from YouTube videos can greatly boost its capacity to deliver compelling and multimedia-enriched responses. Through the integration of the add_youtube function, incorporating YouTube videos becomes seamless, empowering your search agent to utilize video titles, descriptions, and potentially transcribed content for a broader and more varied search encounter.

Function Overview

The add_youtube method is tailor-made for seamlessly integrating YouTube video content into your search agent. It provides a variety of parameters to personalize the processing, indexing, and retrieval of this content.

Parameters

  • urls (List[str]): A list of URLs for the YouTube videos you want to add. Each URL points to a specific video whose content will be made searchable by your agent.
  • system_prompt (str): An optional prompt to guide the system in processing YouTube video content. This can direct the focus towards relevant aspects of the video, such as the description or transcribed speech.
  • query_wrapper_prompt (str): An optional prompt to enhance the relevance of search queries by providing a context specific to the YouTube content.
  • embed_model (Union[str, EmbedType]): The embedding model used for processing and embedding the text content extracted from YouTube videos. Defaults to a standard model suitable for general content.
  • llm_params (dict): Parameters for integrating Large Language Models, enhancing the agent’s understanding of video content and improving query processing.
  • vector_store_params (dict): Configuration for vector storage, detailing how and where the extracted embeddings from video content are stored.
  • service_context_params (dict): Additional parameters to customize the service context for YouTube content integration.
  • query_engine_params (dict): Customization parameters for the query engine, affecting how YouTube content is searched and matched with user queries.
  • retriever_params (dict): Configuration for the document retriever component, determining how YouTube video content is indexed and retrieved.

Example Usage

search_agent.add_youtube(
    urls=[
        "https://www.youtube.com/watch?v=video1",
        "https://www.youtube.com/watch?v=video2"
    ],
)

Integrating YouTube Content into Your Search Agent

Broadening the scope of your search agent to encompass content from YouTube videos can greatly boost its capacity to deliver compelling and multimedia-enriched responses. Through the integration of the add_youtube function, incorporating YouTube videos becomes seamless, empowering your search agent to utilize video titles, descriptions, and potentially transcribed content for a broader and more varied search encounter.

Function Overview

The add_youtube method is tailor-made for seamlessly integrating YouTube video content into your search agent. It provides a variety of parameters to personalize the processing, indexing, and retrieval of this content.

Parameters

  • urls (List[str]): A list of URLs for the YouTube videos you want to add. Each URL points to a specific video whose content will be made searchable by your agent.
  • system_prompt (str): An optional prompt to guide the system in processing YouTube video content. This can direct the focus towards relevant aspects of the video, such as the description or transcribed speech.
  • query_wrapper_prompt (str): An optional prompt to enhance the relevance of search queries by providing a context specific to the YouTube content.
  • embed_model (Union[str, EmbedType]): The embedding model used for processing and embedding the text content extracted from YouTube videos. Defaults to a standard model suitable for general content.
  • llm_params (dict): Parameters for integrating Large Language Models, enhancing the agent’s understanding of video content and improving query processing.
  • vector_store_params (dict): Configuration for vector storage, detailing how and where the extracted embeddings from video content are stored.
  • service_context_params (dict): Additional parameters to customize the service context for YouTube content integration.
  • query_engine_params (dict): Customization parameters for the query engine, affecting how YouTube content is searched and matched with user queries.
  • retriever_params (dict): Configuration for the document retriever component, determining how YouTube video content is indexed and retrieved.

Example Usage

search_agent.add_youtube(
    urls=[
        "https://www.youtube.com/watch?v=video1",
        "https://www.youtube.com/watch?v=video2"
    ],
)

Integrating Webpage Content into Your Search Agent

Incorporating individual webpages into your search agent facilitates the integration of targeted, high caliber content that directly addresses user inquiries. The add_webpage method simplifies the incorporation of content from individual webpages, elevating the search agent's capacity to deliver accurate and pertinent search outcomes.

Function Signature

The add_webpage function is designed specifically for incorporating content from a solitary webpage, offering various parameters to precisely adjust how this content is processed and indexed.

Parameters

  • url (Optional[str]): The URL of the webpage to be added. This specifies the exact page whose content you wish to make searchable.
  • system_prompt (str): An optional prompt to guide the system in how to process the content of the webpage. It can be used to direct the focus of content extraction towards relevant information.
  • query_wrapper_prompt (str): An optional prompt that can enhance the relevance of search queries by wrapping them in a specific context related to the webpage content.
  • embed_model (Union[str, EmbedType]): The embedding model used for processing and embedding the webpage’s text content. Defaults to a standard model suitable for general web content.
  • llm_params (dict): Parameters for integrating Large Language Models to augment content understanding and query processing capabilities.
  • vector_store_params (dict): Configuration for the vector storage, specifying how and where the extracted content embeddings are stored.
  • service_context_params (dict): Additional parameters to customize the service context specific to the webpage’s content.
  • query_engine_params (dict): Customization parameters for the query engine, affecting how the webpage content is searched and matched with queries.
  • retriever_params (dict): Configuration for the document retriever component, determining how the webpage content is indexed and retrieved based on search queries.

Example Usage

search_agent.add_webpage(
    url="https://www.specificpage.com/article",
)

Integrating Website Content into Your Search Agent

Integrating entire websites into your search agent greatly improves its capacity to deliver thorough and pertinent search outcomes. The add_website method is crafted to simplify the seamless assimilation of web content, enriching your agent's knowledge repository with abundant, web-based information.

Function Signature

The add_website function offers an efficient method for acquiring content from designated websites, enabling your search agent to retrieve and index web pages seamlessly.

Parameters

  • url (Optional[str]): The URL of the website to be added. This is the starting point for content integration.
  • system_prompt (str): An optional prompt to guide the system in processing website content. It can be used to specify instructions or objectives for the content retrieval process.
  • query_wrapper_prompt (str): An optional prompt to enhance the relevance of user queries by wrapping them. This can be particularly useful for tailoring the search experience based on the website’s content.
  • embed_model (Union[str, EmbedType]): Specifies the embedding model used for processing the website’s text content. The default setting uses a standard model optimized for web content.
  • llm_params (dict): Parameters for integrating Large Language Models, enhancing content understanding and processing.
  • vector_store_params (dict): Configuration parameters defining how and where the extracted content embeddings are stored.
  • service_context_params (dict): Additional parameters to customize the service context for the website content.
  • query_engine_params (dict): Customization parameters for the query engine, tailoring how search queries are processed and matched with the website content.
  • retriever_params (dict): Configuration for the document retriever component, influencing how web content is retrieved and indexed based on search queries.

Example Usage

search_agent.add_website(
    url="https://www.example.com",
)

Advanced Configuration

A. Vector Databases

a) Weaviate

AI-Horizon + Weaviate Vector Database

Weaviate serves as a vector search engine, facilitating scalable and effective vector searches. You can seamlessly incorporate Weaviate into your AI-Horizon application, whether for local embedded environments or cloud-based or self-hosted setups.

Local Embedded
vector_store_params = {
    "vector_store_type": "WeaviateVectorStore",
    "index_name": "Indexname", 
}
Cloud / Self Hosted
pip install weaviate-client
 
import weaviate
 
resource_owner_config = weaviate.AuthClientPassword(
    username="<username>",
    password="<password>",
)
 
client = weaviate.Client(
    "https://xxyy.weaviate.network",
    auth_client_secret=resource_owner_config,
)
 
vector_store_params = {
    "vector_store_type": "WeaviateVectorStore",
    "weaviate_client": client,
    "api_key": "DB_API_KEY",
    "index_name": "Indexname",  
}
b) Qdrant
# Installation
pip install -U qdrant_client
 
# Cloud / Self Hosted
import qdrant_client
 
client = qdrant_client.QdrantClient( 
    uri="http://<host>:<port>",
    api_key="<qdrant-api-key>",
)
 
vector_store_params = {
    "vector_store_type": "QdrantVectorStore",
    "client": client,
    "collection_name": "some_name",
}
c). Pinecone
Pinecone Setup:

Prior to integrating Pinecone with AI-Horizon, make sure to have a Pinecone account and to create a project and an index within the Pinecone platform. Additionally, you'll need to install the Pinecone client library in your Python environment.

# Installation
# To use Pinecone with AI-Horizon, first, install the Pinecone Python client:
pip install pinecone-client>=4.0.0

Configuration for AI-Horizon Integration To integrate Pinecone with AI-Horizon, you'll need to set up a series of parameters enabling communication between AI-Horizon and Pinecone's vector database services. These parameters encompass defining Pinecone as the vector store type, inputting your Pinecone API key, and specifying the name of the Pinecone index you intend to use.

from pinecone import Pinecone, PodSpec
 
os.environ["OPENAI_API_KEY"] = "sk-"
 
api_key = os.environ["PINECONE_API_KEY"]
 
 
pc = Pinecone(api_key=api_key)
 
pc.create_index(
  name='some_name',
  dimension=1536,
  metric='cosine',
  spec=PodSpec(
   environment='gcp-starter',
   pod_type='starter',
   pods=1
  )
)
 
pinecone_index = pc.Index("some_name")
 
vector_store_params = {
    "vector_store_type":"PineconeVectorStore",
    "pinecone_index":pinecone_index,
}

vector_store_type: Specifies that Pinecone is the vector database to be used. index_name: The name of the Pinecone index you want to use for storing and searching vectors. Ensure this index has been created in your Pinecone project.

d.) Azure Cognitive Search

Azure Cognitive Search is a cloud-based search service empowered with AI functionalities. Incorporating Azure Cognitive Search into your AI-Horizon application enables advanced search functionalities, such as vector search.

Installation

pip install azure-search-documents==11.4.0 azure-identity
import os
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents import SearchClient
from azure.core.credentials import AzureKeyCredential
 
search_service_name = os.environ["Azure Cognitive Search Service Name"]
key = os.environ["Azure Cognitive Search Key"]
 
cognitive_search_credential = AzureKeyCredential(key)
service_endpoint = f"https://{search_service_name}.search.windows.net"
 
 
index_client = SearchIndexClient(
    endpoint=service_endpoint,
    credential=cognitive_search_credential,
)
 
vector_store_params = {
    "vector_store_type": "CognitiveSearchVectorStore",
    "search_or_index_client": index_client,
    "index_name": "some_name",
}

e.) LanceDB

LanceDB provides effective storage solutions for vector and time-series data, ideal for both local and cloud environments where swift data retrieval is essential.

Installation

pip install -U lanced
 
Local
vector_store_params = {
    "vector_store_type": "LanceDBVectorStore",
    "uri": "./lancedb",
    "table_name": "some_name",
}

B. Large Language Model

OpenAI

Incorporating OpenAI’s models into AI-Horizon allows your applications to harness the advanced capabilities of models like GPT-4. This process entails defining a customized set of parameters for OpenAI’s API, which includes specifying the model name, your API key, and generation controls.

Configuration Steps

  1. Model Selection: Choose the appropriate OpenAI model for your needs. Models vary in size, capability, and cost. For example, “gpt-4-0125-preview” represents a version of GPT-4.
  2. API Key: Obtain an API key from OpenAI. This key is essential for authenticating your requests to the OpenAI API.
  3. Generation Parameters: Configure generation parameters like temperature and top_p to control the model’s output characteristics.

Example Configuration

llm_params = {
    "model": "gpt-4-0125-preview",
    "api_key": "sk-<YOUR-API-KEY>",
    "temperature": 0,
    "top_p": 0,
}

Azure OpenAI

For enterprises who favor Microsoft Azure’s infrastructure or have particular needs fulfilled by Azure services, the integration of Azure OpenAI models is simple. This procedure includes incorporating additional parameters such as the Azure API base URL and version.

Configuration Steps

  1. Model Selection: Azure offers various models, including custom and pre-trained options. Select a model that fits your application needs, such as “azure/gpt-35-turbo-16k”.
  2. API Base and Version: Specify your Azure API base URL and the version of the OpenAI service you’re using.
  3. API Key: Similar to OpenAI, an API key from Azure is required for authentication.
  4. Max Retries: Define the maximum number of retries for API requests to handle transient failures gracefully.

Example Configuration

llm_params = {
    "model": "azure/gpt-35-turbo-16k",
    "api_base": "https://<deployment_name>.azure.com/",
    "api_version": "2024-02-15-preview",
    "api_key": "<YOUR-API-KEY>",
    "max_retries": 1
}

Ensure to replace <deployment_name> and <YOUR-API-KEY> with your specific deployment name and API key.

chatIntroduction