How Do Vector Databases Shape the Future of Generative AI Solutions?

Introduction

In the rapidly evolving landscape of generative AI, the pivotal role of vector databases has become increasingly apparent. This article dives into the dynamic synergy between vector databases and generative AI solutions, exploring how these technological bedrocks are shaping the future of artificial intelligence creativity. Join us on a journey through the intricacies of this powerful alliance, unlocking insights into the transformative impact that vector databases bring to the forefront of innovative AI solutions.

Learning Objectives

This article helps you understand the aspects of the Vector Database below.

• Significance of Vector Databases and its key components
• Detailed study of Vector database comparison with Traditional database
• Exploration of Vector Embeddings from an application-point-of-view
• Vector database building using Pincone
• Implementation of Pinecone Vector database using langchain LLM model

This article was published as a part of the Data Science Blogathon.

What is Vector Database?

A vector database is a form of data collection stored in space. Still, here, it is stored in mathematical representations since the format stored in the databases makes it easier for open AI models to memorize the inputs and allows our open AI application to use cognitive search, recommendations, and text generation for various-use cases in the digitally-transformed -industries. Storing data and retrieval is called “Vector Embeddings” or “Embeddings.” Moreover, this is represented in a numerical array format. Searching is much easier than traditional databases used for AI perspectives with massive, indexed capabilities.

Characteristics of Vector Databases

• It leverages the power of these vector embeddings, leading to indexing and searching across a massive dataset.
• Compactable with all data formats (images, text, or data).
• Since it adapts embedding techniques and highly indexed features, it can offer a complete solution for managing data and input for the given problem.
• A vector database organizes data through high-dimensional vectors containing hundreds of dimensions. We can configure them very quickly.
• Each dimension corresponds to a specific feature or property of the data object it represents.

Traditional Vs. Vector Database

• The picture shows the traditional and vector database high-level workflow
• Formal database interactions happen through SQL statements and data stored in row-base and tabular format.
• In the Vector database, interactions happen through plain text (e.g., English) and data stored in mathematical representations.

Likeness of Traditional and Vector Databases

We must consider how Vector databases differ from traditional ones. Let’s discuss this here. One quick difference I can give is that in conventional databases. Data is stored precisely as-is; we could add some business logic to tune the data and merge or split the data based on the business requirements or demands. However, the vector database has a massive transformation, and the data becomes a complex vector representation.

Here’s a map for your understanding and clarity perspective with relational databases against vector databases. The picture below is self-explanatory for understanding vector databases with traditional databases. In short, we can execute inserts and deletes into vector databases, not update statements.

Simple Analogy to Understand Vector Databases

Data is automatically arranged spatially by the content similarity in the stored information. So, let’s consider the departmental store for vector database analogy; all the products are arranged on the shelf based on nature, purpose, manufacture, usage, and quantity-base. In a similar behaviour, the data are
automatically-arranged in the vector database by a similar sort, even if the genre was not well-defined while storing or accessing the data.

The vector databases allow a prominent granularity and dimensions on the specific similarities, so the customer searches for the desired product, manufacturer, and quantity and keeps the item in the cart. Vector database stores all data in a perfect storage structure; here, Machine Learning and AI engineers do not need to label or tag the stored content manually.

Essential theories behind Vector Databases

• Vector Embeddings and their Scope
• Indexing Requirements
• Understanding Semantic and Similarity Search

Vector Embedding and their Scope

A vector embedding is a vector representation in terms of the numerical values. In a compressed format, embeddings capture the inherent properties and associations of the original data, making them a staple in Artificial Intelligence and Machine Learning use cases. Designing embeddings to encode pertinent information about the original data into a lower-dimensional space ensures high-retrieval speed, computational efficiency, and efficient storage.

Capturing the essence of data in a more identically structured manner is the process of vector embedding, forming an ‘Embedding Model.’ Ultimately, these models consider all data objects, extract meaningful patterns and relations within the data source, and transform them into vector embeddings. Subsequently, algorithms leverage these vector embeddings to execute various tasks. Numerous highly developed embedding models, available online as either free or pay-as-you-go, facilitate the accomplishment of vector embedding.

Scope of Vector Embeddings from an Application-point-of-view

These embeddings are compact, contain complex information, inherit relationships among the data stored in a vector database, enable an efficient data-processing analysis to facilitate understanding and decision-making, and dynamically build various innovative data products across any organisation.

Vector embedding techniques are essential in connecting the gap between readable data and complex algorithms. With data types being numerical vectors, we were able to unlock the potential for a large variety of Generative AI applications along with available Open AI models.

Multiple Jobs with Vector Embedding

This vector embedding helps us to do multiple jobs:

• Retrieval of Information: With the help of these powerful techniques, we can build influential search engines that can help us find responses based on user queries from stored files, documents, or media
• Similarity Search Operations: This is well-organised and indexed; it helps us find the similarity between different occurrences in the vector data.
• Classification and Clustering: Using these embedding techniques, we can perform these models to train relevant machine learning algorithms and group and classify them.
• Recommendation Systems: Since the embedding techniques are organized properly, it leads to recommendation systems accurately relating products, media, and articles based on historical data.
• Sentiment Analysis: This embedding model helps us to categorize and derive sentiment solutions.

Indexing Requirements

As we know, the index will improve the search data from the table in traditional databases, similar to vector-databases, and provision the indexing features.

Vector databases provide “Flat indices,” which are the direct representation of the vector embedding. The search capability is comprehensive, and this does not use pre-trained clusters. It performs the query vector is performed across each single vector embedding, and K distances are calculated for each pair.

• Because of the ease of this index, minimal computation is required to create the new indices.
• Indeed, a flat index can handle queries effectively and provide quick retrieval times.

Understanding Semantic and Similarity Search

We perform two different searches in vector databases: semantic and similarity searches.

• Semantic search: While searching for information, instead of searching by keywords, you can find them based on meaningful conversation methodology. Prompt engineering plays a vital role in passing the input to the system. This search undoubtedly allows higher-quality search and results that can be fed for innovative applications, SEO, Text generation, and Summarising.
• Similarity Search: Always in data analysis, the similarity search allows for unstructured, much better-given datasets. Regarding vector databases, we must ascertain the closeness of two vectors and how they resemble each other: tables, text, documents, images, words, and audio files. In the process of understanding, the similarity between vectors is revealed as the similarity between the data objects in the given dataset. This exercise helps us understand interaction, identify patterns, extract insights, and make decisions from application perspectives. The Semantic and Similarity search would help us build the applications below for industry benefits.
• Information Retrieval: Using Open AI and Vector Databases, we would build search engines for information retrieval using business users’ or end users’ queries and indexed documents inside the vector DB.
• Classification and Clustering:Classifying or clustering similar data points or groups of objects involves assigning them to multiple categories based on shared characteristics.
• Anomaly Detection: Discovering abnormalities from usual patterns by measuring the similarity of data points and spotting irregularities.

Types of Similarity Measures in Vector Databases

The measuring methods depend on the nature of the data and the application specific. Commonly, three methods are used to measure the similarity and familiarity with Machine Learning.

Euclidean Distance

In simple terms, the distance between the two vectors is the straight-line distance between the two vector points that measure the st.

Dot Product

This helps us understand the alignment between two vectors, indicating whether they point in the same direction, opposite directions, or are perpendicular to each other.

Cosine Similarity

It assesses the similarity of two vectors by using the angle between them, as shown in the figure. In this case, the values and magnitude of the vectors are insignificant and do not affect the results; only the angle is considered in the calculation.

Traditional databases Search for exact SQL statement matches and retrieve the data in tabular format. At the same time, we deal with vector databases searching for the most similar vector to the input query in plain English using Prompt Engineering techniques. The database uses the Approximate Nearest Neighbour(ANN) search algorithm to find similar data. Always provide reasonably accurate results at high performance, accuracy, and response time.

Working Mechanism

• Vector databases first convert data into embedding vectors, store it in vector databases, and create indexing for quicker searching.
• A query from the application will interact with the embedding vector, searching for the nearest neighbour or similar data in the vector database using an index and retrieving the results passed to the application.
• Basis the business requirements, the retrieved data would be fine-tuned, formatted, and displayed to the end user side or query or action(s) feed.

Creating a Vector Database

Let’s connect with Pinecone.

https://app.pinecone.io/

You can connect to Pinecone using Google, GitHub, or Microsoft ID.

Create a new user login for your usage.

After successful login, you will land on the Index page; you can create an index for your Vector Database purposes. Click on the Create Index button.

Create your new index by providing the Name and Dimensions.

Index list page,

Index details – Name, Region, and Environment – We need all these details to connect our vector database from the model building code.

Project settings details,

You can upgrade your preferences for multiple indexes and keys for project purposes.

So far, we have discussed creating the vector database index and settings in Pinecone.

Vector Database Implementation Using Python

Let’s do some coding now.

Importing libraries

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.llms import OpenAI
from langchain.vectorstores import Pinecone
from langchain.document_loaders import TextLoader
from langchain.chains.question_answering import load_qa_chain
from langchain.chat_models import ChatOpenAI

Providing API key for OpenAI and Vector database

import os
os.environ["OPENAI_API_KEY"] = "xxxxxxxx"

PINECONE_API_KEY = os.environ.get('PINECONE_API_KEY', 'xxxxxxxxxxxxxxxxxxxxxxx')
PINECONE_API_ENV = os.environ.get('PINECONE_API_ENV', 'gcp-starter')
api_keys="xxxxxxxxxxxxxxxxxxxxxx"

llm = OpenAI(OpenAI=api_keys, temperature=0.1)

Initiating the LLM

llm=OpenAI(openai_api_key=os.environ["OPENAI_API_KEY"],temperature=0.6)

Initiating Pinecone

import pinecone
pinecone.init(
    api_key=PINECONE_API_KEY,  
    environment=PINECONE_API_ENV
index_name = "demoindex" 

Loading .csv file for building vector database

from langchain.document_loaders.csv_loader import CSVLoader
loader = CSVLoader(file_path="/content/drive/My Drive/Colab_Notebooks/cereal.csv"
,source_column="name")
data = loader.load()

Split the text into Chunks

text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=20)
text_chunks = text_splitter.split_documents(data)

Finding the text in text_chunk

text_chunks

Output

Building embedding

embeddings = OpenAIEmbeddings()

Create a Pinecone instance for vector database from ‘data’

vectordb = Pinecone.from_documents(text_chunks,embeddings,index_name="demoindex")

Create a retriever for querying the vector database.

retriever = vectordb.as_retriever(score_threshold = 0.7)

Retrieving data from vector database

rdocs = retriever.get_relevant_documents("Cocoa Puffs")
rdocs

Using Prompt and retrieve the data

from langchain.prompts import PromptTemplate

prompt_template = """Given the following context and a question, 
generate an answer based on this context only.
,Please state "I don't know." Don't try to make up an answer.

CONTEXT: {context}

QUESTION: {question}"""


PROMPT = PromptTemplate(
    template=prompt_template, input_variables=["context", "question"]
)
chain_type_kwargs = {"prompt": PROMPT}

from langchain.chains import RetrievalQA

chain = RetrievalQA.from_chain_type(llm=llm,
                            chain_type="stuff",
                            retriever=retriever,
                            input_key="query",
                            return_source_documents=True,
                            chain_type_kwargs=chain_type_kwargs)
                            

Let’s query the data.

chain('Can you please provide cereal recommendation for Kids?')

Output from Query

{'query': 'Can you please provide cereal recommendation for Kids?',
'result': [Document(page_content='name: Crispix\nmfr: K\ntype: C\ncalories: 110\nprotein: 2\nfat: 0\nsodium: 220\nfiber: 1\ncarbo: 21\nsugars: 3\npotass: 30\nvitamins: 25\nshelf: 3\nweight: 1\ncups: 1\nrating: 46.895644\nrecommendation: Kids', metadata={'row': 21.0, 'source': '/content/drive/My Drive/Colab_Notebooks/cereal.csv'}), ..]

Conclusion

Hope you can understand how vector databases work, their components, architecture, and characteristics of Vector Databases in Generative AI solutions . Understand how the vector database is different from traditional database and comparison with conventional database elements. Indeed, the analogy helps you better understand the vector database. Pinecone vector database and indexing steps would help you create a vector database and bring the key for the following code implementation.

Key Takeaways

• Compactable with structured, unstructured, and semi-structured data.
• It adapts embedding techniques and highly indexed features.
• The interactions happen through plain text using a prompt (e.g., English). And data stored in mathematical representations.
• Similarity calibrates in Vector Databases through – Euclidean Distance, Cosine Similarity, and Dot Product.

Frequently Asked Questions

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Shanthababu Pandian 12 Dec 2023