A database that help index and search at blazing speed.
Relational databases (like MySQL) or No-SQL databases (AWS DynamoDB) can store structured or even semi-structured data but there is one inherent problem.
Unstructured data is hard to store in relational databases.
Contents:
- Introduction
- LLMs 101
- Problem
- New Database?!
- Options (Free vs Paid)
- Closing
Introduction
In today’s increasingly globalized world, the ability to communicate in multiple languages has become a highly valuable skill. Language Models (LLMs) have revolutionized the field of natural language processing, bringing unprecedented advancements in understanding and generating human-like text. Developed through the fusion of deep learning techniques and vast amounts of training data, LLMs, such as OpenAI’s GPT-3.5, have become the cornerstone of various applications, ranging from chatbots and virtual assistants to content generation and language translation. These powerful models possess an inherent ability to grasp context, generate coherent responses, and adapt to diverse linguistic patterns.
LLMs are unique in their ability to generate high-quality text that is often indistinguishable from that of a human.
LLMs — 101
What is LLM?
A massive, general purpose neural network (of type transformer), pre-trained on large amounts of text.
Some well known LLMs:
- Generative Pre-trained Transformer(GPT) [1,2,3,4 generations]
- BLOOM by HuggingFace
- Codex by OpenAI
- BloombergGPT — for financial domain
Some LLM providers include: OpenAI, Cohere, HuggingFace, Google, Amazon.
Use Cases of LLMs:
Language Models (LLMs) have a wide range of use cases across various industries. Here are some common applications:
1. Chatbots and Virtual Assistants: LLMs can power chatbots and virtual assistants, enabling natural language conversation with users. They can understand queries, provide information, and offer personalized recommendations.
2. Content Generation: LLMs can generate human-like text, making them useful for content creation in areas such as writing articles, product descriptions, social media posts, and more. They can save time and effort for content creators.
3. Language Translation: LLMs can aid in language translation by providing accurate and contextually relevant translations. They can help bridge language barriers and facilitate communication across different languages.
4. Sentiment Analysis: LLMs can analyze and understand the sentiment expressed in text, helping businesses gauge public opinion, perform market research, and monitor customer feedback.
5. Text Summarization: LLMs can condense lengthy pieces of text into concise summaries, making it easier to extract key information from articles, research papers, or news stories.
6. Question Answering Systems: LLMs can be trained to answer questions based on a given context. They can be used in educational platforms, customer support systems, or as information resources.
7. Content Filtering and Moderation: LLMs can assist in content filtering and moderation by automatically identifying and flagging inappropriate or offensive content.
Limitations of LLMs:
While Language Models (LLMs) have made remarkable advancements, they do come with certain limitations. Here are a few notable ones:
1. Contextual Understanding: LLMs primarily rely on patterns in the data they were trained on and might struggle with understanding context beyond a few sentences. They may generate responses that are contextually incorrect or misleading.
2. Biases and Inaccuracies: LLMs can inherit biases present in the training data, leading to biased or unfair outputs. Additionally, they may generate factually incorrect information, as they do not possess real-time knowledge verification.
3. Sensitivity to Input Phrasing: LLMs can be sensitive to slight changes in input phrasing, resulting in varying responses. This makes them less robust and consistent in providing accurate answers.
4. Ethical and Misuse Concerns: LLMs can be misused to generate deceptive content, misinformation, or spam. There are concerns about the ethical implications of using LLMs for impersonation, generating harmful content, or amplifying existing biases.
5. Comprehension of Complex Topics: LLMs might struggle to comprehend highly specialized or technical subjects that require deep domain knowledge. They may provide inaccurate or superficial responses in such cases.
6. Over-Reliance on Training Data: LLMs are heavily dependent on the quality and diversity of the training data they receive. Inadequate or biased training data can limit their ability to generalize and generate appropriate responses.
Challenges of LLMs:
- Require months of training and huge computing power
- managing context
- ability to persist context longer time [example: conversation memory]
- search across larger data collections
Problem:
While relational databases are good at storing and retrieving well structured data, or maybe even some semi-structured data, but there is one inherent problem — that is , it’s hard to store unstructured data and retrieve at scale.
For example: Below image depicts how we would store student course database.
A new database — Vector Database
A vector DB indexes and searches vector embeddings for faster and similarity search.
Another definition:
A vector DB allow us to search across unstructured data by their content.
- Images
- Audio
- Video
A vector database serves as a storage and retrieval system specifically designed to handle vector representations of data. The purpose of a vector database is to efficiently store, index, and retrieve vectors, enabling fast similarity searches and other operations on the vector data. Here are a few key purposes and benefits of a vector database:
1. Similarity Search: Vector databases excel at performing similarity searches, where you can query for vectors that are similar to a given vector. This is particularly useful in applications such as image search, recommendation systems, and content-based retrieval.
2. High-Dimensional Data: Vector databases are designed to handle high-dimensional data efficiently. Traditional databases may struggle with indexing and querying high-dimensional vectors, but vector databases utilize specialized indexing structures and algorithms tailored for such data.
3. Fast Indexing and Retrieval: Vector databases employ indexing techniques, such as k-d trees, ball trees, or locality-sensitive hashing (LSH), to organize and structure the vector data for efficient search operations. These indexing structures enable fast retrieval of similar vectors based on distance metrics.
4. Scalability: Vector databases are built to handle large-scale datasets. They are designed with scalability in mind, allowing for efficient storage and retrieval of massive collections of vectors.
5. Versatility: Vector databases can accommodate a wide range of vector data types, including numerical vectors, embeddings, textual representations, or even complex multi-modal data. They provide flexibility in working with diverse vector representations.
6. Integration with ML Systems: Vector databases often integrate seamlessly with machine learning pipelines and workflows. They can be used to store and query learned embeddings, enabling efficient training, inference, and retrieval in ML applications.
7. Real-Time Applications: Vector databases are suited for real-time applications that require fast retrieval of similar vectors, such as real-time recommendations, fraud detection, anomaly detection, or image recognition in video streams.
By leveraging specialized indexing structures and optimized algorithms, vector databases simplify the process of working with high-dimensional vector data and facilitate efficient searching and retrieval operations, enabling various applications across domains.
Vector DB Options
Open Source (free):
- Chroma DB
- FAISS — by Facebook
- Postgres
- Milvus
- Weaviate
Closed Source (paid):
- Pinecone
- Elastic Search
- Azure Cognitive Search
- AWS Open Search
- GCP — Vertex AI — Matching Engine
Real-Time Applications of Vector DB:
- Similarity search
- Recommendation engines
- Topic clustering
- Knowledge Base engines
- and in any real-time system that require fast retrieval of similar entities.
In other words — We can think of Vector DB as an appendix for contexts that LLMs use as their memory.
Closing:
In this post, we learned about what are LLMs and their limitations at a high level. Also we explored one of the challenges of LLMs i.e maintaining context memory via stored contexts in a DB using vector embeddings.
