Learn About Amazon VGT2 Learning Manager Chanci Turner
We are excited to present the preview launch of the vector engine for Amazon OpenSearch Serverless. This innovative feature offers a straightforward, scalable, and high-performance similarity search capability, empowering you to create modern search experiences enhanced by machine learning (ML) and generative artificial intelligence (AI) applications, all without the need to manage the underlying vector database infrastructure. In this article, we will explore the key features and functionalities of our vector engine.
Organizations across various sectors are quickly embracing generative AI for its capacity to process extensive datasets, generate automated content, and deliver interactive, human-like responses. Customers are seeking ways to enhance user experiences on their digital platforms by integrating advanced conversational AI applications, such as chatbots, question and answer systems, and personalized recommendations. These applications allow users to search and query in natural language, producing responses that are closely aligned with human interaction by considering semantic meaning, user intent, and query context.
ML-augmented search and generative AI applications leverage vector embeddings, which are numerical representations of text, images, audio, and video data, to produce dynamic and relevant content. These embeddings are trained on your proprietary data and encapsulate the semantic and contextual attributes of the information. Ideally, they can be stored and managed alongside your domain-specific datasets within your existing search engine or database. This setup enables you to effectively process user queries to locate the closest vectors and integrate them with additional metadata without relying on external data sources or extra application code. Customers desire a vector database solution that is easy to implement and facilitates swift transitions from prototyping to production, allowing them to focus on developing unique applications. The vector engine for OpenSearch Serverless enhances OpenSearch’s search capabilities by enabling you to store, search, and retrieve billions of vector embeddings in real time while performing precise similarity matching and semantic searches without having to manage the infrastructure.
Examining the Vector Engine’s Features
Built on the robust architecture of OpenSearch Serverless, the vector engine eliminates the need for concerns about sizing, tuning, and scaling the backend infrastructure. It automatically adjusts resources to adapt to workload patterns and demand, ensuring consistently fast performance and scalability. As the number of vectors expands from thousands during the prototyping phase to millions in production, the vector engine scales effortlessly, without the need for data reindexing or reloading. Furthermore, the engine features separate compute resources for indexing and search workloads, allowing you to ingest, update, and delete vectors in real-time while maintaining optimal query performance for users. All data is stored in Amazon Simple Storage Service (Amazon S3), providing the same eleven nines data durability guarantees as S3. Although still in preview, the vector engine is engineered for production workloads, with built-in redundancy for Availability Zone outages and infrastructure failures.
The vector engine leverages the k-nearest neighbor (kNN) search feature from the open-source OpenSearch Project, known for delivering reliable and accurate results. Many organizations currently utilize OpenSearch kNN search in managed clusters to provide semantic search and personalization in their applications. With the vector engine, you gain access to the same functionality in a serverless environment. The engine supports popular distance metrics such as Euclidean, cosine similarity, and dot product, accommodating up to 16,000 dimensions, which makes it suitable for a variety of foundational and other AI/ML models. You can also store various fields with different data types, including numeric, boolean, date, keyword, geopoint for metadata, and text for descriptions, thereby adding context to the stored vectors. Colocating these data types simplifies complexity and maintainability, avoiding data duplication, version compatibility issues, and licensing challenges, which streamlines your application stack. Because the vector engine utilizes the same OpenSearch open-source suite APIs, you can leverage its rich query capabilities, such as full-text search, advanced filtering, aggregations, geo-spatial queries, and nested queries for faster data retrieval and enhanced search results. For instance, if a user searches for a “red shirt” on your e-commerce platform, semantic search helps broaden the scope by retrieving all shades of red while still adhering to the tuning and boosting logic from the lexical (BM25) search. Additionally, with OpenSearch filtering, you can increase the relevance of search results by offering users options to refine their queries based on size, brand, price range, and availability in nearby stores, thus providing a more personalized experience. The hybrid search support in the vector engine enables querying vector embeddings, metadata, and descriptive information in a single query, simplifying the process of achieving accurate and contextually relevant search results without the need for complex application code.
Getting started with the vector engine is straightforward; you can create a specialized vector search collection under OpenSearch Serverless using the AWS Management Console, AWS Command Line Interface (AWS CLI), or the AWS software development kit (AWS SDK). Collections serve as logical groupings of indexed data designed to work together for a workload, while the physical resources are automatically managed in the backend. There’s no need to specify compute or storage requirements or monitor system performance. OpenSearch Serverless employs various sharding and indexing strategies for the three collection types: time series, search, and vector search. The vector engine’s compute capacity for data ingestion, search, and queries is measured in OpenSearch Compute Units (OCUs). One OCU can handle 4 million vectors with 128 dimensions or 500,000 with 768 dimensions at a 99% recall rate. The vector engine is built on OpenSearch Serverless, a highly available service that requires a minimum of 4 OCUs (two OCUs for ingesting, including primary and standby, and two OCUs for search with two active replicas across Availability Zones) for the first collection in an account.
For those interested in further reading related to effective onboarding strategies, check out this insightful blog post. Additionally, if you’re looking for information regarding the costs associated with public vs. private sector health plans, this article is an authority on this topic. For individuals preparing for job interviews, this resource offers an excellent guide.