Elasticsearch on AWS: 3 Managed Services and How to Get Started
9 min read
How Can You Run Elasticsearch on AWS?
Elasticsearch is a powerful search and analytics engine that allows you to move data from numerous sources, analyze it in near-real time, and visualize it in a variety of charts, tables, and maps. It’s built on Apache Lucene and was first released in 2010 by Elasticsearch N.V. (now known as Elastic).
Elasticsearch is often used for log and event data analysis, and it’s also popular for its full-text search capabilities. It is scalable, secure, and highly available, making it a suitable choice for search-driven applications.
There are three main ways to run Elasticsearch on AWS:
Do it yourself: Elasticsearch is an open source solution, and you can simply install it on virtual machines in Amazon. One way to do this is to use Amazon Machine Images preconfigured with Elasticsearch and deploy them on EC2 instances.
Elastic Cloud: Elastic provides a managed service that lets you run Elasticsearch on several cloud platforms, including AWS. In this scenario, you pay Elastic for the managed service and Amazon for the computing resources.
Amazon OpenSearch: This is a fully managed service by AWS, which lets you deploy, secure, and operate Elasticsearch at scale. However, it is based on OpenSearch, a fork of Elasticsearch 7.10.2, and is not fully compatible with the original Elasticsearch project. In this scenario, there is no cost for the software—you only pay Amazon for the computing resources used by the Elasticsearch cluster.
This is part of a series of articles about ELK Stack.
In this article, you will learn:
Why Should You Run Elasticsearch on AWS?
Running Elasticsearch on AWS offers several benefits, specifically tailored to enhance scalability, reliability, and flexibility in managing search and analytics workloads. Here are the key advantages:
Scalability
AWS provides a highly scalable environment, allowing Elasticsearch clusters to grow or shrink based on demand. This elasticity is vital for handling varying workloads, from small projects to large-scale enterprise applications. AWS’s infrastructure supports automatic scaling for Elasticsearch, enabling it to handle massive volumes of data and high query rates without manual intervention.
Managed Services
With options like Amazon OpenSearch Service (formerly Amazon Elasticsearch Service) and Elastic Cloud on AWS, users can leverage managed services that simplify the operational management of Elasticsearch clusters. These services handle tasks such as provisioning, setup, configuration, patching, and backups automatically, freeing users from the complexities of cluster management and allowing them to focus on their core application logic.
Security
AWS provides robust security features that can be integrated with Elasticsearch, including network isolation using Amazon VPC, encryption at rest using AWS Key Management Service (KMS), and identity and access management with IAM roles and policies. This comprehensive security framework ensures that data is protected at all levels, from physical infrastructure to application access control.
Cost-Effectiveness
Running Elasticsearch on AWS allows for cost optimization based on usage. Users can choose from a variety of instance types and storage options to balance performance and cost. Pay-as-you-go pricing models for AWS resources and managed services mean that users pay only for what they use, optimizing operational expenses.
Integration with AWS Ecosystem
Elasticsearch on AWS benefits from seamless integration with other AWS services, such as Amazon Kinesis for real-time data ingestion, AWS Lambda for serverless computing, and Amazon CloudWatch for monitoring and alerts. This integration enables powerful, scalable, and flexible architectures that can evolve with the needs of any application.
Global Reach and Availability
AWS’s global infrastructure ensures that Elasticsearch deployments can be distributed across multiple geographic locations, enhancing availability and reducing latency for global applications. This global reach, combined with AWS’s reliability and uptime guarantees, ensures that Elasticsearch-backed applications are always available to end-users, regardless of their location.
There are several services available that can help you manage Elasticsearch and deploy it in an AWS environment. The primary ones are:
1. Elastic Cloud
Elastic Cloud is a fully managed service from Elastic, the makers of Elasticsearch. It lets you easily deploy, operate, and scale Elasticsearch and Kibana. It includes all the features of open-source Elasticsearch, with the following additional features:
Automated deployment and scaling: Elastic Cloud simplifies the process of deploying Elasticsearch clusters and Kibana instances. It automatically scales resources based on the workload, ensuring optimal performance and cost-efficiency.
Managed upgrades: Ensures your Elasticsearch and Kibana versions are up-to-date by providing managed upgrades, reducing the maintenance overhead and ensuring access to the latest features and security updates.
Security features: Includes built-in security measures such as encryption at rest and in transit, role-based access control (RBAC), and integration with Elastic’s security features for authentication and authorization.
Monitoring and alerts: Offers comprehensive monitoring capabilities through the Elastic Stack, enabling real-time insights into cluster health and performance. It also supports customizable alerts to notify users of potential issues.
Data backups and recovery: Provides automated snapshots for data backup and facilitates easy recovery, ensuring data durability and resilience.
Multi cloud support: Elastic Cloud is available not only on AWS but also supports Microsoft Azure and Google Cloud.
Support for all Elasticsearch APIs and features: Supports the full spectrum of Elasticsearch APIs and features, including advanced analytics, machine learning, and the latest search capabilities.
Integration with Logstash and Beats: Offers seamless integration with Logstash and Beats for data ingestion, enabling easy data pipeline setup from various sources.
Expert support: Provides access to Elastic’s expert support team, offering guidance and troubleshooting.
2. Amazon OpenSearch Service
Amazon OpenSearch Service is the successor to the now defunct Amazon Elasticsearch (Amazon ES) Service. It offers the following features:
Setup and configuration: Easy setup, with the ability to set the number of instances, instance types, and storage options.
In-place upgrades: Automated upgrades for OpenSearch clusters to newer versions without any.
Event monitoring and alerting: Monitors the data stored in your cluster and automatically sends notifications based on preconfigured thresholds.
Support for multiple query languages: In addition to domain-specific language (DSL), supports SQL and the simplified Piped Processing Language (PPL).
Integration with open-source tools: Offers built-in OpenSearch Dashboards and integrates with Logstash. Also supports the open-source OpenTelemetry standard and Elasticsearch plugins like Kuromoji, Phonetic Analysis, and Ingest Processor.
Security: Enables secure connection from Virtual Private Cloud (Amazon VPC) or through the public internet. Supports authentication using Amazon Cognito or AWS Identity and Access Management (IAM), with granular permissions for indices, documents, or fields.
Compliance: The service is HIPAA-eligible and compliant with PCI DSS, SOC, ISO, and FedRAMP standards.
Due to tensions between Elastic and Amazon, Amazon stopped using the official Elasticsearch distribution, and created OpenSearch, a new open source project derived from Elasticsearch 7.10.2. It includes OpenSearch Dashboards, a visualization and user interface tool similar to Kibana. The main difference from Elasticsearch is the licensing, with OpenSearch maintaining an open-source license, while Elasticsearch has switched to the Server Side Public License (SSPL), which is not a fully open source license.
3. Amazon OpenSearch Serverless
Amazon OpenSearch Serverless is a new addition to the AWS family. It’s a fully managed, serverless search service that automatically scales to handle your workload. This service is suitable for applications with unpredictable or variable workloads, as it can scale up or down instantly in response to demand. It also includes many of the features of Amazon OpenSearch Service.
Tutorial: Getting Started with Amazon OpenSearch Service
Let’s see the high-level steps involved in starting to work with Amazon OpenSearch. The image below illustrates the architecture. Keep in mind that when you use Amazon OpenSearch, you are working with the OpenSearch project, not the official Elasticsearch distribution.
An OpenSearch Service domain is akin to an Elasticsearch cluster and is the hardware, software, and data in your Amazon OpenSearch Service environment. It sets the foundation for your indexing and searching capabilities.
To create a domain, sign in to the AWS Management Console and open the Amazon OpenSearch Service console. From there, choose Create domain. You’ll be prompted to configure your domain by setting the instance type, number of instances, and storage type. It’s important to choose settings that align with your business needs, as these factors will affect your domain’s performance and cost.
Once you’ve configured your domain, you’ll need to set up access policies. These policies determine who can access your domain and what actions they can perform. AWS provides a range of preconfigured policies, or you can create custom policies to better suit your needs. After you’ve set up your access policies, review your domain configuration, confirm, and choose Create. Your domain will then be created and available for use within a few minutes.
Upload Data to Amazon OpenSearch Service for Indexing
Indexing is a process that organizes data into a structure that OpenSearch can search efficiently. It enables you to retrieve data quickly and accurately.
To upload data, you’ll need to use an AWS SDK or the AWS CLI. Both of these tools allow you to send HTTP requests to your domain’s endpoint. These requests should include the data you want to index, formatted as JSON objects. Each object represents a document and includes fields that contain the data you want to search.
Once you’ve uploaded your data, OpenSearch automatically indexes it. It analyzes the data in each field and creates an inverted index, which maps each unique word to all the documents that contain it. This index allows OpenSearch to find relevant documents quickly, even when dealing with large volumes of data.
Search Documents in Amazon OpenSearch Service
OpenSearch provides a powerful search engine that can handle a wide range of queries, from simple full-text searches to more complex analytical queries.
To search your documents, you’ll need to send a GET request to your domain’s search endpoint. This request should include a query, which specifies the conditions that documents must meet to match the search. OpenSearch supports a wide range of query types, so you can construct queries that precisely match your search needs.
After receiving your search request, OpenSearch uses the inverted index to find relevant documents. It then ranks these documents based on their relevance to the query, using a scoring algorithm. The results are then returned to you as a JSON object, which includes the matching documents and metadata about the search.
Delete an Amazon OpenSearch Service Domain
When you no longer need your OpenSearch Service domain, you can delete it to conserve resources. Navigate to the Amazon OpenSearch Service console and choose the domain you want to delete. From the Domains list, select the domain you want to delete and click Delete. You’ll be asked to confirm your decision, as deleting a domain permanently removes all its data and configurations.
Deleting a domain is irreversible, so it’s recommended to back up any data you want to keep before deleting a domain. If you want to reuse the domain’s configurations in the future, consider taking a snapshot of the domain before deleting it.
Observability SaaS – Coralogix
Managing your own ELK stack might be costing you far more than your think with hidden infra costs, dedicated engineering resources and other overhead. If you are looking for a fully managed solution, Coralogix provides full-stack observability with out-of-the-box parsing rules, alerts and dashboards and of course fully customizable view and workflows. On top of this, Coralogix’s unique architecture is not reliant on expensive indexing or hot storage so you can observe all your data for far less cost.
Oh no!We didn’t find anything for “”,
