Running Elasticsearch, Logstash, and Kibana on Kubernetes with Helm
Kubernetes (or “K8s”) is an open-source container orchestration tool developed by Google. In this tutorial, we will be leveraging the power of Kubernetes to look at…
Our next-gen architecture is built to help you make sense of your ever-growing data Watch a 4-min demo video!
Formats: PNG, PDF, and SVG
Files size: 2.8 MB
For brand guidelines, please click here
Kibana is the most popular open-source analytics and visualization platform designed to offer faster and better insights into your data. It is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elasticsearch clusters.
An Elasticsearch cluster contains many moving parts. These clusters need modern authentication mechanisms and they require security controls to be configured to prevent unauthorized access. To prevent unauthorized access to your Elasticsearch cluster, you must have a way to authenticate users.
Using SAML (Security Assertion Mark-up Language) authentication is one of them. SAML authentication allows users to log in to Kibana with an external Identity Provider, such as Google using OAuth as the authentication protocol. The current version is OAuth2.
OAuth2 works by delegating user authentication to the service that hosts a user account and authorizing third-party applications to access that user account. It provides authorization flows for web and desktop applications, and mobile devices.
This diagram shows the basic flow of how the authorization is applied.
To follow is a functional guide on how to use an OAuth authentication system like Google Sign-In to log in to Kibana.
The following assumptions are to be made:
The Elastic Stack supports SAML SSO (single sign-on) into Kibana, using Elasticsearch as a backend service. To enable SAML single-sign-on, you require the Identity Provider, which acts as the service that handles your credentials (such as Google log-in details) and performs that actual authentication of users. This means providing Elasticsearch with information about the OAuth Identity Provider and registering the Elastic Stack as a known Service Provider within the same Identity Provider. A few configuration changes are required in Kibana to activate the SAML authentication provider. Once you enable SAML authentication in Kibana it will affect all users who try to log in.
You must edit your cluster configuration, to point to the SAML Identity Provider before you can complete the configuration in Kibana.
Before any messages can be exchanged between an Identity Provider and a Service Provider, you need to know the configuration details and capabilities of the other. This configuration and the capabilities are encoded in an XML document, that is the SAML Metadata of the SAML entity. This is known as a SAML realm used within the authentication chain for Elasticsearch. It will define the capabilities and features of your identity provider such as Google.
Most Identity Providers such as Google will provide an appropriate metadata file with all the features that the Elastic Stack requires.
Assuming that your Identity Provider is up and running, there are only two prerequisites: TLS for the HTTP layer and the Token Service needs to be enabled in Elasticsearch.
You create a realm by adding the following to your elasticsearch.yml configuration file.
The following configuration example represents a minimal functional configuration. This configuration creates a SAML authentication realm with a saml-realm name of saml1.
xpack.security.authc.realms.saml1: type: saml order: 0 idp.metadata.path: "saml/idp-external.xml" idp.entity_id: "https://sso.example.com/" sp.entity_id: "https://kibana.example.com/" sp.acs:<a href="https://kibana.example.com/api/security/v1/saml" "https://kibana.example.com/api/security/v1/saml" sp.logout: "https://kibana.example.com/logout" attributes.principal: "nameid:persistent"
Where ‘kibana.example.com‘ is represented in the SAML authentication realm, it actually defines the <KIBANA_ENDPOINT_URL>
You can retrieve your <KIBANA_ENDPOINT_URL> from the Elasticsearch Service Console by clicking on the Kibana Copy endpoint in the details of your deployment.
The key configuration properties of a SAML authentication realm consist of:
|Property Name||Property Description|
|Order||This represents the order of the SAML realm in your authentication chain. Allowed values are between 2 and 100. Set to 2 unless you plan on configuring multiple SSO realms for this cluster.|
|idp.metadata.path||This is the file path or the HTTPS URL where your Identity Provider metadata is available, such as https://idpurl.com/sso/saml/metadata. The path that you enter here is relative to your config directory. Elasticsearch will automatically monitor this file for changes and will reload the configuration whenever it is updated.|
|idp.entity_id||This is the SAML EntityID of your Identity Provider. This can be read from the configuration page of the Identity Provider, or its SAML metadata, such as https://idpurl.com/entity_id. It should match the EntityID attribute within the metadata file.|
|sp.entity_id||This is the SAML EntityID of the Service Provider. This can be any URI but it’s good practice to set it to the URL where Kibana is accepting connections. You will use this value when you add Kibana as a service provider within your Identity Provider. It is recommended that you use the base URL for your Kibana instance as the entity ID.|
|sp.acs||This is the Assertion Consumer Service (ACS) URL where Kibana is listening for incoming SAML messages. This ACS endpoint supports the SAML HTTP-POST binding only. It must be a URL that is accessible from the web browser of the user who is attempting to log in to Kibana. It does not need to be directly accessible by Elasticsearch or the Identity Provider. This should be the /api/security/v1/saml endpoint of your Kibana server.|
|sp.logout||This is the SingleLogout endpoint where the Service Provider is listening for incoming SAML LogoutResponse and LogoutRequest messages. This is the URL within Kibana that accepts logout messages from the Identity Provider. This should be the /logout endpoint of your Kibana server.|
|attributes.principal||This defines which SAML attribute is going to be mapped to the principal (username) of the authenticated user in Kibana. You would use the special nameid:persistent which will map the NameID with urn:oasis:names:tc:SAML:2.0:nameid-format:persistent format from the Subject of the SAML Assertion. You can use any SAML attribute that carries the necessary value for your use case in this setting.|
To identify the URL of where your Identity Provider metadata is, it is best to reference the admin console of that Identity Provider. This can provide key configuration settings including the SAML EntityID.
Specific to the Google Identity Provider, the following steps can be followed:
This information will provide the necessary details for the SAML authentication realm. In the preceding steps, further actions or next steps can be taken by adding the <KIBANA_ENDPOINT_URL> to the URL and ID configurations in the Service Provider Details screen.
On completion of these steps, this would enable a SAML application for Kibana using the Google Identity Provider. User Access will still need to be activated from the Google Admin console and select the created application.
Elasticsearch Service supports most of the standard Kibana and X-Pack settings. Through a YAML editor in the console, you can append Kibana properties to the kibana.yml file. Your changes to the configuration file are read on startup.
SAML authentication in Kibana requires a small number of additional settings in addition to the standard Kibana security configuration. In particular, your Elasticsearch nodes will have been configured to use TLS on the HTTP interface, so you must configure Kibana to use a HTTPS URL to connect to Elasticsearch. SAML authentication in Kibana is also subject to the xpack.security.sessionTimeout setting and you may wish to adjust this timeout to meet your local needs.
If you are using a Kibana instance of version 7.7 or later add to the configuration file:
saml.saml1 order: 0 realm: saml-realm-name</em></p>
This configuration disables all other realms and only allows users to authenticate with SAML. If you wish to allow your native realm users to authenticate, you need to also enable the basic provider like this:
xpack.security.authc.providers: saml.saml1: order: 0 realm: saml-realm-name description: "Log in with my SAML" basic.basic1: order: 1
This might come in handy for administrative access and as a fallback authentication mechanism, in case the Identity Provider is unresponsive.
If you are using a Kibana instance between versions 7.3 – 7.6 then add to the configuration file:
xpack.security.authc.providers: [saml] server.csrf.whitelist: [/api/security/v1/saml] xpack.security.authc.saml.realm: saml-realm-name
This configuration disables all other realms and only allows users to authenticate with SAML. If you wish to allow your native realm users to authenticate, you need to also enable the basic provider by setting xpack.security.authc.providers: [saml, basic] in the configuration of Kibana.
If you are using a Kibana instance of version 7.2 then add to the configuration file:
xpack.security.authProviders: [saml] server.xsrf.whitelist: [/api/security/v1/saml] xpack.security.public: protocol: https hostname: [hostname from your Kibana endpoint URL] port: 9243
The cloud endpoint port: 9243 is the port on which Elasticsearch and Kibana are listening.
This configuration will also disable all other realms and only allows users to authenticate with SAML. If you wish to allow your native realm users to authenticate, you can enable the basic authProvider by setting xpack.security.authProviders: [saml, basic] in the configuration of Kibana.
If you wish to have multiple Kibana instances that authenticate against the same Elasticsearch cluster, then each Kibana instance that is configured for SAML authentication requires its own SAML realm.
Each SAML realm must have its own unique Entity ID (sp.entity_id), and its own Assertion Consumer Service (sp.acs). Each Kibana instance will be mapped to the correct realm by looking up the matching sp.acs value.
These realms may use the same Identity Provider but are not required to.
The following is an example of having 2 different Kibana instances, 1 of which uses the same internal Identity Provider, and another which uses a different external Identity Provider (such as Google).
xpack.security.authc.realms.saml.saml_marketing: order: 2 idp.metadata.path: "saml/idp-metadata.xml" idp.entity_id: "https://sso.example.com/" sp.entity_id: "https://kibana.marketing.example.com/" sp.acs: "https://kibana.marketing.example.com/api/security/v1/saml" sp.logout: "https://kibana.marketing.example.com/logout" attributes.principal: "nameid:persistent" xpack.security.authc.realms.saml.saml_eng:
order: 3 idp.metadata.path: "saml/idp-external.xml" idp.entity_id: "https://engineering.sso.example.com/" sp.entity_id: "https://kibana.engineering.example.com/" sp.acs: "https://kibana.engineering.example.com/api/security/v1/saml" sp.logout: "https://kibana.engineering.example.com/logout" attributes.principal: "nameid:persistent"
It is possible to have one or more Kibana instances that use SAML, while other instances use basic authentication against another realm type.
What we’ve been through is using Google to log into your Kibana instance. The Elastic Stack security features now support user authentication, using techniques such as SAML SSO for external identity providers, including OAuth. It is a protocol specifically designed to support authentication via an interactive web browser. There are now Kibana and Elasticsearch security features that work together to enable interactive SAML sessions, and enable you to log in with your Google account.
Being able to log into Kibana with a Google sign-in, will enable you to analyze data source logs in the form of line graphs, bar graphs, pie charts, heat maps, region maps, coordinate maps, gauges, goals, timelion, etc. These visualizations will make it easy to predict or to see the changes in trends of errors or other significant events of the input source.