Enable Kerberos on Hadoop and Spark Cluster using Cloudera Manager

Kerberos Architecture

Let us understand the Kerberos Architecture. Here is the article which will give detailed overview of Kerberos Architecture. Here is the diagram for your reference from that article.

Key Terms

Here are some of the key terms you need to be aware of.

  • krb5 - Kerberos 5
  • KDC - Key Distribution Center
  • PAM - Pluggable Authentication Modules
  • Realm - A Kerberos realm is the domain over which a Kerberos authentication server has the authority to authenticate a user, host or service.
  • Client - Which can authenticate using KDC

Components in KDC

Here are the key components of KDC.

  • An authentication server that performs the initial authentication and issues ticket-granting tickets for users.
  • A ticket granting server that issues service tickets that are based on the initial ticket-granting tickets.
  • A principals database of secret keys for all the users and services that it maintains.

Setup KDC

Let us setup KDC on a server kdc.c.itversity-resources-207518.internal.

Install KDC

Let us install all the libraries that are required to Kerberos on Centos 7.

yum -y install krb5-server krb5-libs krb5-workstation pam_krb5

Configure KDC Components

Let us go ahead and configure KDC Server Components.

  • Location for Configuration Files - /var/kerberos/krb5kdc/
  • Configure kdc.conf for krb5kdc
kdc_ports = 88
kdc_tcp_ports = 88

  #master_key_type = aes256-cts
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
  • Configure kadm5.acl

Configure KDC Server as Client

We need to configure KDC Server as Client as well.

  • Configuration file for client is /etc/krb5.conf
  • Make sure to review [realms] and [domain_realm] as per your organization
# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
default_realm = ITVERSITY.COM
default_ccache_name = KEYRING:persistent:%{uid}

kdc = kdc.c.itversity-resources-207518.internal
admin_server = kdc.c.itversity-resources-207518.internal

.c.itversity-resources-207518.internal = ITVERSITY.COM
c.itversity-resources-207518.internal = ITVERSITY.COM

Create Database

sudo kdb5_util create -s -r ITVERSITY.COM

Start and Enable Kerberos

As KDC Components are setup let us go ahead and start them. We will also enable them to run on startup.

sudo systemctl start krb5kdc kadmin
sudo systemctl enable krb5kdc kadmin
sudo systemctl status krb5kdc
sudo systemctl status kadmin

Validate KDC

As setup is complete, let’s go ahead and validate. We will be configuring SSH Service to be authenticated via Kerberos.

  • Create Principals
  • Allow Kerberos Authentication to the Server
  • Configure SSH Client
  • Validate by using SSH

Create Principals

We can create principals for both users as well hosts using kadmin.local

  • Users (using Password authentication)
addprinc root/admin
addprinc itversity/admin
  • Hosts (using Random key for authentication)
addprinc -randkey host/kdc.c.itversity-resources-207518.internal
ktadd host/kdc.c.itversity-resources-207518.internal

Allow Kerberos Authentication to the Server

From the command line we can allow Kerberos authentication either using authconfig-tui or simply with the command:

sudo authconfig  --enablekrb5 --update

This will make the changes to the PAM configuration.

Configure SSH Client

On the Kerberos client server we need to edit the SSH Client file to allow all clients by default to use Kerberos authentication. To do this, edit the file /etc/ssh/ssh_config and add lines:

GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes

Reload SSH at this stage to be certain everything is in place:

sudo systemctl reload sshd

If firewall is in place you need to open port 88 on TCP and UDP. In our environment all the ports are open using Private Subnets.

Validate KDC Setup

As both server components and clients setup and configuration are done as well as SSH is configured as service, now it is time for us to validate the setup.

  • Make sure you are logged in as user for which principal is created.
  • Try to run SSH to the same user without pointing to private key. It will fail.
  • Run kinit command and enter the password generated while creating principal for this user.
  • Run klist command to validate whether ticket is generated or not.
  • Now try connecting using SSH to the same user and you should be login without any issue.
  • It means setup of Kerberos is done properly.

Prepare CDH Nodes

Let us go ahead and prepare CDH Nodes to enable Kerberos on the cluster. We need to install required libraries on all hosts in the cluster.

  • Install Kerberos Workstation on all nodes in the cluster
ansible all \
  -i hosts \
  -a "yum -y install krb5-libs krb5-workstation" \
  --private-key=~/.ssh/google_compute_engine --become
  • Install JCE Policy File for AES-256 Encryption
ansible all \
  -i hosts \
  -a "yum -y install wget unzip" \
  --private-key=~/.ssh/google_compute_engine --become

ansible all \
  -i hosts \
  -a "wget --no-check-certificate -c --header 'Cookie: oraclelicense=accept-securebackup-cookie' http://download.oracle.com/otn-pub/java/jce/7/UnlimitedJCEPolicyJDK7.zip" \

ansible all \
  -i hosts \
  -a "unzip UnlimitedJCEPolicyJDK7.zip" \
  --private-key=~/.ssh/google_compute_engine --become

ansible all \
  -i hosts \
  -a "sudo cp /home/itversity/UnlimitedJCEPolicy/local_policy.jar /usr/java/jdk1.8.0_221-amd64/jre/lib/security/
" \
  --private-key=~/.ssh/google_compute_engine --become

ansible all \
  -i hosts \
  -a "sudo cp /home/itversity/UnlimitedJCEPolicy/US_export_policy.jar /usr/java/jdk1.8.0_221-amd64/jre/lib/security/
" \
  --private-key=~/.ssh/google_compute_engine --become
  • Copy /etc/krb5.conf to all the hosts
ansible all \
  -i hosts \
  -m synchronize \
  -a "src=etc/krb5.conf dest=/etc" \
  -u root \
  --become \

Create Principals for CM Host and for user cloudera-scm

Let us go ahead and create required principals for Cloudera Manager Host and for user cloudera-scm.
  • Create principal for user cloudera-scm on KDC
addprinc -randkey cloudera-scm/admin@ITVERSITY.COM
xst -k cmf.keytab cloudera-scm/admin@ITVERSITY.COM
  • Create principal for host bigdataserver-1
addprinc -randkey host/bigdataserver-1.c.itversity-resources-207518.internal
ktadd host/bigdataserver-1.c.itversity-resources-207518.internal
  • Copy generated keytab file to /etc/cloudera-scm-server/
  • Create a file named cmf.principal under the folder /etc/cloudera-scm-server/ and add the following line to the file:
  • Change the permissions on the file to cloudera-scm
chown cloudera-scm:cloudera-scm /etc/cloudera-scm-server/cmf.keytab

Now use Cloudera Manager as demonstrated to configure Kerberos.