Add hosts to cluster

Hi @itversity @Sohail

I am getting error while adding hosts to clusters and video with title “Add hosts to cluster” is also not complete.
Below is the error message and screenshot of the error, could you please have a look and help me to get rid of this.

error message: Command Deploy Client Configuration is not currently available for execution.

Hi @itversity @Itversity123

could someone please look into my issue and help me.

Hi @krishnachaitanya

do you have any idea about this error.

Hi @itversity

Could you please look into my error.

Hi @itversity,

Could you please help to get rid of this error.

Where are you getting this issue? Are you setting up the cluster using your virtual machines on your pc or is it being set up on multiple physical nodes?

Hi @itversity,

I am getting this error while setting up the cluster using virtual machines on my PC.

Are you adding for the first time?

HI @itversity,

Nope, prior to this I have also added nodes in cluster but I deleted them due to some issues. I am getting this error when I am trying to add these nodes again.

Hi @itversity.

Nope, prior to this I have also added nodes in cluster but I deleted them due to some issues. I am getting this error when I am trying to add these nodes again.

HI @itversity,

Nope, prior to this I have also added nodes in cluster but I deleted them due to some issues. I am getting this error when I am trying to add these nodes again.

HI @itversity,

Could you please look into my issue and advise to get rid of this error.

Regards,
Shivkant

@Shivkant Could you post recent log details from /var/log/cloudera-scm-agent

Hi @krishnachaitanya,

Thanks for the reply, please find the logs below from file cloudera-scm-agent.log

[root@goli001 cloudera-scm-agent]# grep “17/Mar/2017” cloudera-scm-agent.log
[17/Mar/2017 18:06:40 +0000] 2378 MainThread agent INFO SCM Agent Version: 5.10.0
[17/Mar/2017 18:06:40 +0000] 2378 MainThread agent INFO Agent Protocol Version: 4
[17/Mar/2017 18:06:40 +0000] 2378 MainThread agent INFO Using Host ID: bf12f089-e404-4dda-86b4-08b4773142ae
[17/Mar/2017 18:06:40 +0000] 2378 MainThread agent INFO Using directory: /var/run/cloudera-scm-agent
[17/Mar/2017 18:06:40 +0000] 2378 MainThread agent INFO Using supervisor binary path: /usr/lib64/cmf/agent/build/env/bin/supervisord
[17/Mar/2017 18:06:40 +0000] 2378 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certificates in HTTPS communication. These options can be configured in this agent’s config.ini file to enable certificate validation.
[17/Mar/2017 18:06:40 +0000] 2378 MainThread agent INFO Agent Logging Level: INFO
[17/Mar/2017 18:06:40 +0000] 2378 MainThread agent INFO No command line vars
[17/Mar/2017 18:06:40 +0000] 2378 MainThread agent INFO Missing database jar: /usr/share/java/mysql-connector-java.jar (normal, if you’re not using this database type)
[17/Mar/2017 18:06:40 +0000] 2378 MainThread agent INFO Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you’re not using this database type)
[17/Mar/2017 18:06:40 +0000] 2378 MainThread agent INFO Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
[17/Mar/2017 18:06:40 +0000] 2378 MainThread agent INFO Agent starting as pid 2378 user root(0) group root(0).
[17/Mar/2017 18:06:42 +0000] 2378 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups
[17/Mar/2017 18:06:42 +0000] 2378 MainThread cgroups INFO Found cgroups subsystem: cpu
[17/Mar/2017 18:06:42 +0000] 2378 MainThread cgroups INFO Found cgroups subsystem: cpuacct
[17/Mar/2017 18:06:42 +0000] 2378 MainThread cgroups INFO Found cgroups subsystem: memory
[17/Mar/2017 18:06:42 +0000] 2378 MainThread cgroups INFO Found cgroups subsystem: blkio
[17/Mar/2017 18:06:42 +0000] 2378 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/memory
[17/Mar/2017 18:06:42 +0000] 2378 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpu
[17/Mar/2017 18:06:42 +0000] 2378 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/cpuacct
[17/Mar/2017 18:06:42 +0000] 2378 MainThread cgroups INFO Reusing /var/run/cloudera-scm-agent/cgroups/blkio
[17/Mar/2017 18:06:42 +0000] 2378 MainThread agent INFO Found cgroups capabilities: {‘has_memory’: True, ‘default_memory_limit_in_bytes’: -1, ‘default_memory_soft_limit_in_bytes’: -1, ‘writable_cgroup_dot_procs’: True, ‘default_cpu_rt_runtime_us’: 950000, ‘has_cpu’: True, ‘default_blkio_weight’: 1000, ‘default_cpu_shares’: 1024, ‘has_cpuacct’: True, ‘has_blkio’: True}
[17/Mar/2017 18:06:42 +0000] 2378 MainThread agent INFO Setting up supervisord event monitor.
[17/Mar/2017 18:06:42 +0000] 2378 MainThread filesystem_map INFO Monitored nodev filesystem types: [‘nfs’, ‘nfs4’, ‘tmpfs’]
[17/Mar/2017 18:06:42 +0000] 2378 MainThread filesystem_map INFO Using timeout of 2.000000
[17/Mar/2017 18:06:42 +0000] 2378 MainThread filesystem_map INFO Using join timeout of 0.100000
[17/Mar/2017 18:06:42 +0000] 2378 MainThread filesystem_map INFO Using tolerance of 60.000000
[17/Mar/2017 18:06:42 +0000] 2378 MainThread filesystem_map INFO Local filesystem types whitelist: [‘ext2’, ‘ext3’, ‘ext4’, ‘xfs’]
[17/Mar/2017 18:06:42 +0000] 2378 MainThread kt_renewer INFO Agent wide credential cache set to /var/run/cloudera-scm-agent/krb5cc_cm_agent_0
[17/Mar/2017 18:06:42 +0000] 2378 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000
[17/Mar/2017 18:06:42 +0000] 2378 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000
[17/Mar/2017 18:06:42 +0000] 2378 MainThread agent INFO Using max_collection_wait_seconds of 10.000000
[17/Mar/2017 18:06:42 +0000] 2378 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.10.0-py2.6.egg/cmf/monitor/tasktracker/schema.json
[17/Mar/2017 18:06:42 +0000] 2378 MainThread ntp_monitor INFO Using timeout of 2.000000
[17/Mar/2017 18:06:42 +0000] 2378 MainThread dns_names INFO Using timeout of 30.000000
[17/Mar/2017 18:06:42 +0000] 2378 MainThread init INFO Created DNS monitor.
[17/Mar/2017 18:06:42 +0000] 2378 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880
[17/Mar/2017 18:06:42 +0000] 2378 MainThread init INFO Importing metric schema from file /usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.10.0-py2.6.egg/cmf/monitor/schema.json
[17/Mar/2017 18:06:43 +0000] 2378 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor’s env): {‘CDH_PARQUET_HOME’: ‘/usr/lib/parquet’, ‘JSVC_HOME’: ‘/usr/libexec/bigtop-utils’, ‘CMF_PACKAGE_DIR’: ‘/usr/lib64/cmf/service’, ‘CDH_HADOOP_BIN’: ‘/usr/bin/hadoop’, ‘MGMT_HOME’: ‘/usr/share/cmf’, ‘CDH_IMPALA_HOME’: ‘/usr/lib/impala’, ‘CDH_YARN_HOME’: ‘/usr/lib/hadoop-yarn’, ‘CDH_HDFS_HOME’: ‘/usr/lib/hadoop-hdfs’, ‘PATH’: ‘/sbin:/usr/sbin:/bin:/usr/bin’, ‘CDH_HUE_PLUGINS_HOME’: ‘/usr/lib/hadoop’, ‘CM_STATUS_CODES’: u’STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE’, ‘KEYTRUSTEE_KP_HOME’: ‘/usr/share/keytrustee-keyprovider’, ‘CLOUDERA_ORACLE_CONNECTOR_JAR’: ‘/usr/share/java/oracle-connector-java.jar’, ‘CDH_SQOOP2_HOME’: ‘/usr/lib/sqoop2’, ‘KEYTRUSTEE_SERVER_HOME’: ‘/usr/lib/keytrustee-server’, ‘CDH_MR2_HOME’: ‘/usr/lib/hadoop-mapreduce’, ‘HIVE_DEFAULT_XML’: ‘/etc/hive/conf.dist/hive-default.xml’, ‘CLOUDERA_POSTGRESQL_JDBC_JAR’: ‘/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar’, ‘CDH_KMS_HOME’: ‘/usr/lib/hadoop-kms’, ‘CDH_HBASE_HOME’: ‘/usr/lib/hbase’, ‘CDH_SQOOP_HOME’: ‘/usr/lib/sqoop’, ‘WEBHCAT_DEFAULT_XML’: ‘/etc/hive-webhcat/conf.dist/webhcat-default.xml’, ‘CDH_OOZIE_HOME’: ‘/usr/lib/oozie’, ‘CDH_ZOOKEEPER_HOME’: ‘/usr/lib/zookeeper’, ‘CDH_HUE_HOME’: ‘/usr/lib/hue’, ‘CLOUDERA_MYSQL_CONNECTOR_JAR’: ‘/usr/share/java/mysql-connector-java.jar’, ‘CDH_HBASE_INDEXER_HOME’: ‘/usr/lib/hbase-solr’, ‘CDH_MR1_HOME’: ‘/usr/lib/hadoop-0.20-mapreduce’, ‘CDH_SOLR_HOME’: ‘/usr/lib/solr’, ‘CDH_PIG_HOME’: ‘/usr/lib/pig’, ‘CDH_SENTRY_HOME’: ‘/usr/lib/sentry’, ‘CDH_CRUNCH_HOME’: ‘/usr/lib/crunch’, ‘CDH_LLAMA_HOME’: ‘/usr/lib/llama/’, ‘CDH_HTTPFS_HOME’: ‘/usr/lib/hadoop-httpfs’, ‘CDH_HADOOP_HOME’: ‘/usr/lib/hadoop’, ‘CDH_HIVE_HOME’: ‘/usr/lib/hive’, ‘ORACLE_HOME’: ‘/usr/share/oracle/instantclient’, ‘CDH_HCAT_HOME’: ‘/usr/lib/hive-hcatalog’, ‘CDH_KAFKA_HOME’: ‘/usr/lib/kafka’, ‘CDH_SPARK_HOME’: ‘/usr/lib/spark’, ‘TOMCAT_HOME’: ‘/usr/lib/bigtop-tomcat’, ‘CDH_FLUME_HOME’: ‘/usr/lib/flume-ng’}
[17/Mar/2017 18:06:43 +0000] 2378 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[17/Mar/2017 18:06:43 +0000] 2378 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
[17/Mar/2017 18:06:43 +0000] 2378 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor
[17/Mar/2017 18:06:43 +0000] 2378 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/flood
[17/Mar/2017 18:06:43 +0000] 2378 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
[17/Mar/2017 18:06:43 +0000] 2378 MainThread agent ERROR Failed to connect to previous supervisor.
[17/Mar/2017 18:06:43 +0000] 2378 MainThread tmpfs INFO Successfully mounted tmpfs at /var/run/cloudera-scm-agent/process
[17/Mar/2017 18:06:44 +0000] 2378 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 1)
[17/Mar/2017 18:06:45 +0000] 2378 MainThread agent INFO Supervisor version: 3.0, pid: 2458
[17/Mar/2017 18:06:45 +0000] 2378 MainThread agent INFO Successfully connected to supervisor
[17/Mar/2017 18:06:45 +0000] 2378 MainThread status_server INFO Using maximum impala profile bundle size of 1073741824 bytes.
[17/Mar/2017 18:06:45 +0000] 2378 MainThread status_server INFO Using maximum stacks log bundle size of 1073741824 bytes.
[17/Mar/2017 18:06:45 +0000] 2378 MainThread _cplogging INFO [17/Mar/2017:18:06:45] ENGINE Bus STARTING
[17/Mar/2017 18:06:45 +0000] 2378 MainThread _cplogging INFO [17/Mar/2017:18:06:45] ENGINE Started monitor thread ‘_TimeoutMonitor’.
[17/Mar/2017 18:06:45 +0000] 2378 MainThread _cplogging INFO [17/Mar/2017:18:06:45] ENGINE Serving on goli001.nr.com:9000
[17/Mar/2017 18:06:45 +0000] 2378 MainThread _cplogging INFO [17/Mar/2017:18:06:45] ENGINE Bus STARTED
[17/Mar/2017 18:06:45 +0000] 2378 MainThread init INFO New monitor: (<cmf.monitor.host.HostMonitor object at 0x29ad4d0>,)
[17/Mar/2017 18:06:45 +0000] 2378 MainThread agent INFO Setting default socket timeout to 30
[17/Mar/2017 18:06:45 +0000] 2378 MonitorDaemon-Scheduler init INFO Monitor ready to report: (‘HostMonitor’,)
[17/Mar/2017 18:06:45 +0000] 2378 MainThread agent ERROR Heartbeating to 192.168.70.9:7182 failed.
[17/Mar/2017 18:06:45 +0000] 2378 MainThread agent INFO Loading last saved hb response to complete initialization: /var/lib/cloudera-scm-agent/response.avro
[17/Mar/2017 18:06:45 +0000] 2378 MainThread heartbeat_tracker INFO HB stats (seconds): num:1 LIFE_MIN:0.01 min:0.01 mean:0.01 max:0.01 LIFE_MAX:0.01
[17/Mar/2017 18:06:45 +0000] 2378 MainThread agent INFO CM server guid: 6a701a83-d347-496c-871d-679fbc007423
[17/Mar/2017 18:06:45 +0000] 2378 MainThread agent INFO Using parcels directory from server provided value: /opt/cloudera/parcels
[17/Mar/2017 18:06:45 +0000] 2378 MainThread parcel INFO Agent does create users/groups and apply file permissions
[17/Mar/2017 18:06:45 +0000] 2378 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
[17/Mar/2017 18:06:45 +0000] 2378 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
[17/Mar/2017 18:06:46 +0000] 2378 MainThread agent INFO Flood daemon (re)start attempt
[17/Mar/2017 18:06:46 +0000] 2378 MainThread agent INFO Triggering supervisord update.
[17/Mar/2017 18:06:46 +0000] 2378 MainThread downloader ERROR Failed rack peer update: [Errno 111] Connection refused
[17/Mar/2017 18:06:46 +0000] 2378 MainThread agent INFO Active parcel list updated; recalculating component info.
[17/Mar/2017 18:06:46 +0000] 2378 MainThread throttling_logger INFO Identified java component java7 with full version JAVA_HOME=/usr/lib/jvm/jre-openjdk java version “1.7.0_131” OpenJDK Runtime Environment (rhel-2.6.9.0.el6_8-x86_64 u131-b00) OpenJDK 64-Bit Server VM (build 24.131-b00, mixed mode) for requested version misc.
[17/Mar/2017 18:06:46 +0000] 2378 MainThread throttling_logger INFO Identified java component java7 with full version JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera java version “1.7.0_67” Java™ SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot™ 64-Bit Server VM (build 24.65-b04, mixed mode) for requested version 7.
[17/Mar/2017 18:06:47 +0000] 2378 MainThread throttling_logger INFO Identified java component java8 with full version JAVA_HOME=/usr/java/jdk1.8.0_77 java version “1.8.0_77” Java™ SE Runtime Environment (build 1.8.0_77-b03) Java HotSpot™ 64-Bit Server VM (build 25.77-b03, mixed mode) for requested version 8.
[17/Mar/2017 18:06:47 +0000] 2378 MainThread process WARNING New process is not running: {u’refresh_files’: [], u’config_generation’: 0, u’auto_restart’: False, u’running’: False, u’required_tags’: [], u’one_off’: True, u’special_file_info’: [], u’group’: u’root’, u’id’: 19, u’status_links’: {}, u’name’: u’cluster-host-inspector’, u’extra_groups’: [], u’run_generation’: 2, u’start_timeout_seconds’: None, u’optional_tags’: [], u’program’: u’mgmt/mgmt.sh’, u’arguments’: [u’inspector’, u’input.json’, u’output.json’, u’DEFAULT’], u’parcels’: {}, u’resources’: [], u’user’: u’root’}
[17/Mar/2017 18:06:47 +0000] 2378 MainThread util INFO Using generic audit plugin for process cluster-host-inspector
[17/Mar/2017 18:06:47 +0000] 2378 MainThread util INFO Creating metadata plugin for process cluster-host-inspector
[17/Mar/2017 18:06:47 +0000] 2378 MainThread util INFO Using specific metadata plugin for process cluster-host-inspector
[17/Mar/2017 18:06:47 +0000] 2378 MainThread util INFO Using generic metadata plugin for process cluster-host-inspector
[17/Mar/2017 18:06:47 +0000] 2378 MainThread agent INFO [19-cluster-host-inspector] Instantiating process
[17/Mar/2017 18:06:47 +0000] 2378 MainThread process INFO [19-cluster-host-inspector] Updating process: False {}
[17/Mar/2017 18:06:47 +0000] 2378 MainThread agent INFO Triggering supervisord update.
[17/Mar/2017 18:07:15 +0000] 2378 DnsResolutionMonitor throttling_logger INFO Using java location: ‘/usr/java/jdk1.7.0_67-cloudera/bin/java’.

Hi @krishnachaitanya

Could you please look into the error message from logs shared above.

Some one have to share screen with you and troubleshoot the issue. As of now, I do not have bandwidth to do so. I am busy with 2 trainings as well as supporting the developer lab.

Sorry for being late in responding to your questions.

Hi @krishnachaitanya

Did you get a chance to look into my error.

Hi @krishnachaitanya

could you please look into the error message and advise.

Sorry for the late reply, like @itversity said, you have to share screen and troubleshoot the issues.
Meanwhile, are you able to ssh without password to the new machine…?

MainThread downloader ERROR Failed rack peer update: [Errno 111] Connection refused

Hi @krishnachaitanya,

Thanks for the reply, I can share my screen via team viewer.

Yes, I can ssh without password.