Pig DEFAULT_PARALLEL


#1

i can not use set default_parallel in pig.
i wrote below code.
pig -x tez -useHCatalog
set default_parallel 5;
a = load ‘demo.orders’ USING org.apache.hive.hcatalog.pig.HCatLoader();
STORE a INTO ‘/user/replymarket/manish’ USING PigStorage(’,’);

PLEASE HELP NEXT WEEK I AM GIVING EXAM OF HDPCD.


#2

Hello,
Please elaborate as to what made you think it didn’t work.Did you try going to resource manager and see how many reducers were launched? I do not see any issue with syntax.


#3

It is working, I didn’t see any issue. Please check log below.

[replymarket@gw01 ~]$ pig -x tez -useHCatalog
ls: cannot access /usr/hdp/2.5.0.0-1245/hive/lib/slf4j-api-*.jar: No such file or directory
ls: cannot access /usr/hdp/2.5.0.0-1245/hive-hcatalog/lib/*hbase-storage-handler-*.jar: No such file or directory
18/01/19 12:09:47 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
18/01/19 12:09:47 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
18/01/19 12:09:47 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL
18/01/19 12:09:47 INFO pig.ExecTypeProvider: Trying ExecType : TEZ
18/01/19 12:09:47 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType
2018-01-19 12:09:47,521 [main] INFO  org.apache.pig.Main - Apache Pig version 0.16.0.2.5.0.0-1245 (rexported) compiled Aug 26 2016, 02:07:35
2018-01-19 12:09:47,521 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/replymarket/pig_1516381787498.log
2018-01-19 12:09:47,535 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /home/replymarket/.pigbootup not found
2018-01-19 12:09:48,106 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://nn01.itversity.com:8020
2018-01-19 12:09:48,936 [main] INFO  org.apache.pig.PigServer - Pig Script ID for the session: PIG-default-5021f587-1e5c-4599-8175-82c1f496919e
2018-01-19 12:09:49,301 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
2018-01-19 12:09:49,456 [main] INFO  org.apache.pig.backend.hadoop.PigATSClient - Created ATS Hook
grunt> set default_parallel 5;
grunt> a = load 'demo.orders' USING org.apache.hive.hcatalog.pig.HCatLoader();
2018-01-19 12:09:57,225 [main] INFO  hive.metastore - Trying to connect to metastore with URI thrift://nn01.itversity.com:9083
2018-01-19 12:09:57,306 [main] INFO  hive.metastore - Connected to metastore.
grunt>

#4