Not able to append the Hive table using pyspark

pyspark
#1

Here is my code:

from pyspark.sql import SparkSession
spark = SparkSession.builder.enableHiveSupport()
.config(“hive.exec.dynamic.partition”,“True”)
.config(“hive.exec.dynamic.partition.mode”,“nonstrict”)
.config(“hive.metastore.warehouse.dir”, “/user/hive/warehouse”)
.getOrCreate()
spark.sql(‘use mike’)
DF1 = spark.read.text(“hdfs:///input/employee1.txt”)
DF1.write.mode(“append”).format(“ORC”).insertInto("/apps/hive/warehouse/mike.db/orc_table",overwrite=False)

I want to insert the same file format data in existing ORC table but getting below error. Don’t know what is wrong here?

pyspark.sql.utils.ParseException: u"\nextraneous input ‘/’ expecting {‘SELECT’, ‘FROM’, ‘ADD’, ‘AS’, ‘ALL’, ‘DISTINCT’, ‘WHERE’, ‘GROUP’, ‘BY’, ‘GROUPING’, ‘SETS’, ‘CUBE’, ‘ROLLUP’, ‘ORDER’, ‘HAVING’, ‘LIMIT’,
‘AT’, ‘OR’, ‘AND’, ‘IN’, NOT, ‘NO’, ‘EXISTS’, ‘BETWEEN’, ‘LIKE’, RLIKE, ‘IS’, ‘NULL’, ‘TRUE’, ‘FALSE’, ‘NULLS’, ‘ASC’, ‘DESC’, ‘FOR’, ‘INTERVAL’, ‘CASE’, ‘WHEN’, ‘THEN’, ‘ELSE’, ‘END’, ‘JOIN’, ‘CROSS’, ‘OUTER’,
‘INNER’, ‘LEFT’, ‘SEMI’, ‘RIGHT’, ‘FULL’, ‘NATURAL’, ‘ON’, ‘LATERAL’, ‘WINDOW’, ‘OVER’, ‘PARTITION’, ‘RANGE’, ‘ROWS’, ‘UNBOUNDED’, ‘PRECEDING’, ‘FOLLOWING’, ‘CURRENT’, ‘FIRST’, ‘AFTER’, ‘LAST’, ‘ROW’, ‘WITH’,
‘VALUES’, ‘CREATE’, ‘TABLE’, ‘DIRECTORY’, ‘VIEW’, ‘REPLACE’, ‘INSERT’, ‘DELETE’, ‘INTO’, ‘DESCRIBE’, ‘EXPLAIN’, ‘FORMAT’, ‘LOGICAL’, ‘CODEGEN’, ‘COST’, ‘CAST’, ‘SHOW’, ‘TABLES’, ‘COLUMNS’, ‘COLUMN’, ‘USE’, ‘PAR
TITIONS’, ‘FUNCTIONS’, ‘DROP’, ‘UNION’, ‘EXCEPT’, ‘MINUS’, ‘INTERSECT’, ‘TO’, ‘TABLESAMPLE’, ‘STRATIFY’, ‘ALTER’, ‘RENAME’, ‘ARRAY’, ‘MAP’, ‘STRUCT’, ‘COMMENT’, ‘SET’, ‘RESET’, ‘DATA’, ‘START’, ‘TRANSACTION’, ’
COMMIT’, ‘ROLLBACK’, ‘MACRO’, ‘IGNORE’, ‘BOTH’, ‘LEADING’, ‘TRAILING’, ‘IF’, ‘POSITION’, ‘DIV’, ‘PERCENT’, ‘BUCKET’, ‘OUT’, ‘OF’, ‘SORT’, ‘CLUSTER’, ‘DISTRIBUTE’, ‘OVERWRITE’, ‘TRANSFORM’, ‘REDUCE’, ‘SERDE’, ‘S
ERDEPROPERTIES’, ‘RECORDREADER’, ‘RECORDWRITER’, ‘DELIMITED’, ‘FIELDS’, ‘TERMINATED’, ‘COLLECTION’, ‘ITEMS’, ‘KEYS’, ‘ESCAPED’, ‘LINES’, ‘SEPARATED’, ‘FUNCTION’, ‘EXTENDED’, ‘REFRESH’, ‘CLEAR’, ‘CACHE’, ‘UNCACH
E’, ‘LAZY’, ‘FORMATTED’, ‘GLOBAL’, TEMPORARY, ‘OPTIONS’, ‘UNSET’, ‘TBLPROPERTIES’, ‘DBPROPERTIES’, ‘BUCKETS’, ‘SKEWED’, ‘STORED’, ‘DIRECTORIES’, ‘LOCATION’, ‘EXCHANGE’, ‘ARCHIVE’, ‘UNARCHIVE’, ‘FILEFORMAT’, ‘TO
UCH’, ‘COMPACT’, ‘CONCATENATE’, ‘CHANGE’, ‘CASCADE’, ‘RESTRICT’, ‘CLUSTERED’, ‘SORTED’, ‘PURGE’, ‘INPUTFORMAT’, ‘OUTPUTFORMAT’, DATABASE, DATABASES, ‘DFS’, ‘TRUNCATE’, ‘ANALYZE’, ‘COMPUTE’, ‘LIST’, ‘STATISTICS’
, ‘PARTITIONED’, ‘EXTERNAL’, ‘DEFINED’, ‘REVOKE’, ‘GRANT’, ‘LOCK’, ‘UNLOCK’, ‘MSCK’, ‘REPAIR’, ‘RECOVER’, ‘EXPORT’, ‘IMPORT’, ‘LOAD’, ‘ROLE’, ‘ROLES’, ‘COMPACTIONS’, ‘PRINCIPALS’, ‘TRANSACTIONS’, ‘INDEX’, ‘INDE
XES’, ‘LOCKS’, ‘OPTION’, ‘ANTI’, ‘LOCAL’, ‘INPATH’, IDENTIFIER, BACKQUOTED_IDENTIFIER}(line 1, pos 0)\n\n== SQL ==\n/apps/hive/warehouse/mike.db/orc_table\n^^^\n"
19/05/22 05:57:25 INFO SparkContext: Invoking stop() from shutdown hook

0 Likes