How to apply the not equal condition in spark dataframe join


My use cases is
I have two tables (table1, and table2, both table have same structure) one table contain extra record which is not contain in another table(table2).
Want to find records missing record in table2.

Problem facing: dont know what is not equal operator
Ex for equals operator : table1.join(table2, $“df1Key” === $“df2Key”)

but I achieved through sql query way.

Question 1 : how to achieve the join with not equal attribute in df api

And dataframe function is returning all table1 and table2 data eventhough not needed the all columns. But achieved throw Sql query.

Ques 2 : Is there possible to achieve selected column in the df api

Suresh Selvaraj

Hi suresh,

Condition for not equals

table1.join(table2, $“df1Key” !== $“df2Key”)


Thanks venkat, Will try it andlet you know.

1 Like