Pyspark Error With Udf: Py4j.py4jexception: Method __getnewargs__([]) Does Not Exist Error

I am trying to solve the following error (I am using the databricks platform and spark 2.0) tweets_cleaned.createOrReplaceTempView('tweets_cleanedSQL') def Occ(keyword): occurenc

Solution 1:

the UDF function turns a regular function to a function that is applied on any element of the input column. You cannot have this function call spark functions (in this case you are calling spark.sql which would in turn need to create workers etc. which is not supported.