Error creating sparkcontext locally An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext

We Are Going To Discuss About Error creating sparkcontext locally An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. So lets Start this Python Article.

Error creating sparkcontext locally An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext

  1. How to solve Error creating sparkcontext locally An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext

    I faced a similar problem when attempting to use the amazon/aws-glue-libs:glue_libs_2.0.0_image_01 image. In my case the problem was resolved by re-installing pyspark.
    The java.lang.NoSuchMethodError hints at various dependency versions being used which causes this error to be thrown.
    If you are interested in an image with AWS Glue, take a look at the Dockerfile provided by: https://github.com/DNXLabs/docker-glue-libs/blob/master/Dockerfile but it is just as easy to remove the glue-libs from the Dockerfile. The image can then be used as a remote interpreter in PyCharm (see for example: https://aws.amazon.com/blogs/big-data/developing-aws-glue-etl-jobs-locally-using-a-container/)
    edit:
    In my case the problem was caused by the following JARs that were loaded when using the aws-glue-libs:
    aws-glue-libs/jarsv1/javax.servlet-3.*
    aws-glue-libs/jarsv1/netty-*
    Remove these and the problem is gone

  2. Error creating sparkcontext locally An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext

    I faced a similar problem when attempting to use the amazon/aws-glue-libs:glue_libs_2.0.0_image_01 image. In my case the problem was resolved by re-installing pyspark.
    The java.lang.NoSuchMethodError hints at various dependency versions being used which causes this error to be thrown.
    If you are interested in an image with AWS Glue, take a look at the Dockerfile provided by: https://github.com/DNXLabs/docker-glue-libs/blob/master/Dockerfile but it is just as easy to remove the glue-libs from the Dockerfile. The image can then be used as a remote interpreter in PyCharm (see for example: https://aws.amazon.com/blogs/big-data/developing-aws-glue-etl-jobs-locally-using-a-container/)
    edit:
    In my case the problem was caused by the following JARs that were loaded when using the aws-glue-libs:
    aws-glue-libs/jarsv1/javax.servlet-3.*
    aws-glue-libs/jarsv1/netty-*
    Remove these and the problem is gone

Solution 1

I faced a similar problem when attempting to use the amazon/aws-glue-libs:glue_libs_2.0.0_image_01 image. In my case the problem was resolved by re-installing pyspark.

The java.lang.NoSuchMethodError hints at various dependency versions being used which causes this error to be thrown.

If you are interested in an image with AWS Glue, take a look at the Dockerfile provided by: https://github.com/DNXLabs/docker-glue-libs/blob/master/Dockerfile but it is just as easy to remove the glue-libs from the Dockerfile. The image can then be used as a remote interpreter in PyCharm (see for example: https://aws.amazon.com/blogs/big-data/developing-aws-glue-etl-jobs-locally-using-a-container/)

edit:
In my case the problem was caused by the following JARs that were loaded when using the aws-glue-libs:

  • aws-glue-libs/jarsv1/javax.servlet-3.*
  • aws-glue-libs/jarsv1/netty-*

Remove these and the problem is gone

Original Author Emptyless Of This Content

Solution 2

Try changing the Java version to JDK 8. I faced same error while my JAVA_HOME was mapped to JDK 16. Got it corrected by changing JAVA_HOME to JDK 8.

Original Author Gyan Rath Of This Content

Conclusion

So This is all About This Tutorial. Hope This Tutorial Helped You. Thank You.

Also Read,

ittutorial team

I am an Information Technology Engineer. I have Completed my MCA And I have 4 Year Plus Experience, I am a web developer with knowledge of multiple back-end platforms Like PHP, Node.js, Python and frontend JavaScript frameworks Like Angular, React, and Vue.

Leave a Comment