在 Spark 中找不到适合 jdbc 的驱动程序

No suitable driver found for jdbc in Spark(在 Spark 中找不到适合 jdbc 的驱动程序)

本文介绍了在 Spark 中找不到适合 jdbc 的驱动程序的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我正在使用

df.write.mode("append").jdbc("jdbc:mysql://ip:port/database", "table_name", properties)

插入到 MySQL 中的表中.

to insert into a table in MySQL.

此外,我在代码中添加了 Class.forName("com.mysql.jdbc.Driver").

Also, I have added Class.forName("com.mysql.jdbc.Driver") in my code.

当我提交 Spark 申请时:

When I submit my Spark application:

spark-submit --class MY_MAIN_CLASS
  --master yarn-client
  --jars /path/to/mysql-connector-java-5.0.8-bin.jar
  --driver-class-path /path/to/mysql-connector-java-5.0.8-bin.jar
  MY_APPLICATION.jar

这种纱线客户端模式对我有用.

This yarn-client mode works for me.

但是当我使用纱线簇模式时:

But when I use yarn-cluster mode:

spark-submit --class MY_MAIN_CLASS
  --master yarn-cluster
  --jars /path/to/mysql-connector-java-5.0.8-bin.jar
  --driver-class-path /path/to/mysql-connector-java-5.0.8-bin.jar
  MY_APPLICATION.jar

它不起作用.我也试过设置--conf":

It doens't work. I also tried setting "--conf":

spark-submit --class MY_MAIN_CLASS
  --master yarn-cluster
  --jars /path/to/mysql-connector-java-5.0.8-bin.jar
  --driver-class-path /path/to/mysql-connector-java-5.0.8-bin.jar
  --conf spark.executor.extraClassPath=/path/to/mysql-connector-java-5.0.8-bin.jar
  MY_APPLICATION.jar

但仍然出现找不到适合 jdbc 的驱动程序"错误.

but still get the "No suitable driver found for jdbc" error.

推荐答案

有 3 种可能的解决方案,

There is 3 possible solutions,

  1. 您可能希望使用构建管理器(Maven、SBT)组装您的应用程序,因此您无需在 spark-submit cli 中添加依赖项.
  2. 您可以在 spark-submit cli 中使用以下选项:

  1. You might want to assembly you application with your build manager (Maven,SBT) thus you'll not need to add the dependecies in your spark-submit cli.
  2. You can use the following option in your spark-submit cli :

--jars $(echo ./lib/*.jar | tr ' ' ',')

说明:假设您在项目根目录的 lib 目录中拥有所有 jar,这将读取所有库并将它们添加到应用程序提交中.

Explanation : Supposing that you have all your jars in a lib directory in your project root, this will read all the libraries and add them to the application submit.

您也可以尝试在 SPARK_HOME/conf/spark 中配置这 2 个变量:spark.driver.extraClassPathspark.executor.extraClassPath-default.conf 文件并将这些变量的值指定为jar文件的路径.确保工作节点上存在相同的路径.

You can also try to configure these 2 variables : spark.driver.extraClassPath and spark.executor.extraClassPath in SPARK_HOME/conf/spark-default.conf file and specify the value of these variables as the path of the jar file. Ensure that the same path exists on worker nodes.

这篇关于在 Spark 中找不到适合 jdbc 的驱动程序的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持编程学习网!

本文标题为:在 Spark 中找不到适合 jdbc 的驱动程序

基础教程推荐