Skip to content

Commit 1f58395

Browse files
dongjoon-hyunragnarok56
authored andcommitted
[SPARK-44898][BUILD] Upgrade gcs-connector to 2.2.17
### What changes were proposed in this pull request? This PR aims to upgrade gcs-connector to 2.2.17. ### Why are the changes needed? To have the latest auth updates, - https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15) - GoogleCloudDataproc/hadoop-connectors#1041 ```xml - <google.auth.version>1.12.1</google.auth.version> + <google.auth.version>1.14.0</google.auth.version> - <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version> + <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version> ``` - https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06) - googleapis/google-auth-library-java#1100 - googleapis/google-auth-library-java#993 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs and manual tests. **BUILD** ``` dev/make-distribution.sh -Phadoop-cloud ``` **TEST** ``` $ cd dist $ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json $ export EMAIL=$(jq -r '.client_email' < $KEYFILE) $ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE) $ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)" $ bin/spark-shell \ -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \ -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \ -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY" Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://localhost:4040 Spark context available as 'sc' (master = local[*], app id = local-1692645442153). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312) Type in expressions to have them evaluated. Type :help for more information. scala> spark.read.text("gs://apache-spark-bucket/README.md").count() res0: Long = 124 scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc") scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show() +------+--------------+----------------+ | name|favorite_color|favorite_numbers| +------+--------------+----------------+ |Alyssa| NULL| [3, 9, 15, 20]| | Ben| red| []| +------+--------------+----------------+ ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#42588 from dongjoon-hyun/SPARK-44898. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 69b0e3e commit 1f58395

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

dev/deps/spark-deps-hadoop-3-hive-2.3

+1-1
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ datasketches-memory/2.1.0//datasketches-memory-2.1.0.jar
6262
derby/10.14.2.0//derby-10.14.2.0.jar
6363
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
6464
flatbuffers-java/1.12.0//flatbuffers-java-1.12.0.jar
65-
gcs-connector/hadoop3-2.2.16/shaded/gcs-connector-hadoop3-2.2.16-shaded.jar
65+
gcs-connector/hadoop3-2.2.17/shaded/gcs-connector-hadoop3-2.2.17-shaded.jar
6666
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
6767
gson/2.2.4//gson-2.2.4.jar
6868
guava/14.0.1//guava-14.0.1.jar

pom.xml

+1-1
Original file line numberDiff line numberDiff line change
@@ -160,7 +160,7 @@
160160
<aws.java.sdk.version>1.11.655</aws.java.sdk.version>
161161
<!-- the producer is used in tests -->
162162
<aws.kinesis.producer.version>0.12.8</aws.kinesis.producer.version>
163-
<gcs-connector.version>hadoop3-2.2.16</gcs-connector.version>
163+
<gcs-connector.version>hadoop3-2.2.17</gcs-connector.version>
164164
<!-- org.apache.httpcomponents/httpclient-->
165165
<commons.httpclient.version>4.5.14</commons.httpclient.version>
166166
<commons.httpcore.version>4.4.16</commons.httpcore.version>

0 commit comments

Comments
 (0)