You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-44898][BUILD] Upgrade gcs-connector to 2.2.17
### What changes were proposed in this pull request?
This PR aims to upgrade gcs-connector to 2.2.17.
### Why are the changes needed?
To have the latest auth updates,
- https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15)
- GoogleCloudDataproc/hadoop-connectors#1041
```xml
- <google.auth.version>1.12.1</google.auth.version>
+ <google.auth.version>1.14.0</google.auth.version>
- <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version>
+ <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version>
```
- https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06)
- googleapis/google-auth-library-java#1100
- googleapis/google-auth-library-java#993
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs and manual tests.
**BUILD**
```
dev/make-distribution.sh -Phadoop-cloud
```
**TEST**
```
$ cd dist
$ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json
$ export EMAIL=$(jq -r '.client_email' < $KEYFILE)
$ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE)
$ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)"
$ bin/spark-shell \
-c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \
-c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \
-c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY"
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1692645442153).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT
/_/
Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312)
Type in expressions to have them evaluated.
Type :help for more information.
scala> spark.read.text("gs://apache-spark-bucket/README.md").count()
res0: Long = 124
scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc")
scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show()
+------+--------------+----------------+
| name|favorite_color|favorite_numbers|
+------+--------------+----------------+
|Alyssa| NULL| [3, 9, 15, 20]|
| Ben| red| []|
+------+--------------+----------------+
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closesapache#42588 from dongjoon-hyun/SPARK-44898.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
0 commit comments