Skip to content

Commit 84d9461

Browse files
yikfHyukjinKwon
authored andcommitted
[MINOR][SQL][DOCS] Improve spark.sql.files.minPartitionNum's doc
### What changes were proposed in this pull request? This PR aims to improve `spark.sql.files.minPartitionNum`'s doc. ### Why are the changes needed? Improve description ### Does this PR introduce _any_ user-facing change? Yes, better config description. ### How was this patch tested? GA Closes #40820 from Yikf/config-minPartitionNum-doc. Authored-by: Yikf <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
1 parent 00c2c19 commit 84d9461

File tree

2 files changed

+4
-3
lines changed

2 files changed

+4
-3
lines changed

docs/sql-performance-tuning.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ that these options will be deprecated in future release as more optimizations ar
9090
<td>Default Parallelism</td>
9191
<td>
9292
The suggested (not guaranteed) minimum number of split file partitions. If not set, the default
93-
value is `spark.default.parallelism`. This configuration is effective only when using file-based
93+
value is `spark.sql.leafNodeDefaultParallelism`. This configuration is effective only when using file-based
9494
sources such as Parquet, JSON and ORC.
9595
</td>
9696
<td>3.1.0</td>

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

+3-2
Original file line numberDiff line numberDiff line change
@@ -1740,8 +1740,9 @@ object SQLConf {
17401740

17411741
val FILES_MIN_PARTITION_NUM = buildConf("spark.sql.files.minPartitionNum")
17421742
.doc("The suggested (not guaranteed) minimum number of split file partitions. " +
1743-
"If not set, the default value is `spark.default.parallelism`. This configuration is " +
1744-
"effective only when using file-based sources such as Parquet, JSON and ORC.")
1743+
s"If not set, the default value is `${LEAF_NODE_DEFAULT_PARALLELISM.key}`. " +
1744+
"This configuration is effective only when using file-based sources " +
1745+
"such as Parquet, JSON and ORC.")
17451746
.version("3.1.0")
17461747
.intConf
17471748
.checkValue(v => v > 0, "The min partition number must be a positive integer.")

0 commit comments

Comments
 (0)