Skip to content

[AURON #2020] [BUILD] Add --threads option to control Maven build parallelism.#2021

Open
slfan1989 wants to merge 2 commits intoapache:masterfrom
slfan1989:auron-2020
Open

[AURON #2020] [BUILD] Add --threads option to control Maven build parallelism.#2021
slfan1989 wants to merge 2 commits intoapache:masterfrom
slfan1989:auron-2020

Conversation

@slfan1989
Copy link
Contributor

Which issue does this PR close?

Closes #2020

Rationale for this change

Currently, local builds are always single-threaded, which can be slow on multi-core machines. Docker builds hardcode -T8, which cannot be overridden by users. This change adds a --threads option to auron-build.sh to give users control over Maven build parallelism.

What changes are included in this PR?

  1. auron-build.sh

    • Added --threads parameter parsing and validation
    • Unified thread configuration logic: user-specified value takes precedence, Docker defaults to 8 threads, local defaults to single-threaded
    • Removed hardcoded -T8 from Docker build section
    • Updated help text to document the new option
  2. CONTRIBUTING.md

    • Documented --threads option under Build Options section
    • Described default behavior for local vs Docker builds

Are there any user-facing changes?

Yes. Users can now specify --threads to control Maven build parallelism

Default behavior remains unchanged (backward compatible):

  • Local builds: single-threaded
  • Docker builds: 8 threads

How was this patch tested?

  • Verified ./auron-build.sh --help displays correct usage information

…uild parallelism.

Signed-off-by: slfan1989 <slfan1989@apache.org>
@github-actions github-actions bot added documentation Improvements or additions to documentation build labels Feb 18, 2026
@slfan1989
Copy link
Contributor Author

./auron-build.sh --pre --sparkver 4.1 --scalaver 2.13 --threads 2

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Auron Parent Pom 7.0.0-incubating:
[INFO] 
[INFO] Apache Auron Parent Pom ............................ SUCCESS [ 11.384 s]
[INFO] Apache Auron Proto ................................. SUCCESS [ 28.548 s]
[INFO] Apache Auron Hadoop Shim 2.13 ...................... SUCCESS [  8.290 s]
[INFO] Apache Auron Core .................................. SUCCESS [  9.705 s]
[INFO] Apache Auron Common 2.13 ........................... SUCCESS [ 15.309 s]
[INFO] Apache Auron Spark Version Annotation Macros 2.13 .. SUCCESS [  6.961 s]
[INFO] Apache Auron Spark UI 2.13 ......................... SUCCESS [  6.463 s]
[INFO] Apache Auron Spark Extension 2.13 .................. SUCCESS [ 35.552 s]
[INFO] Apache Auron Spark Extension Shims Spark 4.1.1_2.13  SUCCESS [ 28.957 s]
[INFO] Apache Auron Assembly .............................. SUCCESS [ 12.828 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:07 min (Wall Clock)
[INFO] Finished at: 2026-02-18T14:00:14+08:00
[INFO] ------------------------------------------------------------------------

./auron-build.sh --pre --sparkver 4.1 --scalaver 2.13 --threads 4

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Auron Parent Pom 7.0.0-incubating:
[INFO] 
[INFO] Apache Auron Parent Pom ............................ SUCCESS [  6.504 s]
[INFO] Apache Auron Proto ................................. SUCCESS [ 25.801 s]
[INFO] Apache Auron Hadoop Shim 2.13 ...................... SUCCESS [ 10.265 s]
[INFO] Apache Auron Core .................................. SUCCESS [  8.988 s]
[INFO] Apache Auron Common 2.13 ........................... SUCCESS [ 14.699 s]
[INFO] Apache Auron Spark Version Annotation Macros 2.13 .. SUCCESS [  6.991 s]
[INFO] Apache Auron Spark UI 2.13 ......................... SUCCESS [  6.127 s]
[INFO] Apache Auron Spark Extension 2.13 .................. SUCCESS [ 26.319 s]
[INFO] Apache Auron Spark Extension Shims Spark 4.1.1_2.13  SUCCESS [ 25.859 s]
[INFO] Apache Auron Assembly .............................. SUCCESS [ 14.000 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:47 min (Wall Clock)
[INFO] Finished at: 2026-02-18T14:02:31+08:00
[INFO] ------------------------------------------------------------------------

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new --threads option to auron-build.sh so users can control Maven parallelism, removing the previously hardcoded Docker -T8 behavior while keeping existing defaults (local single-threaded; Docker defaults to 8 threads).

Changes:

  • Add --threads CLI option, plumbed into Maven -T handling with precedence: user-specified > Docker default (8) > local default (unset).
  • Remove the Docker-only hardcoded -T8 addition and centralize thread selection in one place.
  • Document the new option and defaults in CONTRIBUTING.md.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
auron-build.sh Adds --threads parsing and centralizes Maven -T argument selection for local vs Docker builds.
CONTRIBUTING.md Documents the new --threads build option and default behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- `--celeborn`, `--uniffle`, `--paimon`, `--iceberg`: Optional integrations
- `--skiptests`: Skip unit tests (default: true)
- `--sparktests`: Run Spark integration tests
- `--threads`: Maven build threads (e.g. 1, 4, 1C). Defaults to single-threaded local builds; Docker defaults to 8 unless overridden.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Single thread for local builds might be too low, should we at least do 4? If machines are not able to handle it (seems like the edge case to me), they can always tune it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review! I think it's safer to keep the default at 1 thread in this PR to match the current behavior. We can bump it to 4 (or more) in a separate follow-up PR once this lands. Does that sound reasonable to you?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fair, thanks

…uild parallelism.

Signed-off-by: slfan1989 <slfan1989@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUILD] Add --threads option to control Maven build parallelism

3 participants