Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ Run `./auron-build.sh --help` to see all available options, including:
- `--celeborn`, `--uniffle`, `--paimon`, `--iceberg`: Optional integrations
- `--skiptests`: Skip unit tests (default: true)
- `--sparktests`: Run Spark integration tests
- `--threads`: Maven build threads (e.g. 1, 4, 1C). Defaults to single-threaded local builds; Docker defaults to 8 unless overridden.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Single thread for local builds might be too low, should we at least do 4? If machines are not able to handle it (seems like the edge case to me), they can always tune it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review! I think it's safer to keep the default at 1 thread in this PR to match the current behavior. We can bump it to 4 (or more) in a separate follow-up PR once this lands. Does that sound reasonable to you?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fair, thanks


### Running Tests

Expand Down
53 changes: 51 additions & 2 deletions auron-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ print_help() {
echo " --skiptests <true|false> Skip unit tests (default: true)"
echo " --sparktests <true|false> Run spark tests (default: false)"
echo " --docker <true|false> Build in Docker environment (default: false)"
echo " --threads <N|NC> Maven build threads (e.g. 1, 4, 1C). Default: local unset, docker 8"
IFS=','; echo " --image <NAME> Docker image to use (e.g. ${SUPPORTED_OS_IMAGES[*]}, default: ${SUPPORTED_OS_IMAGES[*]:0:1})"; unset IFS
IFS=','; echo " --sparkver <VERSION> Specify Spark version (e.g. ${SUPPORTED_SPARK_VERSIONS[*]})"; unset IFS
IFS=','; echo " --flinkver <VERSION> Specify Flink version (e.g. ${SUPPORTED_FLINK_VERSIONS[*]})"; unset IFS
Expand Down Expand Up @@ -126,6 +127,7 @@ RELEASE_PROFILE=false
CLEAN=true
SKIP_TESTS=true
SPARK_TESTS=false
THREADS=""
SPARK_VER=""
FLINK_VER=""
SCALA_VER=""
Expand Down Expand Up @@ -158,6 +160,21 @@ while [[ $# -gt 0 ]]; do
exit 1
fi
;;
--threads)
if [[ -n "$2" && "$2" != -* ]]; then
THREADS="$2"
# Validate THREADS to prevent command injection when passed through Docker
# Maven -T accepts an integer, optionally followed by 'C' (e.g., 4, 8, 2C)
if [[ ! "$THREADS" =~ ^[0-9]+C?$ ]]; then
echo "ERROR: Invalid --threads value '$THREADS'. Expected digits with optional 'C' (e.g., 4, 8, 2C)." >&2
exit 1
fi
shift 2
else
echo "ERROR: --threads requires a value (e.g. 1, 4, 1C)" >&2
exit 1
fi
;;
--image)
if [[ -n "$2" && "$2" != -* ]]; then
IMAGE_NAME="$2"
Expand Down Expand Up @@ -316,6 +333,31 @@ while [[ $# -gt 0 ]]; do
esac
done

# Detect conflict between --threads and raw Maven -T arguments.
if [[ -n "$THREADS" ]]; then
HAS_MVN_THREADS=false
PREV_MVN_ARG=""
for arg in "$@"; do
if [[ "$PREV_MVN_ARG" == "-T" ]]; then
HAS_MVN_THREADS=true
break
fi
if [[ "$arg" == "-T" ]]; then
PREV_MVN_ARG="-T"
continue
fi
if [[ "$arg" =~ ^-T.+ ]]; then
HAS_MVN_THREADS=true
break
fi
PREV_MVN_ARG=""
done
if [[ "$HAS_MVN_THREADS" == true ]]; then
echo "ERROR: Do not combine --threads with Maven -T options. Use only one." >&2
exit 1
fi
fi

# -----------------------------------------------------------------------------
# Section: Argument Validation
# Description:
Expand Down Expand Up @@ -396,6 +438,15 @@ if [[ -n "$ICEBERG_VER" ]]; then
BUILD_ARGS+=("-Piceberg-$ICEBERG_VER")
fi

# Configure Maven build threads:
# - local builds default to Maven's single-threaded behavior
# - docker builds default to -T8 unless overridden
if [[ -n "$THREADS" ]]; then
BUILD_ARGS+=("-T$THREADS")
elif [[ "$USE_DOCKER" == true ]]; then
BUILD_ARGS+=("-T8")
fi

MVN_ARGS=("${CLEAN_ARGS[@]}" "${BUILD_ARGS[@]}")

# -----------------------------------------------------------------------------
Expand Down Expand Up @@ -470,8 +521,6 @@ fi
# -----------------------------------------------------------------------------
if [[ "$USE_DOCKER" == true ]]; then
echo "[INFO] Compiling inside Docker container using image: $IMAGE_NAME"
# In Docker mode, use multi-threaded Maven build with -T8 for faster compilation
BUILD_ARGS+=("-T8")
if [[ "$CLEAN" == true ]]; then
# Clean the host-side directory that is mounted into the Docker container.
# This avoids "device or resource busy" errors when running `mvn clean` inside the container.
Expand Down
Loading