Skip to content

Add missing "parallelism" variable to CTK Batch job template module. - #6313

Merged
shubpal07 merged 3 commits into
GoogleCloudPlatform:developfrom
jacobw125:batch-job-parallelism-arg
Sep 22, 2026
Merged

shubpal07 merged 3 commits into
GoogleCloudPlatform:developfrom
jacobw125:batch-job-parallelism-arg

Conversation

@jacobw125

@jacobw125 jacobw125 commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor
  • Allows users to configure parallelism of their cloud batch jobs (docs). This is especially useful for array jobs.

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

  • Fork your PR branch from the Toolkit "develop" branch (not main)
  • Test all changes with pre-commit in a local branch #
  • Confirm that "make tests" passes all tests
  • Add or modify unit tests to cover code changes
  • Ensure that unit test coverage remains above 80%
  • Update all applicable documentation
  • Follow Cluster Toolkit Contribution guidelines #

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new configuration option for the CTK Batch job template, enabling users to specify the parallelism level for their cloud batch jobs. This addition is particularly beneficial for managing array jobs more effectively by limiting concurrent task execution.

Highlights

  • Configuration Enhancement: Added a new 'parallelism' variable to the CTK Batch job template to allow users to control the maximum number of tasks running in parallel.
  • Template Integration: Updated the batch-job-base.yaml.tftpl template to conditionally include the parallelism setting if provided.
  • Documentation Update: Updated the module README to include the new 'parallelism' input variable.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot added the external PR from external contributor label Sep 11, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new parallelism variable to the batch-job-template module, allowing users to configure the maximum number of tasks that can run in parallel across a batch job. This change updates the Terraform variables, local configurations, the job template, and the README documentation. Feedback is provided to add a validation block to the parallelism variable to ensure it is a positive integer, preventing invalid configurations and potential API errors.

@jacobw125
jacobw125 force-pushed the batch-job-parallelism-arg branch from f239c1c to ad6e66e Compare September 11, 2026 22:42
@jacobw125
jacobw125 marked this pull request as ready for review September 11, 2026 22:42
@jacobw125
jacobw125 requested a review from a team as a code owner September 11, 2026 22:42
@mufaqam-gcl mufaqam-gcl added the release-module-improvements Added to release notes under the "Module Improvements" heading. label Sep 11, 2026
@mufaqam-gcl
mufaqam-gcl requested review from Neelabh94 and removed request for agrawalkhushi18 and vikramvs-gg September 11, 2026 22:45
@Neelabh94 Neelabh94 assigned jacobw125 and unassigned Neelabh94 Sep 12, 2026

@shubpal07 shubpal07 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to @Neelabh94's comments regarding the plan-time barrier precondition and adding CI coverage in tools/validate_configs/test_configs/multiple-batch-jobs.yaml.
To complement that:

  1. Auto-fallback in main.tf: Recommend setting parallelism = var.parallelism != null ? var.parallelism : (var.mpi_mode ? var.task_count : null) (mirroring tasks_per_node on line 25). This automatically handles the omission case in MPI jobs cleanly.
  2. Precondition: Keep Neelabh's precondition in local_file.job_template to guard against explicit mismatches (parallelism != task_count when mpi_mode = true).
  3. Docs: Update the description in variables.tf and run make terraform-format.
  4. CI: Verify tests pass with make validate_configs after adding parallelism: 2 to multiple-batch-jobs.yaml.

@jacobw125

Copy link
Copy Markdown
Contributor Author

@shubpal07 and @Neelabh94, thanks for your reviews. I have applied your suggestions.

@aslam-quad

Copy link
Copy Markdown
Contributor

/gcbrun

@Neelabh94 Neelabh94 removed their assignment Sep 15, 2026
@AdarshK15
AdarshK15 self-requested a review September 17, 2026 14:21
@AdarshK15 AdarshK15 self-assigned this Sep 17, 2026
@github-actions

Copy link
Copy Markdown

this PR has been inactive for 2 days and has changes requested. @jacobw125, please address the requested changes or close the PR if it's no longer needed.

@github-actions

Copy link
Copy Markdown

This is a second reminder that this PR has been inactive for 5 days and has changes requested. @jacobw125, please address the requested changes or close the PR if it's no longer needed.

@shubpal07
shubpal07 enabled auto-merge (squash) September 22, 2026 06:32
@shubpal07
shubpal07 merged commit 3c0e878 into GoogleCloudPlatform:develop Sep 22, 2026
27 of 90 checks passed
ep-nag pushed a commit to nagconsulting/cluster-toolkit that referenced this pull request Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external PR from external contributor release-module-improvements Added to release notes under the "Module Improvements" heading.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

⌂ 换网址