[GCP] Optimize BigQuery TriggerCopyJobs performance for WRITE_APPEND#38981
[GCP] Optimize BigQuery TriggerCopyJobs performance for WRITE_APPEND#38981stankiewicz wants to merge 6 commits into
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request improves the performance of the BigQuery FILE_LOADS pipeline in the Python SDK. By allowing asynchronous execution of copy jobs for append-only write operations, the pipeline avoids blocking worker threads unnecessarily, which was previously causing performance bottlenecks when handling multiple partitions or dynamic destinations. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request modifies bigquery_file_loads.py to conditionally set wait_for_job based on whether the write disposition is not 'WRITE_APPEND'. The reviewer suggests using the BigQueryDisposition.WRITE_APPEND constant instead of a hardcoded string literal to maintain consistency and improve code quality.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Updates BigQuery file loads in Python SDK to use multi-source copy jobs
when copying temporary tables to the final destination table.
* Update BigQueryWrapper._insert_copy_job to support a list of source
tables, utilizing BigQuery's multi-source copy capability.
* Update TriggerCopyJobs to process temporary tables in batch, splitting
them into chunks of 1,200 (BigQuery limit) and triggering
multi-source copy jobs.
* Implement inline wait for the first chunk in TriggerCopyJobs when
write disposition is WRITE_TRUNCATE or WRITE_EMPTY and there are
multiple chunks. This ensures the destination table is initialized
by the first job before subsequent chunks append to it.
* Fix grouping key in _load_data for WRITE_TRUNCATE/WRITE_EMPTY to use
the full hashable destination instead of just tableId, preventing
incorrect grouping of tables with the same name in different datasets.
* Fix TriggerLoadJobs to use bq_wrapper with mock client in tests,
resolving credential refresh warnings.
* Fix PartitionFiles to avoid yielding empty partitions when a file
exceeds limits.
TAG=agy
CONV=126370d2-f42e-4132-a237-16bd5ccf72a3
…le loads * Restore bq_io_metadata initialization fallback in TriggerCopyJobs.process. * Revert unrelated BigQueryWrapper change in TriggerLoadJobs to avoid credential warnings in tests. * Revert unconditional grouping for WRITE_APPEND to preserve graph structure. TAG=agy CONV=126370d2-f42e-4132-a237-16bd5ccf72a3
1 participant
This PR optimizes the BigQuery
FILE_LOADScopy pipeline branch in the Python SDK for append-only pipelines.When writing to multiple partitions or dynamic destinations, the pipeline loads data into temporary tables first, then runs a series of BigQuery copy jobs to merge them into the final destinations.
Previously,
TriggerCopyJobsblocked the worker thread synchronously on the first copy job for every unique destination table. While blocking is critical to maintain atomicity forWRITE_TRUNCATEandWRITE_EMPTY(ensuring the table is cleared before subsequent appends begin), it is unnecessary forWRITE_APPEND.This change updates the
wait_for_jobpolicy so it is onlyTruewhen the write disposition is notWRITE_APPEND.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.