July 3, 2024

Weekly GitHub Report for Tensorflow - 2024-07-03 10:04:43

Weekly GitHub Report for Tensorflow

Thank you for subscribing to our weekly newsletter! Each week, we deliver a comprehensive summary of your GitHub project's latest activity right to your inbox, including an overview of your project's issues, pull requests, contributors, and commit activity.

---

I. Issues

1.1 Open Issues

Open Issues This Week: 26

Summarized Issues:

• TensorFlow Version Compatibility Issues: Compatibility issues between different TensorFlow and Keras versions can lead to errors when importing models. For instance, models trained in TensorFlow 2.15 may not be importable in TensorFlow 2.16 due to changes in Keras, resulting in ValueErrors. Similarly, TensorFlow 2.15.1 and Keras 2.15.0 may fail to load saved models due to deserialization errors.
  • github.com/tensorflow/tensorflow/issues/70394
  • github.com/tensorflow/tensorflow/issues/70456

• TensorFlow Lite Model Conversion and Execution Issues: Converting and executing TensorFlow Lite models can present various challenges. Issues include ReLU activation functions returning negative values after int8 quantization, and LSTM-based Keras models failing to convert using certain methods. Additionally, custom delegates and GPU delegates may not load or execute models correctly.
  • github.com/tensorflow/tensorflow/issues/70496
  • github.com/tensorflow/tensorflow/issues/70502
  • github.com/tensorflow/tensorflow/issues/70399
  • github.com/tensorflow/tensorflow/issues/70664

• CUDA and GPU Recognition Issues: TensorFlow may not recognize GPUs on certain systems despite correct CUDA and cuDNN installations. This can occur on systems like Ubuntu 22.04 with CUDA 12.4 and cuDNN 9.0, where TensorFlow fails to detect the GPU even though nvidia-smi identifies it correctly.
  • github.com/tensorflow/tensorflow/issues/70444

• TensorFlow Build and Installation Issues: Building and installing TensorFlow can be problematic on various platforms. Users may encounter errors related to local CUDA configurations, Python version specifiers, and missing dependencies. These issues can arise on systems like Arch Linux, Windows, and Raspberry Pi.
  • github.com/tensorflow/tensorflow/issues/70603
  • github.com/tensorflow/tensorflow/issues/70617
  • github.com/tensorflow/tensorflow/issues/70658
  • github.com/tensorflow/tensorflow/issues/70659
  • github.com/tensorflow/tensorflow/issues/70730
  • github.com/tensorflow/tensorflow/issues/70744
  • github.com/tensorflow/tensorflow/issues/70747

• TensorFlow Data Handling and Performance Issues: Data handling operations in TensorFlow, such as shuffling and sharding, can lead to unexpected behavior and performance degradation. For example, tf.data.Dataset.shuffle can be unintentionally triggered, and dynamic sharding policies can significantly reduce performance.
  • github.com/tensorflow/tensorflow/issues/70521
  • github.com/tensorflow/tensorflow/issues/70541

• TensorFlow C++ API and Model Loading Issues: The TensorFlow C++ API may lack support for new model formats and encounter memory allocation errors. Users have requested support for loading models saved in the new .keras format and reported issues with saving and restoring weights using the TFLite API.
  • github.com/tensorflow/tensorflow/issues/70514
  • github.com/tensorflow/tensorflow/issues/70726

• TensorFlow Operation and Layer Bugs: Bugs in TensorFlow operations and layers can disrupt model training and inference. Issues include incorrect batch dimension handling in tfm.nlp.layers.RelativePositionEmbedding and inconsistent results in tf.matmul operations compared to NumPy.
  • github.com/tensorflow/tensorflow/issues/70520
  • github.com/tensorflow/tensorflow/issues/70672

• TensorFlow Model Execution and Inference Issues: Executing TensorFlow models on different platforms can lead to unexpected results or crashes. For example, TensorFlow Lite models may not display expected outputs on Android devices, and segmentation faults can occur on Raspberry Pi.
  • github.com/tensorflow/tensorflow/issues/70682
  • github.com/tensorflow/tensorflow/issues/70747

• Documentation and Feature Requests: Users have requested better documentation and new features to improve TensorFlow's usability. Requests include guidance on assigning operations to specific devices and effective data augmentation techniques for object detection.
  • github.com/tensorflow/tensorflow/issues/70670
  • github.com/tensorflow/tensorflow/issues/70687

• Security and Advisory Issues: Incorrect information in security advisories can lead to confusion. For instance, GitHub advisory GHSA-84mw-34w6-2q43 contains incorrect POC codes, which actually belong to another advisory, GHSA-772p-x54p-hjrv.
  • github.com/tensorflow/tensorflow/issues/70724

• Kubernetes and Cluster Issues: TensorFlow's integration with Kubernetes can encounter issues, such as the Kubernetes cluster resolver failing due to missing kubectl config files, resulting in ConfigException errors.
  • github.com/tensorflow/tensorflow/issues/70581

• Model Evaluation and Metric Issues: Errors during model evaluation can occur, such as indexing errors with the tf.keras.metrics.OneHotIoU metric during the evaluation phase of a VGG16 model fine-tuning process.
  • github.com/tensorflow/tensorflow/issues/70673

1.2 Top 5 Active Issues:

We consider active issues to be issues that have generated much discussion in the issue's comments.

1. Cannot take the length of shape with unknown rank.: This issue is about a user encountering a ValueError when trying to train a U-Net model using TensorFlow, specifically due to an error in taking the length of a shape with an unknown rank. The user has shared their training code and the error traceback, seeking help to resolve the problem.

• A commenter suggests that the build_unet method should accept two arguments: the shape of the input as a tuple and the number of dimensions, and provides a code snippet to illustrate the correct usage.
  • Number of comments: 17

1. What is preventing TF to use GPU when used in native windows?: This issue is about the lack of GPU support for TensorFlow on native Windows environments for versions greater than 2.10, and the user is seeking an explanation and guidance on how to implement this support independently. The user also inquires about the necessary Python package requirements and native Windows compiler requirements to achieve this.

• The first comment explains that the TensorFlow team has limited resources, making it challenging to support native Windows, and suggests using WSL2 or other alternatives for GPU support.
  • Number of comments: 16

1. GCS gfile operations fail in TF 2.17.0rc0 and 2.18 nightly when not running in GCP: This issue pertains to GCS operations using tf.io.gfile failing in TensorFlow versions 2.17.0rc0 and 2.18 nightly when not running on a Google Cloud Platform (GCP) virtual machine. The problem does not occur in TensorFlow 2.16 and is reproducible on various local environments, including macOS, Docker containers, GitHub actions, and Kaggle notebooks.

   • A user suggested that a specific commit might be influencing the issue, but testing is difficult due to the unavailability of older tf-nightly wheels on PyPI.
   • Number of comments: 9

2. Data Service Dynamic sharding policy seems extremely slow: This issue reports a significant performance degradation when using the dynamic sharding policy in TensorFlow's data service, compared to when the sharding policy is set to off. The user provides a simple example where changing the sharding policy to dynamic results in a substantial slowdown, even for a basic dataset.

   • A commenter confirms the issue by running the provided code on Colab with TensorFlow v2.15.0 and the nightly version, experiencing the same performance problem.
   • Number of comments: 8

3. "invalid static_cast" on AVX512FP16 (e.g. Sapphire Rapids): This issue involves a build failure in TensorFlow version 2.15.1 when compiling for CPUs that support AVX512 FP16 extensions, such as Sapphire Rapids, due to an invalid static_cast in the Eigen library. The problem has been identified as an issue in Eigen, which has been addressed in a recent merge request, and the suggested solutions include either applying the patch or updating Eigen once the merge request is merged.

• The issue was reopened, and a user mentioned that the relevant merge request in Eigen has been merged, potentially fixing the problem upstream.
  • Number of comments: 7

1.3 Top 5 Quiet Issues:

We consider quiet issues to be issues that have been opened in this project for the longest time. The team should work together to get these issues resolved and closed as soon as possible.

1. "invalid static_cast" on AVX512FP16 (e.g. Sapphire Rapids): This issue pertains to a build failure in TensorFlow version 2.15.1 when compiling for CPUs that support AVX512 FP16 extensions, such as Sapphire Rapids, due to an invalid static_cast in the Eigen library. The problem has been identified as an issue in Eigen, and a potential fix is available through a merge request, which TensorFlow can either apply as a patch or wait for it to be merged into the Eigen library.

   • Open for 19 days, 09 hours, 42 minutes

2. TFlite usage on android: This issue involves optimizing the process of converting a Bitmap to a ByteBuffer for real-time video processing on Android using a TFlite model. The current method is too time-consuming, and the user is seeking alternative strategies to improve the efficiency of both the conversion and the subsequent copying of the image from ByteBuffer to Bitmap.

   • Open for 19 days, 07 hours, 33 minutes

3. Cannot cross-compile minimal tflite example: This issue pertains to the inability to cross-compile a minimal TensorFlow Lite example on Rocky Linux 8.1 using a specified CMake toolchain file for the RISC-V architecture. The error occurs during the linking stage, where the linker fails due to an undefined reference to the cpuinfo_riscv_linux_init function, resulting in a final link failure.

   • Open for 19 days, 04 hours, 34 minutes

4. Segmentation fault in tf.raw_ops.CollectiveGatherV2: This issue reports a segmentation fault occurring in the tf.raw_ops.CollectiveGatherV2 operation when provided with specific input parameters in TensorFlow version 2.16 on a Linux Ubuntu 22.04.3 LTS system. The problem has been reproduced using TensorFlow Nightly and involves a core dump error, as demonstrated by the provided standalone code and log output.

   • Open for 19 days, 02 hours, 30 minutes

5. Segmentation fault in tf.raw_ops.CollectiveAllToAllV2: This issue reports a segmentation fault occurring in the tf.raw_ops.CollectiveAllToAllV2 operation when provided with specific input parameters, leading to a core dump error. The problem has been reproduced using TensorFlow version 2.16 on a Linux Ubuntu 22.04.3 LTS system with Python 3.9.13, and includes a code snippet to replicate the fault.

   • Open for 19 days, 02 hours, 27 minutes

1.4 Closed Issues

Closed Issues This Week: 9

Average Issue Close Time (This Week): 7.88 days

Average Issue Close Time (All Time): 4.53 days

Summarized Issues:

• Compiler Change to Clang on Windows: This issue questions why TensorFlow has switched to using Clang as the default compiler on Windows and seeks to understand the benefits of this change. The discussion revolves around performance improvements, better compatibility with modern C++ standards, and potential for more consistent cross-platform behavior. Users are curious about the specific advantages and any potential drawbacks of this transition.
  • github.com/tensorflow/tensorflow/issues/69665

• UnsortedSegment Operations Bug: This issue describes a bug in TensorFlow 2.16 where specific input to the tf.raw_ops.UnsortedSegmentJoin, UnsortedSegmentMin, UnsortedSegmentMax, UnsortedSegmentProd, and UnsortedSegmentSum APIs causes the session to crash due to excessive RAM usage. Additionally, tf.raw_ops.UnsortedSegmentJoin triggers an "Aborted (core dumped)" error. The problem highlights the need for better memory management and error handling in these operations.
  • github.com/tensorflow/tensorflow/issues/69779

• Learning Rate Update Issue: This issue describes a bug where compiling a reloaded TensorFlow model does not update its learning rate as expected, but instead changes the learning rate of the optimizer. This was resolved in TensorFlow version 2.16.1. The resolution involved ensuring that the learning rate parameter is correctly applied to the model rather than the optimizer.
  • github.com/tensorflow/tensorflow/issues/70050

• Scam Posts on TensorFlow Website: This issue reports that scammers are spamming the TensorFlow website with gambling and unsafe health supplement posts. These posts are causing financial losses to users and there is a request for immediate action to address the problem. The issue underscores the importance of robust moderation and security measures on community platforms.
  • github.com/tensorflow/tensorflow/issues/70085

• UnicodeEncodeError on Windows 11: This issue describes a problem where the user is unable to use the fit function in TensorFlow's Sequential model on Windows 11 with Python 3.12. The user encounters a UnicodeEncodeError due to the system's default encoding not being set to UTF-8. The issue calls for better handling of encoding settings in TensorFlow to ensure compatibility across different systems.
  • github.com/tensorflow/tensorflow/issues/70261

• TensorFlow Lite Conversion Issues: This issue pertains to difficulties encountered during the conversion of a TensorFlow model to TensorFlow Lite. Problems include incorrect model results, reduced accuracy, or slower performance post-conversion. The issue highlights the need for improved tools and documentation to assist users in the conversion process.
  • github.com/tensorflow/tensorflow/issues/70265

• Model Loading Error: This issue involves an error encountered when attempting to load a TensorFlow model using tensorflow.keras.models.load_model. The process fails due to a missing model.weights.h5 file, despite the model being saved in .keras format. The issue points to potential gaps in the model saving and loading mechanisms that need to be addressed.
  • github.com/tensorflow/tensorflow/issues/70273

• ModuleNotFoundError in TensorFlow Lite: This issue pertains to a ModuleNotFoundError encountered when attempting to import specific modules (vision, core, and processor) from tflite_support.task in a TensorFlow Lite project. This is likely due to the deprecation of the TensorFlow Lite Task Library and its replacement by MediaPipe Tasks. The issue calls for updated guidance and support for transitioning to the new library.
  • github.com/tensorflow/tensorflow/issues/70546

• Tutorial Notebook Compatibility Issues: This issue pertains to a TensorFlow tutorial notebook that is currently producing errors due to compatibility issues with TensorFlow v2.16 and Keras 3.0. There is a request for an update to the notebook to resolve these errors. The issue highlights the need for continuous maintenance of educational resources to keep them in sync with the latest software versions.
  • github.com/tensorflow/tensorflow/issues/70562

1.5 Issue Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open issues within the past week to identify potentially heated exchanges and to maintain a constructive project environment.

Based on our analysis, there are no instances of toxic discussions in the project's open issues from the past week.

---

II. Pull Requests

2.1 Open Pull Requests

Open Pull Requests This Week: 5

Pull Requests:

• Google Cloud Storage operations in TensorFlow: This pull request aims to resolve issues with TensorFlow's Google Cloud Storage operations in specific environments. It reverts a previous commit that added a trailing dot to the storage.googleapis.com domain. This should address the problem reported in issue #69789 for the 2.17 release.
  • github.com/tensorflow/tensorflow/pull/70452

• Enhancements in TensorFlow's tb_summary.py: This pull request enhances the tb_summary.py file in the TensorFlow project. It adds a mesh summary feature, as indicated by its title. This enhancement aims to improve the functionality and usability of the TensorFlow summary tools.
  • github.com/tensorflow/tensorflow/pull/70591

• Support for broadcasting unequal rank inputs in TensorFlow: This pull request adds support for broadcasting unequal rank inputs in the TFL_BatchMatMulOp. It does so when lowering to tosa.matmul in TensorFlow. This enhancement aims to improve the flexibility and robustness of matrix multiplication operations.
  • github.com/tensorflow/tensorflow/pull/70679

• KubernetesClusterResolver enhancements: This pull request addresses issue #70581 in the TensorFlow repository. It enables the caller to specify whether the KubernetesClusterResolver is being used within a Kubernetes cluster or in an external environment. This enhancement aims to provide better configuration options for users.
  • github.com/tensorflow/tensorflow/pull/70691

• Build error resolution in TensorFlow Lite: This pull request addresses and resolves the build error in the TensorFlow Lite label_image example. The issue occurs when using CMake, as referenced in issue #70659. This fix ensures that the example can be built successfully without errors.
  • github.com/tensorflow/tensorflow/pull/70742

2.2 Closed Pull Requests

Closed Pull Requests This Week: 0

Summarized Pull Requests:

As of our latest update, there are no closed pull requests for the project this week.

2.3 Pull Request Discussion Insights

This section will analyze the tone and sentiment of discussions within this project's open pull requests within the past week to identify potentially heated exchanges and to maintain a constructive project environment.

1. Fix issues with TF GCS operations not working in certain environments.
   • Toxicity Score: 0.45 (Frustration expressed, defensive tone, potential for future conflicts)
   • This GitHub conversation begins with belitskiy proposing a reversion of a previous commit to address an issue in a specific release. The initial tone is neutral and focused on problem-solving. As other users join the conversation, there is a mix of agreement and constructive feedback. However, tension arises when username1 expresses frustration that username2's solution did not work, leading to a more defensive tone from username2. The conversation remains mostly professional, but the underlying frustration hints at potential future conflicts.

---

III. Commits

3.1 Commits

Commits This Week: 101

Summarized Commits:

• Batch Stats Module Enhancement: This commit introduces a feature to the batch stats module that adds a cumulative query count for each model, enhancing the tracking and analysis of model queries over time.

• XLA:Python Optimization: This commit introduces a minor optimization in the XLA:Python project by defining a tp_hash slot instead of a __hash__ method to improve performance slightly. It also notes that tp_richcompare is marginally faster to dispatch than __eq__, based on profiling observations.

• Redundant Code Removal: This commit removes a redundant bounds check in the code, as the condition is already verified in the EvaluateEpilogue function where thread_id[1] is constrained to {0, 0}. Another commit removes the dependency on libtensorflow_framework.so.2, which was initially added as a workaround for a Bazel bug that now appears to be resolved.

• Compilation and Build Fixes: Several commits address and fix compilation errors and build failures in various tests and targets within the XLA service on GPU, ensuring compatibility and successful execution in the open-source setting.

• Dependency Management: This commit inlines the XLA_DEVICE_DEPS and runs build_cleaner to avoid the antipattern of factoring out common dependencies, which complicates dependency management and can lead to the inclusion of unnecessary dependencies in targets that reference the "common" dependencies.

• Custom Call Registry API: This commit introduces the recording of command buffer custom call targets for the legacy custom call registry API in XLA:GPU, providing temporary support for lowering TE custom call kernels to command buffers via a whitelist until the FFI API migration is complete.

• Performance Improvements: This commit updates the ConstraintExpression to utilize llvm::SmallVector internally, resulting in a 20% improvement in compile-time performance on benchmarks. Another commit optimizes the computation of buffer sizes in bytes by using statically known buffer ranks, resulting in performance improvements for various benchmarks.

• Forward Compatibility and Version Updates: Multiple commits update the forward compatibility horizon and GraphDef version to ensure the project remains compatible with future changes and developments.

• Automated Code Changes: Several commits involve automated code changes, as indicated by the commit messages and associated PiperOrigin-RevIds.

• Refactoring and Code Cleanup: Various commits involve refactoring code, such as splitting multi-row reduction functionality into a separate class, renaming functions for clarity, and removing redundant include directives to streamline the codebase.

• New Features and Enhancements: This commit introduces support for uint8_t dot operation tests and integrates the necessary functionality into the HLO evaluator. Another commit introduces broadcast splitting before sharding propagation in XLA:SPMD to prevent unwanted sharding propagation.

• Documentation Updates: This commit updates the docs/operation_semantics.md document by adding detailed descriptions for several element-wise binary operations. Another commit adds documentation and examples on how to use the multihost_hlo_runner.

• Bug Fixes and Error Handling: Several commits address and fix bugs, such as correcting file paths in target patterns, fixing constructor conversion errors, and enhancing error messages for better clarity.

• LLVM Integration: Multiple commits integrate the LLVM project at specific commits, updating the project's LLVM usage to align with the changes made in those particular revisions.

• Test Enhancements: This commit introduces new test cases and benchmarks to verify the deduplication of HLO operations and the management of execution threads in the XLA compiler.

• Build Configuration Updates: This commit updates the build configuration to use specific CPU tag filters for the CPU build and CUDA tag filters for the CUDA build, ensuring appropriate build settings for different environments.

• FFI and PJRT Enhancements: This commit introduces FFI custom call thunk runtime support to the PJRT CPU client and includes a benchmark to evaluate the PJRT CPU Client's performance.

• Miscellaneous Updates: Various commits include updates such as adding a defensive mechanism for exception handling in user FFI calls, updating the google/fuzztest dependency, and enhancing the RemapPlan::DebugString() function.

---

IV. Contributors

4.1 Contributors

Active Contributors:

We consider an active contributor in this project to be any contributor who has made at least 1 commit, opened at least 1 issue, or created at least 1 pull request in the past month.

Contributor	Commits	Pull Requests	Issues
A. Unique TensorFlower	31	0	0
Johannes Reifferscheid	8	0	0
Eugene Zhulenev	7	0	0
Adrian Kuegel	6	0	0
Sergey Kozub	5	0	0
David Dunleavy	5	0	0
x0w3n	0	0	5
Adam Banaś	3	0	0
Dimitar (Mitko) Asenov	2	0	0
Chi Zeng	2	0	0
Penporn Koanantakool	2	0	0
Alexander Pivovarov	2	0	0
Hyeontaek Lim	2	0	0
msteiner-google	0	1	1
NobuoTsukamoto	0	1	1
zoghin	0	0	2
tirk999	0	0	2
lgeiger	0	0	2
orikkwak	0	0	2
MengXiangDing	0	0	2
fujunwei	0	0	2
ravin00	0	0	2
Peter Hawkins	1	0	0
Shaogang Wang	1	0	0
Benjamin Chetioui	1	0	0
Oleg Shyshkov	1	0	0
Nicolas Perez	1	0	0
Kevin Gleason	1	0	0
Andrew Audibert	1	0	0
Tongfei Guo	1	0	0
Marissa Ikonomidis	1	0	0
Dan Foreman-Mackey	1	0	0
Clive Verghese	1	0	0
Quentin Khan	1	0	0
Feng Wang	1	0	0
Christian Sigg	1	0	0
Sandeep Dasgupta	1	0	0
Tamás Danyluk	1	0	0
Anlun Xu	1	0	0
lingzhi98	1	0	0
Shawn Wang	1	0	0
Vadym Matsishevskyi	1	0	0
TensorFlower Gardener	1	0	0
Vamsi Manchala	1	0	0
Changhui Lin	1	0	0
Tomás Longeri	1	0	0
Frédéric Bastien	1	0	0
Farzin Houshmand	1	0	0
belitskiy	0	1	0
Mirgahney	0	1	0
psunn	0	1	0
risemeup1	0	0	1
Flamefire	0	0	1
Rad-Monk	0	0	1
dhruvsingla273	0	0	1
abeyene	0	0	1
nrad	0	0	1
ludiusvox	0	0	1
marcobornstein	0	0	1
roshni-warrantylife	0	0	1
eabase	0	0	1
dingjieliu	0	0	1
arthurflor23	0	0	1
Barooni-saeede	0	0	1
jsasswis	0	0	1
captainst	0	0	1
AnyLifeZLB	0	0	1
MAlmuzaini	0	0	1
SMH17	0	0	1
Dustglaxy	0	0	1
Jerry-yz	0	0	1
GreatKampfstern	0	0	1
gachombamuthiri	0	0	1
johnlarkin1	0	0	1
willendzw	0	0	1
ninjaguardian	0	0	1
MatthiasNeumueller	0	0	1
chiamp	0	0	1
myh1234567	0	0	1
hui-li-xf	0	0	1
kindaTall	0	0	1
BenCrulis	0	0	1
H4dr1en	0	0	1
maxwellbrien1990	0	0	1
Nayana-ibm	0	0	1
yanncalec	0	0	1
Suryansh-patidar	0	0	1
kupix	0	0	1
feranick	0	0	1
cpappasILMX	0	0	1
ThorvaldAagaard	0	0	1
cz2h	0	0	1
yousifj129	0	0	1
manswad	0	0	1
SsomsakTH	0	0	1
weebao	0	0	1
basi880830	0	0	1
WeichenXu123	0	0	1
NoranNabil	0	0	1
mrezaanony	0	0	1
kkk-jpeg	0	0	1
CV-42	0	0	1
tomneo2004	0	0	1
pekaalto	0	0	1
npanpaliya	0	0	1
pataneanant	0	0	1
suyash-narain	0	0	1
hamzaziizzz	0	0	1
arpanptnk85	0	0	1
luanft	0	0	1
KarenMars	0	0	1
Sklaebe	0	0	1
Leo-Lifeblood	0	0	1
Aweptimum	0	0	1
alykhantejani	0	0	1
baqwas	0	0	1
anilkumarvvss	0	0	1
asp616848	0	0	1
hiwothadush	0	0	1
junwha0511	0	0	1
ramonhollands	0	0	1
immusferr	0	0	1
syedhamzamohiuddin	0	0	1
beatsea20	0	0	1
panhu	0	0	1
Non1187	0	0	1
Sachi-27	0	0	1
bossebandowski	0	0	1
SakshiFadnavis2003	0	0	1
damjandakic93	0	0	1