This alert triggers when the memory usage of a Google Cloud TPU node exceeds a set threshold. High memory usage can lead to out-of-memory errors and potentially cause TPU processes to fail. Customization Guidance: Threshold: Set the threshold to trigger if memory usage exceeds 80% of the total memory available on the TPU node. Adjust the percentage based on the expected memory usage for your workloads. Node Specificity: Focus on critical TPU nodes and workers that are handling memory-intensive workloads or tasks. Notification Frequency: Set alerts to trigger when memory usage is consistently high to prevent out-of-memory issues and task failures. Action: Investigate high memory usage to identify memory leaks or inefficient processes. Consider optimizing memory usage or scaling TPU resources to better manage workloads.

This alert triggers when the CPU utilization of a Google Cloud TPU node exceeds a predefined threshold. High CPU utilization can indicate that the TPU is under heavy load and may lead to performance bottlenecks. Customization Guidance: Threshold: Set the threshold to trigger when CPU utilization exceeds 80% over a 10-minute window. This threshold may vary depending on your TPU workload and expected performance. Node Specificity: Monitor high CPU utilization on critical TPU nodes and workers that handle important workloads or tasks. Notification Frequency: Set alerts to trigger upon sustained high CPU utilization to avoid performance degradation or task failures. Action: Investigate the cause of high CPU load, such as inefficient processing or increased workload. Consider optimizing task allocation or scaling resources to handle the increased load.