Add new rule for monitoring performance alert resolution changes. #2535

gmierz · 2024-11-05T22:04:10Z

This PR adds a new rule to monitor perfalert resolution changes. It will send out an email which provides a summary of all the performance alerts which have had their resolution changed within the last day. It includes the resolution comment, and author in the email as well.

bugbot/rules/perfalert_resolved_regression.py

templates/perfalert_resolved_regression.html

bugbot/rules/perfalert_resolved_regression.py

gmierz · 2024-11-14T15:22:36Z

Example run for the email:

****************************
* DRYRUN: not sending mail *
****************************
Receivers: ['[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]']
Message:
Content-Type: multipart/mixed; boundary="===============4356218612956373498=="
MIME-Version: 1.0
From: [email protected]
To: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: [bugbot] PerfAlert regressions whose resolution has changed recently for the 2024-11-14
Cc: 
Bcc: 
X-Mailer: bugbot

--===============4356218612956373498==
Content-Type: text/html; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

<!DOCTYPE html>
<html lang="en-us">
    <head>
        <meta charset="utf-8">
    </head>
    <body>
        <p>Hi there,</p>
        
        <p>The following bug list contains performance alerts whose resolution has changed in the last week:</p>
<table style="border:1px solid black;border-collapse:collapse;" border="1">
    <thead>
        <tr>
            <th>Bug</th>
            <th>Summary</th>
            <th>Status</th>
            <th>Resolution</th>
        </tr>
    </thead>
    <tbody>
        <tr bgcolor="#E0E0E0"
            >
            <td>
                <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1930747">1930747</a>
            </td>
            <td>
                <table>
                    <tr>
                        <td>8.62 - 7.33% cpstartup content-process-startup / cpstartup content-process-startup (Linux) regression on Thu October 31 2024</td>
                    </tr>
                    <tr>
                        <td>
                            <b>Resolution Comment:</b> Unfortunately we likely have to accept this regression, because we shouldn't keep deferring the update to a more recent ICU. Otherwise we run into cross-browser compatibility issues (e.g. bug 1909112 or bug 1919905) and we run behind in sec-updates (e.g. 1871635, 1871636, 1871623).
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <i>Resolved by [email protected]</i>
                        </td>
                    </tr>
                </table>
            </td>
            <td>RESOLVED</td>
            <td>--- -> WONTFIX</td>
        </tr>
    <tr >
            <td>
                <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1927052">1927052</a>
            </td>
            <td>
                <table>
                    <tr>
                        <td>2.07 - 1.93% speedometer EmberJS-Debug-TodoMVC/CompletingAllItems/Sync / speedometer EmberJS-Debug-TodoMVC/CompletingAllItems/Sync + 3 more (Windows) regression on Sat October 12 2024</td>
                    </tr>
                    <tr>
                        <td>
                            <b>Resolution Comment:</b> The update is no update -- and realistically, WONTFIX.
                        </td>
                    </tr>
                    <tr>
                        <td>
                            <i>Resolved by [email protected]</i>
                        </td>
                    </tr>
                </table>
            </td>
            <td>RESOLVED</td>
            <td>--- -> WONTFIX</td>
        </tr>
    </tbody>
</table>
        <p>
            Sincerely,
            <br>
            BugBot
        </p>
        <p>
                See the search query on <a href="https://mzl.la/3Z5GPtl">Bugzilla</a>.
            </p>
        
        <p>
            See the source / report issues on the <a href="https://github.com/mozilla/bugbot/">GitHub repo</a>
        </p>
    </body>
</html>

Example run for the needinfo comment:

2024-11-14 10:14:54,268 - INFO - The bugs: 1927120
 will be autofixed with:
{'comment': {'body': 'A comment containing a reason for why the performance regression was resolved as FIXED could not be found. It should provided when the status of the bug is changed.\n\n:dao, since you resolved this bug, could you provide a comment explaining the resolution? If one has already been provided, this needinfo can be ignored/removed.\n\nIf you need additional information/help, reach out in [#perftest](https://matrix.to/#/#perftest:mozilla.org), or [#perfsheriffs](https://matrix.to/#/#perfsheriffs:mozilla.org) on Element.\n\nFor more information, please visit [BugBot documentation](https://wiki.mozilla.org/BugBot#perfalert_resolved_regression.py).\n'}, 'flags': [{'name': 'needinfo', 'requestee': '[email protected]', 'status': '?', 'new': 'true'}]}
2024-11-14 10:14:54,269 - INFO - The bugs: 1929843
 will be autofixed with:
{'comment': {'body': 'A comment containing a reason for why the performance regression was resolved as INVALID could not be found. It should provided when the status of the bug is changed.\n\n:bacasandrei, since you resolved this bug, could you provide a comment explaining the resolution? If one has already been provided, this needinfo can be ignored/removed.\n\nIf you need additional information/help, reach out in [#perftest](https://matrix.to/#/#perftest:mozilla.org), or [#perfsheriffs](https://matrix.to/#/#perfsheriffs:mozilla.org) on Element.\n\nFor more information, please visit [BugBot documentation](https://wiki.mozilla.org/BugBot#perfalert_resolved_regression.py).\n'}, 'flags': [{'name': 'needinfo', 'requestee': '[email protected]', 'status': '?', 'new': 'true'}]}

bugbot/rules/perfalert_resolved_regression.py

Add new rule for monitoring perfalert resolution changes.

a3ab301

gmierz force-pushed the resolved-alerts-rule branch from 52610e3 to a3ab301 Compare November 5, 2024 22:08

This comment was marked as outdated.

Sign in to view

gmierz commented Nov 6, 2024

View reviewed changes

bugbot/rules/perfalert_resolved_regression.py Outdated Show resolved Hide resolved

gmierz commented Nov 6, 2024

View reviewed changes

templates/perfalert_resolved_regression.html Show resolved Hide resolved

Add needinfo to resolved regressions with no explicit reason.

13a19bd

This comment was marked as outdated.

Sign in to view

suhaibmujahid reviewed Nov 13, 2024

View reviewed changes

bugbot/rules/perfalert_resolved_regression.py Outdated Show resolved Hide resolved

bugbot/rules/perfalert_resolved_regression.py Outdated Show resolved Hide resolved

Rework rule to gather comments/history in handle_bug.

78b512f

This comment was marked as outdated.

Sign in to view

Use a start/end date for bz query params.

28320d6

gmierz requested a review from suhaibmujahid November 13, 2024 16:49

suhaibmujahid reviewed Nov 13, 2024

View reviewed changes

bugbot/rules/perfalert_resolved_regression.py Outdated Show resolved Hide resolved

bugbot/rules/perfalert_resolved_regression.py Outdated Show resolved Hide resolved

bugbot/rules/perfalert_resolved_regression.py Outdated Show resolved Hide resolved

suhaibmujahid mentioned this pull request Nov 13, 2024

Automatically retrieve nicknames for needinfos when they are not provided when adding the needinfo #2540

Open

gmierz added 2 commits November 14, 2024 10:18

Rework handle_bug, and email template extras.

acb41eb

Ignore cases when a nickname doesn't exist.

e6b74af

gmierz requested a review from suhaibmujahid November 14, 2024 15:24

suhaibmujahid reviewed Nov 14, 2024

View reviewed changes

bugbot/rules/perfalert_resolved_regression.py Outdated Show resolved Hide resolved

bugbot/rules/perfalert_resolved_regression.py Outdated Show resolved Hide resolved

bugbot/rules/perfalert_resolved_regression.py Outdated Show resolved Hide resolved

Compare times as strings, and change default resolution comment.

4941b59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new rule for monitoring performance alert resolution changes. #2535

Add new rule for monitoring performance alert resolution changes. #2535

gmierz commented Nov 5, 2024

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

gmierz commented Nov 14, 2024

Add new rule for monitoring performance alert resolution changes. #2535

Are you sure you want to change the base?

Add new rule for monitoring performance alert resolution changes. #2535

Conversation

gmierz commented Nov 5, 2024

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

gmierz commented Nov 14, 2024