DST starting/ending causes overlap/gap in schedules

[zackman0010]

What went wrong?

When using the API to create a schedule that has a time zone with DST, there will be a one hour gap in the schedule when DST ends and a one hour overlap when DST begins.

How do we reproduce it?

1. Create a weekly rotating shift
   POST {baseUrl}/api/v1/on_call_shifts

{
    "name": "Test API Shift",
    "type": "rolling_users",
    "start": "2024-10-01T09:00:00",
    "duration": 604800,
    "frequency": "weekly",
    "week_start": "MO",
    "rolling_users": [["SomeUserId"]],
    "start_rotation_from_user_index": 0
}

2. Create a schedule using the shift created previously. A time zone that has DST must be used.
   POST {baseUrl}/api/v1/schedules

{
    "name": "Test API Schedule",
    "type": "calendar",
    "time_zone": "America/Chicago",
    "enable_web_overrides": true,
    "shifts": ["IdOfShiftFromStep1"]
}

3. In Grafana, look at the scheduled shifts for when DST ends (Such as November 3, 2024 for America/Chicago). You should see a one hour gap.

4. In Grafana, look at the scheduled shifts for when DST begins (Such as March 9, 2024 for America/Chicago). You should see a one hour overlap.

Grafana OnCall Version

Cloud (Plugin management says v1.14.1 is installed)

Product Area

Schedules

Grafana OnCall Platform?

I use Grafana Cloud

User's Browser?

No response

Anything else to add?

I found the PR #4103 that says it fixes schedule gaps, but it appears that it only fixed the visual gaps caused by a schedule being in UTC and the user's time zone being non-UTC. In that situation, there is no actual schedule gap because the schedule was in UTC, so the only issue was a visual bug. In this case, I believe there actually is a one hour gap.

[zackman0010]

Looking into it, the best way to fix this requires updating the recurring_ical_events dependency to a newer version, as the version currently in use does not call pytz's .normalize function on the end date, causing it to be an invalid date (ie - Nov 3, 2024 9:00 AM CDT instead of the correct 8:00 AM CST). However, some of the tests are currently failing with the updated dependencies. I'm looking into it now to see if I can tell what exactly changed, but in the meantime I also have an alternate fix that doesn't involve updating the dependencies.

The tests that are failing with the updated dependencies show changed behavior in recurring_ical_events that I'm not sure how to rectify, so I'm abandoning that line of research and will just stick with the fix that doesn't require updating dependencies.

Fix with dependency update (Failing tests): zackman0010@d918955
Fix without dependency update: zackman0010@9d48c88