GitHub Projects experimentation

[jdangerx]

We were talking on the inframundo team about experimenting with using GitHub Projects to track our sprint since Zenhub has been acting up a bit lately.

In the very short-term we'll just buy a few more Zenhub licenses and sort that out, but, we'd like to play around with GitHub Projects and see what works well and doesn't work well for our use cases - then we can present our findings to the rest of Catalyst.

Before we go ahead and do that, I think we need buy-in from the rest of the Catalyst folks to see if this is a direction worth exploring. We also need specific concerns / feature-exploration requests from the other teams, since everyone uses different features of Zenhub.

Currently we know we want to investigate at least the following questions:

• project-based vs team-based boards (or... both? can we put issues into a project-focused board as well as a "here's everything the inframundo team is working on" board?)
• roadmaps
• how to group issues together for epics
  • if project boards are how we organize issues by, well, project, does that group things well enough?
• issue estimation
• how much automation can we do re: sorting issues, putting issues in boards, etc.?

[zaneselvans]

Overview of the new GitHub Projects.

Blog post about the new Roadmap View

[zaneselvans]

Given the new Projects seamless integration with GitHub, the fact that it's free, and what appears to be a much more flexible/powerful underlying design, I suspect that ZenHub is now a zombie product and we'll have no choice but to migrate in the long run. 🧟‍♂️

[TrentonBush]

Project management tools are inextricably linked to the management structure of the organization. I think we should be clear about which we are proposing to change! For example, "project-based vs team-based boards" seems like a change to both. Changes are welcome but I think we should be careful not to conflate them and risk letting tool choices drive management choices instead of the other way around.

[TrentonBush]

By the way here is a nice overview of different management structures, including project vs team (aka functional), and their pros and cons.

[bendnorman]

I don't see having project-based boards that can be tagged by teams that are working on it as a disruption to our management structure. I'm assuming we would maintain our team structure with our transition to GitHub projects.

I'm open to a discussion about our desired management structure. I think our teams structure works well right now but might not scale well.

[jdangerx]

I agree that we should not stumble into a new management structure - thanks for flagging that @TrentonBush !

It sounded like there was a lot of pain around "the boundaries of our work don't line up neatly with our teams" - driven, for example, by the recent XBRL work which spanned the Inframundo and RMI teams. Maybe some people who were actually involved in that can chime in with some concrete pain points. It sounded to like it was hard to focus in on what was going on in the project across multiple teams, which makes sense.

That being said, I think we can kick the "management structure" can down the road - we can continue to work in teams, but figure out how to make GH projects work well for that. Since Projects supports many views onto the same issue database, I think we can alleviate the "hard to see all the relevant things" issue with just "competent usage of the various filtering options."

[jdangerx]

I've played around with GH Projects a bit. Apologies for all the noise.

What I actually did

I made One Big Catalyst Project - with the hope that we could then slice & dice however we wanted.

I added a handful of tickets that I knew about and then added the most recently updated 25 issues or requests from several repos:

• pudl
• pudl-archiver
• pudl-usage-metrics
• cheshire
• ferc-xbrl-extractor
• rmi*

I then tried to label issues by team. A lot of stuff ended up in Inframundo because there's just a lot of "bring this data in" or "something is wrong with something we did" or "there's some maintenance work to do" tickets.

Here's the Inframundo board (filtered by an #inframundo label that I've added to a variety of repositories). We have a lot of things in backlog, but that's nothing new.

There's also an RMI board. I left most of these tickets in "New" because I don't know anything about y'all's project management. (I did tag everything in the rmi* repos with the RMI label, though, so it would show up.)

I also tried a couple project boards: Dagster, XBRL. With Dagster I used a milestone and with XBRL I just used the old XBRL label.

Takeaways

Pain points so far:

• adding new issues is annoying - you have to add a GH action to every individual repo that you want to watch 👎
  • though you could e.g. auto-label everything in pudl-archiver with inframundo and everything in rmi* with rmi, too, with these actions, so that's nice
• can't bulk import existing issues, though the default "25 most recent things" is probably a good place to start. In some sense this basically makes "all issues that aren't actively in a project" our icebox. Not terrible.
• milestones and labels don't link up across repositories natively, but we could have actions do this for us.
• UI is kind of funky - no multi-select actions, drag and drop doesn't scroll the screen, issues show up in a sidebar but prs open new tab

Nice things so far:

• filtering is really easy and powerful! can filter based on issue/pr, draft status, label, etc.
  • so it's easy to have "all the stuff + views for each work team"
• fast!
• you can do project stuff from PR/issue views
• using labels and milestones to filter by team vs. project is nice.
• linking PRs and Issues is nice and clean
• auto-archiving old tickets seems like it could be useful
• draft issues seem like they would be super useful, but I haven't played with them yet.

[jdangerx]

I'm wary of auto-archiving old issues since we often create issues to remind ourselves of a problem that we know needs to be addressed, even if we're not going to do it right now. As a kind of "known issues" documentation. But of course then there are just hella stale issues that aren't relevant any more that would be good to get purge.

So GH projects has a 1,200 active items limit and a 10,000 active + archived total items limit per project. I think the idea is that the issues will still be open and around in the repos themselves - effectively the icebox. It's pretty easy to add issues back to the project through the issue search.

The initial bulk import seems annoying but that's a one-time problem, right? If we auto-import all issues to the catalyst megaproject, and add specific projects to individual issues when we create them they'd end up in the right places. We need to be much better about adding metadata (labels, etc.) to issues.

Yeah, it is a one-time problem. I'm not sure we need to import every GD issue, though, because all issues in the Catalyst org are implicitly available. One of the cool things about the design of Projects I think. Like, I think "auto-add new issues" combined with "add things you are working on" will basically get us to a usable state with the megaproject.

Having a Milestone view for (say) everything we want to get done in Q1 seems like it would be great.

Milestones & labels are both repo-specific, I think we'll have to do some automation with the GH API to sync them across repos. But I think that's probably worth it...

I was using a milestone to track progress in an epic (dagster phase 1) and it seems to work well, but that could also be a label. I think the main differences between milestones and labels are:

• you only get one milestone per issue
• you get a fancy progress bar for the milestone, which you don't get with labels
• milestones have due dates and longer descriptions

These all sort of feel like nice things to have for epic-level tracking, actually - maybe we track epics in milestones and title them [2023 Q1] Do Cool Thing so we can easily see what's going on per quarter?

[bendnorman]

Sounds like we still need to decide on a GitHub Projects Epic equivalent. Our three options are to use labels, milestones, task lists or separate projects. I don't love the idea of using labels because our list of labels will keep expanding and get overwhelming. Milestones or Projects seem like good options. What does Github recommend? Can we create github project views for sub issues within a task list? For example, if the Dagster ETL epic becomes an issue with subtasks can we create a view with all of the sub tasks?

[jdangerx]

So the WIF stuff is across a million repos so I thought it'd be a good test bed for a project:
https://github.com/orgs/catalyst-cooperative/projects/11/views/1

Pros:

• super easy to add stuff to a project from multiple repos!

Cons:

• doesn't sync status with the mega-project - so each issue should really only have one project at a time - otherwise state is gonna get all messed up.

That con means that moving to feature-based Projects pushes us into basically a reorg into feature-based teams instead of function-based teams. I think we should stay away from that topic if we want to get this Projects transition done at all 😅

[jdangerx]

It looks like tasklists don't have any native integration with Projects right now. At least not outside of the private beta.

That being said, we can also add a custom "epic" field to the project. It can be a string (which lets you make typos) or a single select (which will have a very long list of epics eventually). Check out what it looks like here: https://github.com/orgs/catalyst-cooperative/projects/9/views/1?filterQuery=Test+task+epic%3A%22test+epic%22&visibleFields=%5B%22Title%22%2C%22Assignees%22%2C%22Status%22%2C31293530%2C31293531%2C%22Linked+pull+requests%22%2C%22Labels%22%2C%22Tracked+by%22%2C32298922%5D

I think our options are basically:

• milestones that we sync across all repositories (and eventually they say cross-repo milestones will become natively supported...)
• custom epic field (I slightly prefer the single select & dealing with the epic cruft as we ship a lot of stuff)

The second one is less work, but the milestone progress bar is nice and having a milestone description is also nice. 🤷

[jdangerx]

Oh - and finally, if we were to get into the task list beta I think it would be the obvious choice.

[zaneselvans]

Is there any per-project functionality that will be messed by if we keep everything in a single mega-project? Like the Roadmap View? Or can we filter what appears in all the different kinds of views?

Can we add numerical estimates of issue size / complexity in order to be able to track "velocity" and understand whether we're taking on too much work in a given chunk of time?

How do start/end date fields interact with temporal metadata like Milestones?

[jdangerx]

Roadmap view shows whatever issues you've filtered down to, and arranges items based on start/end date or iteration.

We can add a numerical "points" field, their template gave us a categorical "size" field with very cute animal emoji. We can use the points field instead.

I don't know how the milestone due date works with the roadmap view, I can take a look.

[jdangerx]

The milestone due date doesn't appear to show up in the roadmap view - it just looks at the iteration start/end date for now. I wouldn't personally do this, but I think it can be configured to use start/ends on an item-by-item basis.

[zaneselvans]

I can imagine larger projects extending for more than one milestone -- like the Dagster refactor, or the XBRL integration. It is really nice to have the green progress bar building toward a new release! I often find myself looking at the pandas or other package milestones for the next release to get a sense of how much stuff is left until a new version comes out. I wonder if there will be similar public visibility with the new projects.

I'm really excited to be able to have a public project management view so folks can see what we're doing.

[zaneselvans]

Thankfully they've now added the ability to display Milestones as "markers" (along with sprint end dates) on the Roadmap view.

[TrentonBush]

How does GH Projects handle tasks that are in private repos? Can you put both private and public issues on the same board and roadmap view without exposing them publicly?

[jdangerx]

According to the docs, this works the way we would want it to:

If your project includes items from a private repository, people who are not collaborators in the repository will not be able to view items from that repository.

[jdangerx]

For posterity - this is how I have been getting the list of issues to manually add to the project:

> gh search issues --include-prs --owner=catalyst-cooperative --no-project --updated=">=2023-02-08" --state open --json "repository,number,title"

If there was better CLI support for adding projects to issues we could just chain some more CLI commands together - unfortunately gh issues edit <URL> --add-project appears to be for legacy projects.

Additionally, auto-adding issues is for Enterprise Cloud users only. But I think we can find a way around it. We might even be able to use that gh search command above with the access token stuff in the official add-to-project action to make something that always pulls in everything new!

[zaneselvans]

The @catalyst-cooperative/inframundo team has been using the new GitHub Projects for a couple of sprints now, and we've also gotten access to the new Tasklists functionality which is currently in private beta. I spent some time yesterday playing with the new Roadmap view, and I feel like I have a sense of how the different components could map to functionality we need and may be familiar with from our ZenHub usage, so I wanted to share and see what other people think.

The tools that GitHub Projects provides to organize time and tasks are a bit different from what ZenHub had.

• For Time I think the main tools are Iterations and Milestones
• For Tasks (features, functionality) I think the main tools are Issues and Tasklists

Tasks

Issues

Issues are the basic building block of our project management. Discrete chunks of work to be done, hopefully containing enough context that several people could pick them up and run with them if they're at the top of the priority list.

Issues have been extended by Tasklists so that some issues can also be used as containers for grouping together other related issues.

Issues don't have any sense of time associated with them, unless they've been put into an Iteration.

Tasklists

Tasklists add support for hierarchies of issues on GitHub, helping you to keep track of your issues, divide your issues into smaller subtasks, and create new relationships between your issues.

Tasklists show relevant metadata about the contained issues, and provide a checklist-style sense of progress as issues are closed. They also show a little progress wheel when a Tasklist issue shows up in some other contexts which is nice. In addition to containing issues, Tasklists can contain PRs, draft issues, or even just a simple todo-list style task with no additional structure, reducing the overhead for tracking things that we need to remember, but that don't need all the bells and whistles of a full blown issue. They also make it easy to turn those tasks and draft issues into full blown issues when needed.

Tasklists provide the functionality of Epics in ZenHub, and maybe a bit of what ZenHub Projects did (the task grouping, not the timeboxing). Creating a Tasklist in an issue makes that issue a container for tracking other issues: "Tracks" and "Tracked by" fields that can be used to filter issues in the board and table views, and to group related issues in the Roadmap view. This lets us easily see both the Time and Tasks dimensions at the same time. This will work best if we try and put most issues into a Tasklist, but we can still issues that aren't in a Tasklist in the Roadmap view -- they just all get grouped together as orphans.

The temporal extent of a Tasklist in the Roadmap view is determined by the start/end dates of all the Iterations associated with the tasks that are inside the Tasklist.

Time

Iterations

Iterations are relatively small, repeating time boxes, in which discrete units of work (issues) get completed. They correspond to Sprints in ZenHub (and we've renamed them to be called Sprints in our Project).

Iterations can help us focus on the day-to-day work we're moving across the board, so we don't get distracted by the ocean of stuff in the backlog or long-term goals outside of sprint + quarterly planning sessions. A Milestone is a deadline -- an issue could get completed any time before it -- while an Iteration is when the issue actually did get done. If an issue can't be completed in a single iteration, it should be broken down into smaller tasks that can.

You can create future Iterations, and put issues into them if you think they're the next thing that needs to be tackled, but won't get done in the current Iteration, and then review whether that still seems correct during sprint planning.

Milestones

Milestones could be used to organize any collection of issues into groups, but in open source projects theyr'e frequently used to track progress toward a future software release, while Tasklists are designed specifically to group tasks and track the linkages between issues, including nested groups of tasks. Milestones can also have a due date. We need to organize work on timescales longer than a 2-week sprint, and I think Milestones can fill this need nicely. As with Iterations, each issue can only be part of one Milestone. Milestones are roughly analogous to the timeboxing component of "Projects" in ZenHub.

Having Milestones dedicated to a quarterly release cycle (1 per ~12 weeks), or a semi-quarterly release cycle (1 per ~6 weeks) since we want to have more frequent big-picture check-ins would let us organize work into a time-box beyond the scale of getting a single issue done.

Milestones show a very prominent progress bar in the side bar of every issue that they contain, and can help us stay aware of whether we're on top of our longer-term (6-12 week) goals.

Milestones are publicly visible and simple, and will help users see what's in the near term roadmap.

In the Roadmap view, you can also group and filter by Milestone, allowing to see a given quarter's worth of work at a glance, either on its own, or in relation to other quarters.

[jdangerx]

(If you actually click in to catalyst-cooperative/.github#4 it shows a little "tracked by #3" badge but that doesn't appear to propagate to the Projects view.)

[zaneselvans]

Maybe I'm not understanding what you mean about the root / nested tasklists.

I have a top level task list in the roadmap for tracking the archiver automation and inside it a define success metrics for the archivers issue which is a tasklist too. When I group by "Tracked by" it creates two separate groups, one for issue 61 and one for issue 70. It would be niceer if they were displayed nested (maybe that will come later?) but it seemed to pick up both of the tasklists. I have noticed that it takes several minutes for the tracked by metadata to update sometimes.

I was imagining having Milestones for the quarterly plans and having those represent releases, but putting all of the individual issues into them rather than just the Epics. It seemed like some Epics would potentially span multiple Milestones. And having lots of little issues would be nice to make the green bar move gradually over time.

Hard agree that we would need to stay vigilant against just adding a bunch-o-stuff to the current Milestone. Some of that will happen as we refine the work plan or fix little things that come up. It seems like the mid-quarter check-in would be an opportunity to revisit the scope and see if all the things remaining in the current Milestone seem reasonable or not.

[jdangerx]

Ah, I was just too impatient with the root/nested tasklists - everything else propagates so quickly that I figured 1 minute was long enough to wait. Thanks for clarifying!

In terms of putting all the issues into a milestone:

• yes, it's nice to see granular progress
• yes, some epics could span multiple milestones
• we all know "discovering more facets to the work as we go" is a fundamental property of working on stuff, so:
  • we can either accept this and add our new discoveries to the milestone as we go - we'll probably start with a small # of issues, all open, then create a ton of new issues as we discover problems, and then finally start closing the gap a bit. I'm inclined to say that when we close out an epic we should take all the issues we decided not to do and close them out, or put them into The Icebox 🥶
    • this will make the progress bar do weird things, but be more granular, and also take more book-keeping
  • or, we can not add our new discoveries to the milestone as we go and use it for epics only.
    • this will make the progress bar relatively static, but we won't ever make negative progress, and it takes less book-keeping

I think it kind of depends on what the goal of the milestones is:

• are we trying to show the outside world what we're doing? they probably only need the high level view, and if they're curious about progress on each feature they can just click into that
• are we trying to figure out "are we going to hit our quarterly goals or not?" then we might want the more granular view - though maybe a combination of the epic-specific progress + high level quarterly progress could work too.
• something else?

Anyways. I think either can work, and am slightly inclined to the 'less-bookkeeping' option, for that reason only. I am happy to do more bookkeeping if you see value in it!

[zaneselvans]

I see value in having the Milestone represent our current understanding of functionality we think will go into "Release X" (which we've nominally decided to try and do quarterly) even though it will necessarily be somewhat fluid as we lead up to the end of the Milestone. With most of the other open source projects I follow there's typically a final clearout at the end where issues either get rolled over into the next milestone or otherwise disposed of. I.e. at the end of the Milestone everything within it is kind of axiomatically done -- either it's been completed, or descoped / deprioritized / set aside for later.

I think epics can/should probably work similarly. If there's a continuous stream of related work / bigger feature that we're working on then we keep them open and potentially add new things to them. But at some point we either decide it's good enough, or decide that additional work in the same vein should go into a separate container and close it out -- and make sure that any issues in the Epic have either been completed or removed (potentially added to another Epic, or left to the icebox) when we close the Epic.

I think I'd like to be able to see the negative progress if it's there -- and see it as a warning! If we're not steadily approaching done, especially in the 2nd half of the quarter, then we probably need to have a scoping / prioritization conversation.

With the table view and monthly / semi-quarterly scoping check-ins, I think we can quickly go down the list and re-set the appropriate milestone if we think we're not going to get to something, and this would probably be a good practice for making sure we're all on the same page as to what is in and out of the current stack of medium-term priorities.

[jdangerx]

That all makes sense to me! It's easy enough to add something to a milestone at creation time :)

[jdangerx]

Here's an attempted overview of what we've learned in the last few weeks. It aims to cover more ground but be less in-depth than Zane's comment above - @e-belfer @cmgosnell @bendnorman @zaneselvans @zschira please comment & then we can edit this live Wednesday before we try to present to the rest of the org :)

Core definitions

• items: issues or PRs - the basic unit of work in a GH project.
• tasklists: how we can group related items such as for an epic or just for subtasks of a larger task.
• milestones: how we track progress towards our quarterly goals.
• iterations: two-week planning cycles, which we've renamed "Sprints" in our project to better track the Zenhub terminology.
• views: tabs in the main project page that provide different filters & layouts for you to look at items however you want.
• layouts: "board", "table", "roadmap": different visual layouts.

How to migrate a team from Zenhub to GH Projects

1. Click around the Inframundo views to get a feel for how they work. current sprint kanban board; grouped by epic; roadmap view

2. Pull work items into the mega-project. Official documentation is here but you can use this one-liner to grab recently-updated issues:

   > gh search issues --include-prs --owner=catalyst-cooperative --no-project --updated=">=2023-02-08" --state open --json "repository,number,title"
   

   You can then add them to the project from the "Projects" section on the right side of the page:
   You don't need to pull in everything you've ever worked on, just whatever is actually useful from a sprint planning perspective. Since you have access to all repos and all issues, you can easily add more issues to the project when the need arises.

3. Create your own boards:

   1. Add the team field for all the items that belong to your team. The Items without teams view is helpful here. Table view is more compact for mass edits like this.
   2. Click the "New view" button at the far right of all the tabs. Name it after your team and add a team:your-team filter. More complicated filters follow the GH filter syntax.
   3. Make sure items are in the right status.
   4. Tag items with sprints and priority/size if your team does that.

How to migrate in-progress Zenhub epics

You can convert an epic into GH projects by adding a tasklist to the Epic issue. Then you can filter by tracked-by:"catalyst-cooperative/pudl#XXX"

<body of issue...>

```[tasklist]
### Subtasks (where the #XXXX just corresponds to issues in this epic)
- [ ] #XXXX
- [ ] #XXXY 
```

How to use GH Projects to manage your day-to-day work

We don't prescribe a specific sprint process, but in the interest of having clear communication and visibility across the whole org, you should make sure that some metadata is filled out for all your items:

• development status
• team

Other metadata (priority, size, sprint, etc.) is useful, also, but may not be necessary depending on your team's sprint process.

Gotchas

• if you save changes for a view, that propagates for everyone, so don't be surprised when you clobber a useful view.
• views stick around, so if you made a view and nobody is using it, it will just clutter the space. we should just delete these.
• automatic ingestion doesn't completely cover our needs - it can only ingest from a single repository. if this becomes a burden we can easily script it with their API though.
• if you click on the link for an issue, it will helpfully open in a side modal. If you click on the link for a PR it will open in a new tab.
• if you create an issue directly from the projects view, it can be a little finicky about whether the issue will show up in your current view or not.
• there's a 10,000 item limit per project and a 1,200 active items limit - there's an option to auto-archive issues that haven't been touched in some time window, which we can engage
• milestones, labels, and issues are currently repo-specific, but if they're named the same thing, the Projects filters treat them the same.

[e-belfer]

One short suggestion to add here is to use filtering by label (e.g. label:rmi) in the initial adding to teams step in the migration

[zaneselvans]

In the ZenHub (back) to GitHub migration... there's a bunch of metadata that we're going to lose. Like which issues are grouped into what Epics. Do we care? Are there Epics/Releases in progress in ZenHub that we want to replicate in GitHub Projects so they aren't truncated?

[zaneselvans]

Oh we should update our existing Epic template to use Tasklists.

[zaneselvans]

There's 3 different kinds of views that I've really appreciated:

• The kanban board for actually working through the sprint.
• A big table view for Sprint Planning that's really information dense and lets us change fields easily.
• A roadmap view that lets us look at what we're doing and what we've done on a quarterly scale.

[jdangerx]

Thanks for the comments all! I think they're mostly incorporated into the post. We can spend some time to share out either at the Monday meeting or earlier.