Two quarters in the DNF
After being a part of a Python-oriented team developing a CI for automating release syncing and shifting the testing process to left (to the original project rather than at the distribution-level) for five and a half years I felt like I need a bit of a change.
And that’s why I’ve seeked the opportunity to switch up for some time with a colleague that felt in a similar way working on a different project within the Red Hat.
And that’s how I ended up for 2 quarters in the DNF Team working on dnf5. I’ll try to sum up my experience over the 2 quarters.
Agile processes
As of the time of me switching from the Packit Team to the DNF Team, our organization has been in the process of hiring a dedicated Scrum Master.
In the Packit Team we are used to rotating the Kanban Lead role (among the other regular / maintainer roles) around the team.
It was my first time being a part of a team that has a dedicated agile practitioner. I was not used to this experience.
On one hand it feels nice to have someone having your back and handling most of the annoying stuff, e.g., Jira shenanigans, or calendar and such, and, in general, being in control of the whole agile process and ceremonies tied to it.
Sadly I have to admit that as the Kanban Lead role grew in the Packit Team, I have managed to forgotten smaller things here and there, e.g., least recently updated issue (when the stand up took longer than usual), or posting the summary of merge requests waiting for review.
Even if your whole job revolves around the agile practice, Jira is a miraculous creation that will pull the rug from under your feet one way or another…
In Packit we are using a scrumban1, so getting back to scrum kind of made me relive some of the annoyances of scrum by itself, e.g., the time constraint to 2 weeks.
Some time after I joined the DNF Team we have tried out the 3-week long sprint instead of 2-week long.
It had quite a good reception, especially because of the lower frequency of planning meetings.
AFAIK the 3-week sprints have been kept.
In general it is more common to shorten the “iterations” rather than prolong them. Overall I think this impacts your capability of planning, because you’re planning for a longer period of time and that, by definition, is less accurate.
Carrying tickets over to the next sprint is quite common and also pulling in tickets during the sprint too…
Meetings
One thing that caught me by surprise was the low amount of meetings, partially contributed by the asynchronous stand ups2.
Also the DNF team is spread across multiple time zones, so the meetings are set in the afternoon for the European folk.
Story points
As always, estimation of size of tasks is quite relative thing. There are teams that measure the story points by days, weeks, etc.
In the Packit Team we’re using the Fibonacci scale with a few rules that give quite basic idea of the sizing.
- 1 points = less than 20 minutes
- 2 points = recurrent task, manual, similar to 1 point one
- 3 and 5 points = normal tasks
- 8 points = one week and potentially over one week
- 13 points = two weeks (for our former scrum and 2-week sprints = whole sprint)
After a quick glance through the DNF’s backlog I’ve noticed mostly higher points being assigned, which is totally OK, since it’s relative. However it also clashed a bit with my perception of sizing after 5 years in Packit 😁
Another difference to Packit was the fact that the story points were assigned by the colleagues that were triaging the new issues/tickets at the time. Because there was no agreement across the team on the difficulty assessment, the story points vary (IMO at least).
Understandably, both approaches have its merits:
-
when you assign points during triage
- it saves time from the meetings, and also
- you have nicely “pregroomed” backlog with story points almost3 everywhere; this also gives you a general idea of work in backlog (apart from the simple metric of how many tasks are waiting)
-
assigning points together — unfortunately consumes time, but instigates a discussion over the topic
Technical perspective
Developing for Linux on macOS can be challenging at times. One would expect that developing service revolving around Linux packaging and testing would be even worse, but it’s not that problematic in reality, since everything is nicely containerized.
Developing a package manager on the other hand… is also not an issue, unless you need to run the tests.
Development environment
Setting everything up was a bit challenging. In the end I have settled for a Fedora Rawhide container where I mounted both sources and tests4.
Running the test suite based on behave was however not possible… Main cause of failure was coming from the fact there were some tests for testing architecture fields in specfiles which, of course, breaks the assumptions on the Macs. To be fair, the different architecture (Apple Silicon) is to blame, not the OS itself.
In my opinion, BDD is something that could probably simplify Packit Service tests, but at the same time… the time/cost of converting the test suite would be disgraceful.
One of the other ARM5 quirks I hit, was not being able to utilize
parallel builds. When I tried compiling with the classic make -j$(nproc) on
Linux, it worked just fine. However from within the container on the Mac, it hit
some race conditions that caused the build to fail.
First RHEL bug
For the first time in 5 years, I have been assigned a task that affected RHEL directly…
Finding the issue and fixing it was probably the easiest part. Getting the fix all the way down to RHEL is where things complicate.
DNF Team handles the downstream patching in such way that they keep the RHEL branches on the upstream. To these branch you backport (most commonly simple git cherry pick) the fixes and then you can easily generate patches that are added to the RHEL specfile.
The whole process is simple and straightforward. Except… resolving the ticket consists of:
- fix in the source repo and test coverage (2 MRs, because tests are on its own)
- backporting both fix and tests (+2 MRs)
- then actually adjusting the specfile on downstream (+2 MRs; depends on how many releases is the fix propagated to)
Each of those MRs should be reviewed. Even if it has already been reviewed, cause, don’t forget, 2 of those are simple cherry picks and the specfiles, in the end, consist again of the already reviewed patches.
And even then it took me additional two merge requests, because neither I, nor my reviewer, have realized that I am missing part of the tests in the backport. Those have been added during the triage a long time before I got the ticket assigned.
Understandably, this “bureaucratic” work has no easy solution. The current approach is menial and annoying, but you’re guaranteed audit through the whole process. And also it nicely prevents conflicts on the downstream when backporting fixes.
Summary
Overall it was a very pleasant experience trying to onboard a different project with different workflows and processes. Maybe I’m biased prior to my previous experience, but the way we set up agile in Packit feels a lot more intuitive.
I’m also content that I was able to contribute to the package manager that I use on a daily basis as a user.