The best branch model for AI augmented software teams.
I recommend upstream-first strategy to simplify release management and enhance team efficiency.
TLDR: I propose upstream-first, with backport-action is the one true way to manage release branches for any software team who’s graduated the GithubFlow.
OpenAI Codex is an active member of our team: CRUD operations, writing the failing test case for a suspected bug (red/green refactor), Codex is a great teammate for knocking stuff out. I find myself, and encourage my team to orchestrating system state more than coding now.
A key aspect to unlocking this orchestration is a source code branching strategy. A solid branching strategy defines your testing and release process and if your branch model doesn’t allow for smooth hot fixes, it will be exhausting to take builds through regression testing. Especially if you have any hardware in the mix.
There are a ton of branching models out there for git. In fact, the presence of so many models is a testament to git’s flexibility and merging capabilities (git has about 6 merge strategies.) If you are old enough (I mean experienced enough) to have used CVS, Subversion (SVN) or even ClearCase, you’ve no doubt said to yourself, “…forget this merge, I’ll just re-code it.” Indeed, git has been a game changer for teams and individual developers alike. However, unlike centralized version control systems, git does not have a single recommended branching method. Over the course of my career, I’ve used 3 distinct branch models: Git Flow, GitHub Flow and Upstream First. Until recently, git flow has been my default for any team larger than 2 people. For 2 people, the efficiency of “github flow” (https://githubflow.github.io/) is hard to beat. However, when it comes time to graduate from simple GitHub flow, I now recommend upstream first.
When do you outgrow Github Flow
In github flow, everyone works on a feature branch based from main/master. Feature branches merge to main, and main is deployed to production. It is simple. It creates focus. It creates a situation where everyone is working on the same system state: database migrations are easier, testing is straightforward, etc.
I recommend engineers get in the habit of recognizing the system they are operating in, and the transitions of that system state. Sometimes that system is code, sometimes it’s team/people, sometimes it’s data. State Transitions are the key.
However, as soon as you need to manage multiple releases, things can be difficult. At Mythia this happened when we shipped a mobile app; When we release builds to the app stores, the builds go through review, then eventually customers update. This is a state transition of our day-to-day operating process. Suddenly, we didn’t control when a particular frontend build would reach customers. While, Mythia uses over the air updates, binary compatibility for plugins, or other native layer components have to be managed separately. How we manage that as software engineers is called release management, or configuration management. Your branching model directly affects the options available to you for that process. GithubFlow breaks down at this point. Since the feature branch has to be merged to multiple release branches, and separately integrated to each release, the merges can be quite tedious to manage.
When gitflow works
Gitflow is incredibly flexible. It allows you to assemble builds of various configurations, shipping hot fixes and patch releases ahead without forcing one to ship long-term feature branches that aren’t ready.
However in the last year, one of my team leads transitioned her development process to the “upstream-first” model. I was skeptical at first, but after using upstream-first I learned how incredibly straightforward it is to automate!
This automation alone alone has kept the merge conflicts to a minimum, and allowed her team to effectively manage releases at a high cadence without a great deal of release management overhead. I’d first like to discuss the challenges of GitFlow so we can fully appreciate how upstream-first can be efficiently deployed.
Challenges in GitFlow
The biggest challenge I’ve experienced with GitFlow is automating it. Each feature branch gets merged to develop, but which feature branch should develop be upstreamed to? At larger organizations I’ve worked with delivery managers who pair with the development team to identify and manage the release merges, however, when the team gets busy and the release branch misses some merges, the ordering of hot-fixes and merges can construct some truly challenging merges. However as software teams become more streamlined, meetings and planning are just wasted time we no longer have. Instead the process should define the next steps, and automate them. Hence, my recommendation is not a technical one, but a people process one. By simplifying the processes we can still apply software product line principles, while automating much of the management. While GitFlow can be automated, frequently it requires “exceptions”, which is almost worse than no automation.
Upstream First is simpler & automatable
Upstream first instead has a mainline branch. Feature branches are merged to the mainline, and tagged with the release branch they merge directly to. We then apply backport-action to automate the merge to the release branch or branches!
At Memo we consistently had 2 parallel release branches. The features set for that build were unchanging, but the builds themselves were pipelined such that each was in different stages of test completion (you’re using a test-case-management-system right?). Bug fixes then needed to be merged to each release branch so the fixes were applied to main first, then back-ported to release/x and release/y.
This doesn’t sound very agile…
In practice, once projects grow a bit bigger, projects with deep manual workflows, or specialized hardware testing, or sometimes you just need to see how a feature “feels” before you get it further down the road, the ROI of automating the test may not be present. In these cases, I’ve seen a staggered approach with testing team and software team. Teams “cuts the release” or “snaps a line” to represents a fixed set of features, new and regression that need to be executed. Also, it gives marketing and sales teams a heads up as to what high-confidence features and bug fixes will be shipped in the coming period.
Testing continues
Testing can continue into the test plan against this next fixed release. When bug fixes are discovered, developer’s fix them on the mainline, and label the release with the target release branches current in test phase. Sometimes at this point a bug is discovered in product already in the field. In this case, if it’s important enough to ship a hot-fix, target that build as well in upstream first it’s simply another label added to the PR.
Conclusion
Upstream first this allows engineering teams to maintain concurrent release branches, while keeping a simple automated strategy for bug fixes during the testing phase. Instead of a meeting, and a deliberation of what branches a particular feature needs to be merged to, It’s a simple set of labels added to the PR. PR’s back port, get reviewed, and final CD takes over to ship those features out to the testing team.