"At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second"
Well this all makes sense now. My enterprise laid off most of our QA team and I see a constant stream of rollback announcements. Who needs regression testing anymore?
Feature flags are great. Working on a crash / memory corruption in an optional subsystem? Just disable the subsystem to unblock coworkers on the same branch while you track down the cause.
Feature flags are terrible. Working on a crash / memory corruption in an optional subsystem? You disabled it previously for your local build, and you'll lose hours failing to repro despite QA giving excellent repro steps.
(For my own gamedev background, I learned to mute audio by setting volume to 0 instead of by disabling the audio subsystem.)
I agree with the entire premise but I do think the cost argument is a bit overblown. Adding "unnecessary" feature flags isn't really that big of a deal imo, feature flags are cheap to add and maintain. Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
I think the true cost is that feature flags can cause code bloat and readability issues, since engineers typically aren't great about cleaning up feature flags after things have been rolled out. I think that's an easily solvable problem that doesn't really necessitate a scarcity mindset of "just use less feature flags / only when necessary" though. LaunchDarkly makes it pretty easy to track feature flag usage and remind people to clean up old ones.
That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. Teams need to be in the habit of performing a rollback though. Sometimes, the rollback process can be black magic if the engineer handling an incident isn't familiar with that process. Having a bunch of flags in a system is a great way to end up with nondeterministic errors.
And that brings us to another great point, which is too many flags is problematic. So often there's an excuse made in the nature of, "we'll go back and remove this later," but later never comes.
At the end of the day, it's rigor that separates good teams from bad teams. Good teams will rigorously review old code and remove it; it's all too easy to do the opposite.
"At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second"
Well this all makes sense now. My enterprise laid off most of our QA team and I see a constant stream of rollback announcements. Who needs regression testing anymore?
Feature flags are great. Working on a crash / memory corruption in an optional subsystem? Just disable the subsystem to unblock coworkers on the same branch while you track down the cause.
Feature flags are terrible. Working on a crash / memory corruption in an optional subsystem? You disabled it previously for your local build, and you'll lose hours failing to repro despite QA giving excellent repro steps.
(For my own gamedev background, I learned to mute audio by setting volume to 0 instead of by disabling the audio subsystem.)
I agree with the entire premise but I do think the cost argument is a bit overblown. Adding "unnecessary" feature flags isn't really that big of a deal imo, feature flags are cheap to add and maintain. Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
I think the true cost is that feature flags can cause code bloat and readability issues, since engineers typically aren't great about cleaning up feature flags after things have been rolled out. I think that's an easily solvable problem that doesn't really necessitate a scarcity mindset of "just use less feature flags / only when necessary" though. LaunchDarkly makes it pretty easy to track feature flag usage and remind people to clean up old ones.
I liked the article. Every feature flag contributes to hockey stick growth of version variations of your software.
1 feature flag = 2 behaviors 2 feature flags = 4 behaviors
So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.
That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. Teams need to be in the habit of performing a rollback though. Sometimes, the rollback process can be black magic if the engineer handling an incident isn't familiar with that process. Having a bunch of flags in a system is a great way to end up with nondeterministic errors.
And that brings us to another great point, which is too many flags is problematic. So often there's an excuse made in the nature of, "we'll go back and remove this later," but later never comes.
At the end of the day, it's rigor that separates good teams from bad teams. Good teams will rigorously review old code and remove it; it's all too easy to do the opposite.