Toggle All Things!

(Note: this post was started quite a long time ago and has lived in draft mode whilst I intended to provide a concrete code example. I'm not sure when, if ever, that will come, but I think that the lessons learned are still valuable to share)

It's not an uncommon workflow: the client / company wants some features in a certain sprint and the features have to be signed off by a QA / testing team. So if we take the example of a two week sprint, what often happens is that the testers sit idle (or are tasked onto a different project) for the first week, while development happens at a furious pace, then testers are given domain to test the week's work and we enter a development freeze, where developers are unable to work ahead for fear of interrupting testing and must be on standby for any bugs picked up by QA.

Hopefully, if you're testing your own work proficiently, QA picks up nothing, or perhaps some small stuff like a misaligned label or similar. However, for that time period, you're stagnant -- which causes you to work harder when the first week of a sprint comes around again. It's not a healthy place to be in, if, for no other reason, than because it provides ammunition for the "production line" mindset and guilting you into working longer hours during the "productive" part of the sprint. And if you were on top of things, the business is also losing during the development freeze because you aren't free to forge ahead with new stories.

No-one is winning.

One possible strategy is to adopt a "feature branch per developer" strategy with a regular "merge day" where some poor soul attempts to merge the disparate work of the last two weeks. You can minimise the pain with more frequent merges during the active development phase, but whilst you're in development freeze, you're going to drift and merge hell is eventually inevitable.

One of the most useful strategies we've used to date is that of feature toggles, which, simply put, are mechanisms for turning parts of your software on or off. So here are some thoughts on that particular topic, played out over the development cycle of a SPA web application.

1. A simplistic approach: compile-time toggles (ie: releases with feature-sets)

One method is good old #if / #ifdef and compile-time constants. Bake those features in and release a version with a specified feature matrix. This works well for software where the required featureset is unlikely to change, such as for Gentoo packages, where your USE flags determine the features available to you at run-time until such time as you decide that you need more or fewer features. As the owner and maintainer of said Gentoo system, you decide what you want from your software and you compile it that way. When your requirements change, you update your flags and re-compile.

This works well when the user / tester is the developer. But not as well when the user/tester is someone else -- because then they have to contact the developer and request a rebuild and redeployment every time a feature needs to be turned on or off.

It's not an invalid strategy -- it's just not very agile. Additionally, you can't easily a automate testing (unit / integration testing) against features, because doing so requires changing compile parameters.

2. Better: static values in the code

Another version of this is compiling with static / constant values and branching in code. This allows testing against the different features (especially if you're using static values instead of constants). This alleviates the automated testing conundrum but still leaves the human testing department out in the cold. They will still contact you for a new build containing the required feature matrix.

Rebuilding and redeploying to satisfy a required feature matrix becomes impractical the more features you enable and the more agile you'd like to be. Suppose you think feature [X] is ready, but 1/2 an hour before sign-off, testing finds a defect. Now you have to rush to provide a deployment package without [X]. In addition, testing may not yet have had the time to prove that there are no unintended side-effects from not including feature [X].

3. Even better: run-time toggles determined by app / web configuration

By now you've realised that the toggles should be configurable and they have been pushed them into your web config (for example). Great! Toggles can be changed fairly quickly, without requiring a re-deployment. The problem is that you still need someone fairly technical (and who has access to the staging machines) to update the enabled feature-set.

My experience has been that if you can make the toggling of features relatively quick and painless, you can offload the decision on which features are in an accepted release to someone in testing or management. You don't want to waste your time and resources on fighting for feature inclusion / exclusion -- you want to focus on making new things, solving new problems! So whilst feature toggles in your web config are better than compile-time, you could push this just a little further for a lot of win.

4. Best: run-time toggles determined per user interaction

In the case of a web system, this would mean "per web request". Imagine if the testers had the ability to test just the stuff they were confident in, and, if they had time, could toggle on a beta feature and test if they have capacity. Imagine if, when a flaw is found in the implementation or design of a feature, it didn't have to hold up the entire release -- just toggle it off, continue testing other features and sign off what is available.

Our implementation was system-wide, re-evaluated per web request, but you could even push this further with per-user configuration included in request headers. Practically, this may be more effort than it's worth.

Features of good feature toggles

• It should be easy to toggle features and difficult to end up with a set of conflicting behaviors from the application. 
• It should be easy to determine the feature-set state of the application and difficult to be confused about interactions which cause experienced defects. I'm suggesting that if you have old feature [X] and new feature [Y], which replaces [X], there shouldn't be two toggles ("enable [X]" and "enable [Y]") -- whilst this technically solves the problem, you're expecting testers to understand the repercussions of the ability of enabling both. Instead, you should strive towards well-defined feature toggles which explain what they do and provide only two well-known states (the "off" and "on" states) which cannot be confused by interaction with other parts of the system.
• It should be as frictionless as possible to develop against (adding features and consuming them) to encourage using the framework so that the agility of feature toggles is experienced as often as possible. More toggles are better than fewer toggles, especially for non-inter-dependent features.
• Toggles should be easy to remove as well: once QA is satisfied with a feature, the toggle should be removed along with any branches of logic which disabled the feature.

This can all be achieved, and it's really not that tricky. Here are the steps we went through:

A. First pass: development freeze and the rush around sign-off time sucks. We need to be able to add new features which can be easily disabled!

So, first off, we define a class / interface which has boolean toggles on it. And we get the web app to put persist into our storage (sql, document db, whatever) on startup, adding toggles which are missing. We save three versions of this configuration, for three proposed feature-sets: "development", "staging", "production" and we let annotations on the configuration class determine default toggle states for features.

For example, we create a feature toggle with the feature defaulted on for "development" and off for "staging" and "production". In this way, developers on the team default to seeing the current work (and interacting with it) and the feature isn't inadvertently introduced to QA or the production servers. Of course, the moment QA is ready, they can manually enable the feature. Production only gets the feature enabled when a deployment to production is explicitly made with a script to enable the feature.

We allow selection of a feature set ("development", "staging", "production") in the web config, because this is unlikely to change on a deployment target, so probably won't need human interaction (let deployment scripts take care of this) and surface these toggles to the user via a simple UI. The UI itself is toggled with a feature toggle, so the production site doesn't have it at all -- no way for a user to enable a feature they shouldn't or disable a feature which should be there.

This process also makes adding / removing feature toggles trivial for other developers on the system.

Consuming toggles, server-side (back-end developer):
We could make the decision about defined feature classes to inject by manipulating the IoC container -- inject implementation "A" of interface "I" or, if the toggle is enabled, inject implementation "B". This may work with differing degrees of efficiency according to your IoC container and you can end up with two implementations which are incredibly similar, but which differ only in one minor aspect.

Alternatively, components of the system could, perhaps, request feature toggle configuration via IoC injection branch subtly as needed. We make sure that anything which needs feature toggles has a per-web-request lifetime (as does the feature toggles injectible) so that the web app doesn't have to be restarted to bring a toggle into effect

Consuming toggles, client-side (front-end developer):

The current feature toggles are exposed as a calculated stylesheet where disabled features literally have their display set to "none !important". So when the page loads, incomplete features are not available for interaction. On feature toggle, this stylesheet can be reloaded to get instant UI feedback of toggled features without even a page reload.

Toggling toggles (user):

Initially, toggling a feature simply toggles the boolean in storage and re-requests the featureset stylesheet, immediately updating the UI. Calls to the api get a new instance of the feature toggles entity and can change their behavior accordingly.

Typically, a "big hammer" approach works well: a controller action returns null or an empty collection when the toggle is off or calls into defined logic to produce a calculated result when the feature is enabled. Sometimes, a finer chisel is called into play -- some component further down the logic chain subtly changes behavior based on the toggle.

B. New features may replace older ones -- we need a toggle to turn one bit on and simultaneously turn another off

We already have this capability at the server -- we can branch code according to an injected feature toggles matrix configuration object -- but at the client, we are just hiding UI elements which expose disabled features.
Thus we added in more client-side logic: we expose a Javascript blob in a generated js file with the feature toggle matrix and emit an event when this is re-downloaded.
To ease developer consumption, a framework function is provided to tuck away the complexity of listening for the correct event.
Also, whilst we've already had calculated stylesheets to toggle incomplete features off, we can now add in calculated stylesheets to toggle fallback features (ie, legacy feature) back on again.

Winning at features

The net result of employing this tactic has meant that we've managed to pull out far in front of the testing team which used to drive deadlines. Simply put, we're able to work on sprint [N+1] whilst testing is signing off sprint [N], such that we've essentially ended up a sprint ahead and had the breathing-room to address technical debt -- a win for everyone!

Feature toggles provided:

• A way to alleviate stress for testers and developers: if a feature isn't deemed finished by sign-off time, it's simply toggled out for deployment
• A way to alleviate stress for developers: whilst features for this sprint are under human testing, developers don't have to sit idle -- they can move forward with the next set of requirements.
• A way to alleviate stress for project managers: it's far easier to say "we managed to get 4/5 of the required features deployed" than "we couldn't deploy because one feature couldn't be signed off"

Interactive, per-request feature toggles meant that:

• Testers could test at their own pace, enabling features on their environment when they were ready and disabling if they thought that those features were interacting negatively with others or if they considered them incomplete at sign-off time for the sprint.
• Testers can preview a feature during development to give early feedback to the developers if they have capacity
• "What-if" conversations could be had: "what if feature [X] was enabled for the users without [Y], which appears to be a little dodgy right now?"
• There was no rush at deployment time for getting "just the right build" to deploy: simply deploy the latest and toggle off the features which aren't considered fully-baked.