I disagree slightly, in that I don’t see Agile as a way to ‘manage stuff’ or a ‘development method’, To me, it’s a philosophy of focusing on the bits that matter and getting things out there quickly. I think that’s very much compatible with UX; sure, we need to adapt our toolset, but it’s doable in many circumstances (not all). Not sure if you read my ALA article, but it gives a stab at some techniques I’ve found useful: http://www.alistapart.com/articles/gettingrealaboutagiled…

Anyway, what really interests me is your last question: “How do agile development methods measure user experience?”

Good question. I’ve actually got a proposal in for the Agile2009 conference about this. My experience of the status quo is that, basically, UX doesn’t get managed. In a lot of so-called Agile projects, UX seems to live ‘off the grid’, with design activities largely separate from development.

I don’t like this approach, since it limits collaboration and misses the opportunity to demonstrate why UX design is as important to the project as development. After all, much of our work is intangible and I think we need all the help we can get when justifying our activities and RoI.

So I’ve been thinking about and experimenting where possible with things like exposing UX estimates at iteration planning sessions, including design progress on burndown charts (although is design really reducible in this way?), and even setting up semi-formal acceptance criteria via quantitative testing per iteration.

But I think it’s easy to go too far, however. I’ve worked in environments similar to Doug Bowman’s recent Google experience [http://stopdesign.com/archive/2009/03/20/goodbye-google.h…] where A/B tests and multivariate testing have a stranglehold over the site’s evolution. For obvious reasons this is antithetical to good user experience. So I think, if we are to measure this stuff, we need to do it in as lightweight a way as possible. I also worry whether over-zealous management can hamper the formation of a coherent design vision, and whether measurement affects velocity itself (an Agile version of Heisenberg’s Uncertainty Principle, if you like).

Be interested to hear your thoughts (and others’) – as you say Austin, this is an important question and the sooner we can find some answers, the better.

I’m buying Austin’s one sprint lag. The argument over the creation of story cycles >1 can actually be pretty simple: “Okay,we can get the story cycle number back down, just add 4-6 weeks to the sprint.” Nobody’s going to want that.

There is one line in Austin’s new post that I think is a little hinky. When he says “Design and architecture CAN’T be managed using agile methods” I think an absolute statement like that wastes the most useful commonality between design and agile development, that they are both iterative. That’s not a minor point.

In an agile environment, some part of the design solution should be flexible enough to allow for things that are discovered from one sprint to the next. This requires making an educated guess on the overall design so as not to hold each sprint’s development hostage to that design.

There is one thing obvious about UX and Agile: they are both iterative and incremental with frequent feedback. So why is making them work together so subject to discussions? My vision is that we expect different things from a Design iteration and from a Development iteration.

IMHO, when doing design your first expectation is to fail, you look for things users get stucked with. Then you try to make them even more happy with better design, but the minimal goal is to make users succeed in their tasks.
On the other hand, when developping running software, your first expectation is to make it work as “specified”. But if the specifications are “wrong”, you develop the wrong software. The Agile answer is: “Make it work, show it, and correct.” That’s the real revolution with Agile (I’m an Agile evangelist), but I think it is creating waste that can be avoided.

By doing Design upfront to the implementation, you can fail for cheap (e.g. with paper prototyping, low-fi prototying and quick user testing). You waste very little when you are wrong, this is the best aspect of a good and efficient Design process. So when you eventually implement your (iteratively and incrementaly) refined prototype, you minimize the risk of being wrong. You don’t expect to be perfectly right, but only to avoid major mistakes or misunderstandings of users needs. If you are still wrong—which is very probable—you can quickly adapt, and this is where Agile rocks.

So IMHO doing design upfront Agile iteration is the obvious. The only question is how and when? On my current project—huge internal software just starting—we are producing the product backlog from the results of design prototypes (which vary in fidelity depending on the complexity of the problem to solve), and the implementation iterations will start with a strong confidence about the users’ context, tasks, environment, and in UI principles, all attached to the User Stories.

We are not implementing yet (usual poticial issues in big companies are slowing us down in the process), but the first results with our Design process has revealed big requirements mistakes and we upgraded them from the start. We have some mid- to hi- fidelity prototypes which make users understand and approve our understanding of their needs and our proposed solution. I keen on seeing how this way of doing performs when implementation has started.

I’m concerned with Dave’s response to staggered UX/dev work. I did some reading on the “Staggered Iterative Waterfall” concept, and there are some good points out there:

* Splitting work across iterations makes it harder to adapt to change requirements * It’s harder to track “doneness” at the story level * Reduces opportunities for collaboration

That said, staggered design/development still seems to be one of the better approaches to allowing time for design efforts. I would love to hear more of Dave’s thoughts on this.

As for measuring UX work, I agree with Cennydd that it’s an important topic. Without a way of assessing doneness at the design level it’s hard to track burndown, and hard to transition from design to development. When we don’t use UX-centric acceptance tests, we tend to leave design evaluation till it’s too late to change things easily.

Lastly, I’ve tried tracking UX work separately from from dev work, including UX estimates on stories. The problem is that it’s damn hard to figure out when a design is done, and even harder to figure that out ahead of time. More often than nought we ended up estimating how long the deliverables would take to produce instead of the real time it would take to figure out the design.

As usual, Austin, you are correct – I do disagree with you : -)

Thanks to Cennydd’s comments above, I do however not need to elaborate much on why, except to say that I largely agree with him, with a couple additional notes…

Lest we forget, Agile was originated back in 2001 by a group of enterprise software developers for the purpose of finding common ground across several light-weight development methodologies. In other words, they had their hands full trying to agree on the software side of things, not to speak of the UX/design side (and at least a few of those who were there lamented that other disciplines could not have been represented).

That said, while I absolutely agree with Cennydd that Agile is more than about management—way more in fact—and that it really is more of a philosophy, what is perhaps even more important is that it is a work in progress, which is why I prefer the term used by several of the original signatories, referring to Agile as a movement. (I find Jeff Patton’s definition of it as a ‘quality’ to be bit mushy.)

The point here is that the ideas underlying Agile can and should be applied to Agile itself – Agile ideas themselves are in a state of continual refinement and evolution, particularly as we now are exploring ways to unify Agile and UX practices.

So when we talk about that Agile means this or that, I think we are confining ourselves to some unhealthy orthodoxy. So, should design and development happen in the same iteration or should they be uncoupled? There really is no correct answer to this question because it will always depend on the particular situation.

That said, in my experience, trying to confine the UX work to the same iteration simply has not worked or made sense. I’ve talked to several other Agile/UX folks who have had the same experience, and would say that UX needs to be uncoupled from Agile, with parallel iterations, in which UX is one or two iterations ahead (though getting too far ahead can lead to a different set of problems.)

In other words, while I disagree re. the mgt thing, I actually think we agree on most other fronts.

“Keeping UX and development in synch assumes engineering and design are the same kind of work. This isn’t true. Holistic, contextualized user experiences require time to frame and synthesize the experience. This synthesis happens long before you can start coding features.”

Web Product Manager speaking here:

I absolutely agree that quality user experiences take time to frame, synthesize, and create. I wholeheartedly disagree that engineering and design are different kinds of work, and that synthesis must happen before code is written. It’s a product of old school engineering “measure twice, cut once” thinking. That just doesn’t apply on the Web.

We need to stop treating code like it’s an asset. When we stop seeing code as an asset, we can start to see developers as designers and manage them as such. No one freaks out when a designer throws away a prototype or produces multiple versions of an interface, but that’s not the same for developers. Both designers and developers do research, prototypes, tests, and iterations, and both are completely worthless on a tech project w/out the other. Doesn’t anyone else think it’s odd that we seldom hear Creative Directors debating project management process, but developers argue about it constantly?

So what do we do? Segregating design from development is at the heart of the problem, not the solution. I believe we need to treat developers like designers and get them involved in the process as early as possible. Have them work alongside designers while prototyping, commenting on technical feasibility, involved in user testing, and generally there synthesizing the experience. Then we let them code early and refactor often, and put the focus on the creation of features that make for quality user experiences instead of code as an asset. As for specific project management methodology to help keep stuff on track, it’s simple. Take some pointers from Scrum and let the folks doing the work give real input into timeframe and scope, then make them commit. I promise it’s much easier to get commitments from people who really understand what the goals of a product are than those who are fed designs and just told to build it.

That said, user experiences and product quality are not measured within the realm of project management methodologies or philosophies like Agile. Project management focuses on delivering projects on time w/ constraints like time, resources and scope. Creating something on time and w/in budget has little to do w/ whether or not the product is any good or should even have been built in the first place. Measuring user experience is specifically about quality product design. Something that, in many cases, can’t be known until well into the product lifecycle. Basically, quality UX is a goal and project management methodologies are there to manage the means.

Austin, I’m quite interested in this post and how you’ve thought about the de-coupling of the UX work from the development iteration because I tend to think of my UX work as an integral part of agile development cycles. In practice however, I realize some the UX pieces are often out in front of a development phase. I guess that’s the nature of the beast.

I’m grateful for your work with Livia Labate on the UX Healthcheck (http://uxhealthcheck.com/)
Do you feel it can be used in Agile projects to help managers measure and monitor the UX quality over time?

If we are exploring ways to unify Agile and UX practices, why wouldn’t you mention the UX Health Check as something we should consider, or at the very least think about adapting for agile practitioners?