As I’ve been developing Zed9, and reflecting on what we’re doing at Replicate, I’ve spent a lot of time trying to make sure I’m building the “right” thing.  So what’s right?  One theory is “Build it for yourself”.  As recently discussed by Tim Bray, John Gruber, and 37 Signals, the basic idea is: unless you’re a very strange individual, make a product you use all the time and chances are others will too.  You can focus on what you know, and make something great.  Build for yourself suffers from survivorship bias - it’s a necessary and helpful precondition, but far from sufficient.  There are countless products built for their creator that never made it anywhere.  The very fact that you have the skill set to create a product to solve your needs means that you ARE NOT LIKE the vast majority of your potential user base.  Using yourself as the prototypical customer can lead to dangerous assumptions.  What’s easy for you isn’t for others.  What’s obvious to your user could be totally obscure for you.

In creating any product, there’s always a disconnect.  You, as the product manager, startup founder or whatever, are either trying to solve a product you’ve personally experienced, or one you’ve seen.  Either way, it’s based on your experience.  And as much as we all like to scratch our own itches, selling to yourself is a hard way to make a living.  Many times you’re not creating this product to make a living, it’s an artifact of some other focus.  That’s certainly what happened with Rails, and I believe for Gruber with Markdown as well.  And when that happens, GREAT!  We all love the serendipitous success.  But what happens when you’re setting out from the start to actually make money on your product?  

That’s where the MVP comes in.  MVP stands for “Minimum Viable Product“.  Closely related to agile practices, the idea is put together the real minimum product to get customer feedback, and use that to validate and move forward.  Sometimes, your MVP can be as simple as a slide deck or even just an adword.  If people click and sign up for a waiting list, it’s probably a good sign that they’re interested, and it’s worth pursuing.  Don’t build a 2 month alpha if a prototype will suffice.  Don’t build a prototype if a marketing landing page will suffice.  The key here, of course, is the word “suffice”.  The MVP is all about getting customer feedback.  And apathy is the death of any feedback process.  We know what to do with negative results (try something new!), with positive (do more!), but what about no results?  

Apathy is really the scariest thing that a product owner can experience.  The lack of feedback gives us nothing to hold on to.  We start to breath our own fumes, going in circles.  What are the main causes for apathy? Three stand out: 

• Not a large enough sample size. We’re aiming for early adopters with any MVP based feedback.  If we’re lucky, they make up 5% of our target population.  Depending on your marketing and engagement practices, this means you might need to kiss 1000 frogs just to get 10 qualified responses.  Let’s take the ad-words example.  Your ads have a 2% CTR, and you expect 10% of the visits (a high number) to translate into actual feedback.  To get 10 actions, that’s 100 CTR, and 5000 impressions. Want 100 user feedback base?  Now you’re talking 50,000 impressions!  Dealing with contacts?  Assume you can get a 15% conversion rate and you’ll still need to talk to 66 people just to get 10.  This is probably one of the biggest issues many startups face.  From day 1, you need to be VERY aggressive about talking to as many people as you can to increase the population base.
• Didn’t actually make a minimum product. This is probably the first place any engineer will go.  ”Clearly, if we just add feature X and Y, THEN they’ll understand what we’re doing, and give us feedback”.  This is the most dangerous path to go down.  It’s the one where you eventually throw away the MVP, because you’ve never satisfied with the minimum.  It’s one I’m personally struggling with on Zed9 right now in fact.  We’ve launched a minimal product, that at the least gives the user some interesting rudimentary comparative analytics.  It’s different from other offerings, but not earth shatteringly so.  It hints at where I’m taking it.  I’ve had some interest, but not droves beating down my doors.  I keep thinking “if I just add this feature, then I’ll get 100x more people interested”.  This way madness lies.  You need to set clear metrics, and hear from prospects.  Go back to step 1 – not a large enough sample size.
• Not solving a broad/interesting problem. At some point, you need to call it.  Remember, the point of an MVP is specifically to find out IF IT’S GOING TO WORK.  It MAY NOT.  Hell, it probably won’t.  If you’ve talked to enough people, and still not getting a decent response, it just might be time to move on to the next idea.  If you keep going, you’re not building a product, you’re satisfying a hobby.  A hobby that may well turn into something interesting, but for now a hobby.

For me, I’m redoubling my efforts on increasing the sample size.  Before I spend a few more long nights and weekends adding all these cool features, I’m going to go out into the field, and talk to people.  I’m heading out to some bike and running stores during the weekday, to talk to the sales guys and see what HW is moving, what tools they recommend, and their thoughts on the viability of the product.  I’ll be joining some group rides, checking out the local meetup groups, and just talking to people about their problems today.  I’ll let you know how the next few weeks go!

After all these posts on agile process and tools, if you’re interested in hearing live how I’m using Jira and Greenhopper, then sign up for this webinar Wednesday.

After introductions from Atlassian and Pyxis, I’ll be talking for 15 min on how we’re using them here, on our live dataset.  I’m talking about why we choose Jira/Greenhopper, how the entire team uses it, and what some of my favorite features are.

We’re at that stage right now where the whole team is clicking.  Every 24 hours I’m seeing improvements that blow my mind away.  The more we get done, the fewer items left for us to do, the more I’m looking back and realize how TOTALLY INSANE my expectations were when we started.

When I first joined Replicate, our founders had already spent a bunch of time on the core databae and inference engine that drives everything we do.  Their up-front investment is what has enabled us to get to where we are today.  The rest beyond the core engine was good, but not what I was hoping for.  After a month of getting to know what’s going on, we emabarked on a massive push to completly overhaul our UI, the workflows, and everything about the product experience.

User experience is incredibly important to me, and I wasn’t happy with our state.  I confidently laid a plan for ripping the entire thing up, changing everything to the very core.  I committed to my boss that we could get this, “no problem”.  I signed up for all sorts of improvements, from new visualizations, new interface paradigms, new toolkits, and more.   Now, as we’re just two weeks away from release, I know I was totally nuts.  Not because we didn’t deliver everything I hoped we would – we did/have/are!  No, I was nuts because looking back, if each and every little thing hadn’t gone just right, we would never have made it here.  If we hadn’t hired an AWESOME UI engineer.  If our lead engineer wasn’t a rock star.  If we hadn’t managed to get an automation engineer and harness to remove the friction.  If the entire team wasn’t flexible enough to deal with massive change daily.  But we did, he is, we did and they are.  And so now, on time and on schedule,  we’re two weeks away from launching a product that I’m really excited about.

We’re a startup.  Odds say, we’re going to fail.  It’s only by having unrealistic expectations in the first place that any of us even can get into this job.  If we don’t keep having unrealistic expectations every day, we’ll get trapped by what we can’t do, and deliver just another piece of software.  IF we can keep being unrealistic, signing up for insane deliveribles every day, then I think we just might make it.

Over at my job, we’re just two weeks away from our first release!  I thought this would be a great time to look back a little at our tools and process.

Process

From the day I joined, we’ve been developing via a scrum process.  For those not familiar, the highlights are:

• Everything you want to do goes onto a prioritized list called the backlog.
• Write requirements in the form of user stories.  (As a <USER> I can <FUNCTION> so that <REASON>)
• Anyone can add, only product owner can prioritize.
• Team agrees on what backlog items they are going to work on during a planning meeting.
• Create todo list from user stories.
• Track progress through iteration.
• Repeat!

Tools

Say what you want about pen and paper, but I’m a geek, and love the toys. We use a bunch of tools here:

• Jira as the single source of truth
• Greenhopper as the plugin that makes Jira usable in an agile enviornment
• Confluence for all our wiki needs
• Bamboo as our build server
• Subversion for version control
• Balsamiq to mock up our front end.

Overall, this set of tools has been fantastic.  We’ve spent right around $5,000 for all the software.  That’s a significant chunk for a small company like us, especially when there are such great free tools all around, and I’d spend it again in a heartbeat.  The quality of these products is just outstanding.  The support is great from everyone, and Atlassian in particular needs to called out.  I’ve had 4 issues worth submitting as tickets, 3 of which they handled in < 2 hours, and 1 was my fault.

Although it took some time, I’ve decided on the tools.  As the title says, I’m going with Atlassian’s suite, plus a really nifty plugin I’ve come across called Greenhopper.  Here’s the overview:

• Create user stories in Jira.
• Create wiki page for each release (not iteration, but release).  Link up to Jira dashboard.  Provide details, necessary information, any supporting stuff available.
• Planning poker to estimate user story size.
• Prioritize user stories with drag and drop via Greenhopper.
• Drag user stories into iteration releases via Greenhopper.
• Create sub-tasks for each user story to track hours.
• Drag tasks on planning board for open, in progress and done via Greenhopper.
• Use Greenhopper to generate burn down, velocity, and other fun graphs.

Now, all that said, there are times when paper is just the way to go.  We currently have ~85 user stories, and trying to figure out the ranking really is easier when you can have paper in front of you.  Luckily, it’s easy to do both!  By exporting the issues into excel, and then using mail merge from word, finally printing to Avery postcards, I’ve got a great way to create cards.  You can use my attached user-story-cards word file if you want to create your own.

I’ve concluded the tools analysis that I kicked off a few weeks ago. What initially began as a look into requirements management tools quickly expanded into the entire product development lifecycle. Tools can not make up for broken process, so that’s the place to start. As I looked back at what’s worked and not in the past, I realized that the MOST critical success factor has always been alignment. If engineering, product management, sales and the customer aren’t all aligned on what, when, how, and why, we create massive headaches for ourselves. To ensure alignment, we need one source of truth, one place to track everything that’s going on. If we need to keep two system in sync, we lose. If everyone in the company can’t see everything, we lose. If everyone isn’t bought into the process, we lose.

In order to get everyone on the same page, we need to look at the toughest participant. I’m sure this won’t come as a surprise to anyone, but usually that’s engineering. Not only are they the largest team, they also have the strongest opinions. Starting from that vantage, I realized the plan of record has to be kept in a bug database. It’s the only thing engineering will update. In my past lives I’ve many times tried to keep a separate list, one focused on features, and engineering’s DB focused on bugs. But what happens when you classify a bug as a feature enhancement? It’s such a grey area, do you start replicating and moving data? Duplicating it? Either way, it’s a disaster. I’ve gotten into massive fights over this before, and even if I was “right”, is it worth fighting that over and over?Yes, tools like Rally and Accept can interop with Bugzilla or Jira. But if I need to have the bug DB anyway, is the extra effort and cost worth while? For such a small shop as ours, I’m not sure the answer is yes. I’ve used Accept for a few years, and have loved the tool. For traceability and analysis, it’s the perfect tool. I think I may have let those aspects get in the way of focusing on what the requirements themselves are, and how we get alignment in the groups. Frankly, all the tools I looked at in the previous post just seem like total overkill for any <50 (and maybe even <300) person company.

Now clearly, the bug database by itself isn’t sufficient. You need a place for more unstructured data, for planning, notes, etc. For marketing data, sales plans, and just the rest of the company. Obviously, a wiki is the best place to put this. This leaves us needing a great bug DB, and a great Wiki. In a stroke, I’ve thrown out almost all the tools I was looking at before: Accept, Rally, Feature Plan, VersionOne. They all are… complex. I’m sure for a large company, they’d work great, and have their advantages, but for a startup looking at Scrum and frankly trying to save some money, I just couldn’t see the value. So that leaves us with only four options that I could see out there:

• Fogbugz (with or without external wiki)
• Twiki + Bugzilla
• Trac
• Confluence + Jira

Fogbugz I quickly ruled out. Although it’s got a great UI, and I hear good things about it, it’s also just not the right fit. Some of the additional features were lackluster (the wiki is atrocious, the forum useless compared to phpBB), plus it’s very general (good), and needs to be shoehorned to fit Scrum or other agile processes (bad).

The next three was a much tougher decision. Trac looks great. For an internal focused project, or an open source project, it’s probably the one I’d use. However, we have need for both private and external content, tighter security controls, and some more advanced commercial features that they don’t seem to cover. I REALLY liked Trac, but I didn’t see it fitting into our commercial environment.

Now it’s really hard. I LOVE twiki. Better than Confluence. I like Jira better than Bugzilla. Ultimately, the support & integration that Atlassian is providing, plus the larger suite of products such as Fisheye, Bamboo, etc all pushed me over the edge. Maybe it’s just a dream, but the thought of having a view into CI data, test status, check-ins, all tied and traceable, makes me a tingle.

From a cost perspective, for a small team like ours we’re looking at a $3,600 investment for Jira and Confluence, plus $1,800 for maintenance. We’ll put them in VMs on existing HW, running Fedora. The incremental admin costs for two new machines should be minimal. I’m figuring if we buy into their whole stack, we may wind up spending 10K with them. On the one hand, that ain’t chump change, especially when we could be getting it for “free” with Twiki/Bugzilla or even Trac for internal + Twiki for external wiki needs. I’m counting on the support, the features, and the overall new development will all make this pay off.

I know many other people out there on the internets have gone through similar evaluations. Which way did you go? 6 months or a year later, are you happy about it?

BTW, the photos are from a recent drive I did. They are there just for decoration.

What should I build? And who says so? When do they need it? Will we be able to deliver?I’m working with my team and engineers every day to answer these. Although tools will never make up for a terrible process, a good tool can also make things go much much smoother. For 2 years, I’ve been using Accept 360 for requirement tracking. My wife is leading up an evaluation at her company right now, which sparked some more investigation on my part.To look at all the players out there, you need to break the SW development process down. I’d say there are three phases, irrespective of development methodolgy. Regardless if you’re a Scrumer, a XP developer, or a plain old waterfall, it seems like you always have three tasks:

1. CRM/Customer interface. Collect data from customer, define and justify your requirements with data. This is almost always driven by email, personal interaction, meetings, notes. The tool is here to help the product manager collect their thoughts, organize the data, collaborate with other marketing people, and feed the status back to the field and customer. Very focused on traceability – show me who is asking for what, when and how. Very CRM like.
2. Project Management. Manage the development, what is in what release. Know when and what you’ll have. A very blury line often between product management, engineering, and gobs of other people. Focus on the deliverables, the specifications, and getting releases out. Heavily used during development by all involved.
3. Release management. Bugs on released products, feedback into step 1. Close the loop. Collect the feedback from users, community, post-release QA. Feed back in to Phase 1 and continue ahead. I see this as a combination of bug database and Wiki.

I looked at bunch of tools. All except featureplan I’ve looked at in the past month – featureplan I last looked at over a year ago, so much may have changed.

• Accept 360
  • Phase 1 Focused
  • Web UI, Slowest interaction, though still acceptable
  • Very focused on the requirements gathering phase. Very strong relationship definition.
  • Recreating the CRM – if you use anything else gets frustrating to enter in data duplication.
  • Methodology agnostic. Neither helps nor hurts with Scrum.
  • Weak Phase 2. Not worth using IMO. Too complex, can’t actual inteagrate. Very heavy project management overhead.
  • Seems big-company centric. Doesn’t feel “cool”
  • Doesn’t publish pricing, but it’s >$1000/user/year.
• Rally Software
  • Phase 2 focused. Plugins with others to satisfy phase 1 and 3.
  • Medium-complex UI. Very usable, fast.
  • All about Agile/Scrum. Uses terminology, flows that force you to use agile.
  • Focused on what they can do – doesn’t try to solve it all
  • Best integration – subversion, eclipse, Visual studio, jira, salesforce, etc.
  • Super cool wiki/community integration with end-to-end salesforce tracking as well.
  • $35/user/month or $420/user/year. Salesforce integration extra.
• FogBugz
  • Phase 2 & 3 focus
  • No methodology implied, do whatever you want.
  • Integrated wiki appears weak – separate from reset of product. Would personally use mediawiki instead.
  • Very slick UI, very fast, VERY easy to use.
  • $25/user/month
• VersionOne
  • Waiting to get a demo account still. Demerits to them. Who requires talking on the phone in this day and age to get a demo account?
  • $30/user/month
• Feature Plan
  • Phase 1 only, but crushes it.
  • Deep requirements planning. Integrates with Rally.
  • Development methodology agnostic.
  • Only windows only client (they’ve got a web UI, didn’t exist when I looked last)
  • All about pragmatic marketing (but then, I am too. :)
  • Darn expensive as I recall.
• Trac
  • Phase 2 and 3.
  • The dark horse in this group – open source, PITA to install
  • Scrum plugins to make project management in Scrum very reasonable.
  • Good integrated Wiki features
  • Possibly better as a project tracking than a product – need investigation but initial feeling is this works GREAT for open source projects, but may run into issues with wiki and others for real product release lifecycle (i.e. permissions on documentation).
  • “free”

Rally really impressed me. I used it 18 months ago, and was underwhelmed, now it seems like a great tool. I especially like the integration – Salesforce for your CRM (and bugs even), take continuous integration data, and have it all useful from one tool. It’s probably a high investment strategy, but I could see starting small, just trying it out, and growing to make it a core part of the platform.Alternatively, fogbugz looks like a great choice. It’s less of the uber-tool, and more focused on the development. For Scrum management, I’d suggest using good old paper (burn down, planning, etc).The data geek in me says go for Rally. KISS says Fogbugz and paper.