Ontic Oren

We decided to dip our toes into Google Adwords recently, and see if we could attract some good users. Bottom line: We didn’t. At least, not with my limited adwords-fu. The goal was to see if we generated traffic from google adwords, how does that traffic compare with our general user base. Is the quality of people who self select and come to us today via blogs, twitter, etc better or worse than what you get from adwords? This wasn’t about optimizing the lead funnel – that’s ongoing and in parallel.

Let’s leave aside the click-fraud disaster that was the content network – that’s another blog post. Let’s just look at the results from our direct search results.

Before I dive into the results – a word on how we selected the ads. We spent a number of hours looking at keywords based on existing incoming organic traffic, as well as keywords we’ve identified as potentially interesting but not currently sending traffic our way. We then setup some simple ads, and spent a few days letting those run seeing what kind of traffic we were getting. We did this running 7 ads split across the keywords, running A/B testing to figure out which words were performing best in the ads. After a few weeks of low volume limited spend to get data on decent ads and keyword matches, we had a decent idea of what ads were working. We wound up with 44 keywords across 4 ads. Since the goal here was to see the world of users, not validate the cost efficacy of adwords, we let google auto-bid for plaement, and got top 3 on almost all the keywords we went after. We included a few poor performing keywords as we thought they were important, despite the early data, and wanted to see what happened.

Keyword CTR ranged from 5%+(!!) to 0.2%, with an average of 1%.

I’ve recently started using Dave McClure’s AARRR metrics.  I like that it gives a good common basis for looking at things.  Here’s how I look at that for us:

• Acquistion: % of visits who don’t bounce
• Activation: % of visits who signup
• Retention: % of users who push code up to heroku, average time since last git push
• Referral: % of people who collaborate with someone else
• Revenue: % of people who convert.  ARPU (average revenue per user)

So, how did the adwords people do?  Terribly.

11% of our signups in December came from the Google adwords campaign.  For reference, about 1/3 of our signups come from organic searches and another big chunk from direct traffic (probably twitter).

• Acquisition.  Sadly, a SNAFU makes this hard to tell.  The analytics account isn’t tied to adwords, despite many go-rounds with google on this.
• Activation: Equal!  Same for adwords and regular users.
• Retention: Adwords users were 50% of our standard user here.  They pushed half as much, AND did it much less recently.
• Referral: Infinitely worse. :)  There were 0 from the adwords people.
• Revenue: And here’s the killer: only half the percentage converted, and those that did paid us half as much.  That’s 25% per user.  4:1!!!

The results above apply to various slices of the adwords. Even the very best performing keywords still don’t come close to comparing with the organic results. For comparison, a single good blog post would drive more registrations and engaged users than a month of the google campaign.

Next step – weed out the low performing ads and keywords, and start experimenting with the impact of various bid strategies. Iterate and see if there’s a good stream hidden in the noise here.

EDITS: Thanks to the feedback from the always awesome lean startup circle, I clarified the adwords process we went through a bit more.

6 months in at my job, and the past two weeks I started feeling odd; I’ve been busy, cranking out more than ever, but feeling a bit burned out.  At first I chalked it up to working hard, and enjoyed my time-off for the holidays.  But I came back, and didn’t feel that same passion.  What’s going on?  I doubled down, cranked out more than ever, and buried myself in working hard.

Last week, an office mate was going over our expenses, and asked me about a really LARGE charge from Google.  Big enough I’m embarrassed to even post the number here.  I logged in to adwords, and sure enough we were hitting our daily cap every day, and that cap wasn’t low!  

Some history – Two months ago, the question came up: are our users representative of the market overall?  Since we’ve done no marketing to date and all our users come from blogs and word of mouth, we weren’t sure if they were self-selecting in some non-representative way. We had the idea to get a more random sample of users by using adwords – set up a campaign, and see how the usage of people from adwords compare with our general user base.  We ran the campaign, leaving google to it’s auto-bidding thing.  We grabbed a bunch of keywords, set a super high daily cap for the experiment, and watched it closely.  After 3 weeks, we were running about $20/day in ads, and getting 1-2 signups a day.  Then I forgot about the account.

Jump back to last week, and we have our huge bill.  I quickly shut adwords down, and moved back to getting stuff done off my todo list.  A day later, in passing, I mentioned the adwords SNAFU to James, one of our founders.  His immediate question: “At least we have a bunch of people – how do they validate our hypothesis?”  SMACK.  Kick to the head.   I had forgotten the whole reason we ran the experiment in the first place!  I got so caught up in doing things, I’d missed the whole point.

At a startup, regardless of your position, you’re there because you’re a self starter.  You like setting your own priorities, figuring out the right items to focus on.  The key is, not to get trapped in the work you’re doing.  For me, it’s great to get success stories out, push some marketing activities, get customer feedback, push forward the features through engineering, and drive process improvements.  That’s what I do.  But why?  And are they the right things to do now to answer those goals.  For me, the answer was no.

I’ve now got a clear set of goals I’m working towards: Better define and optimize our funnel.  Understand our customer profile.  Validate our hypothesis around customer segmentation.  I’ve still got my list of todos (Things rocks). Now they’re hyper focused on working towards those goals, not just working to checking off todo items.

I’m so excited to go into the office tomorrow, I can’t wait.

I’ve been working on some stats tracking.  This means a lot of complex queries, and sometimes ActiveRecord just doesn’t cut it.  Doing it the ruby way (or at least, the ruby way I know how to do) would often mean queries > 30 seconds.  A simple SQL query could get me the same data in <100ms.  For some reason, I was having a really hard time figuring out how to do direct SQL queries though. After banging my head, here’s what I figured out.

There are two ways to directly query the DB with AR: find_by_sql and ActiveRecord::Base.connection.execute. They each have their place, and I use both.

find_by_sql
find_by_sql is super easy, as it actually returns an ActiveRecord object. This means you don’t need to do anything unusual. It works on an object, and works great when you’re selecting fields from that object specifically. Here’s an example of how I use find_by_sql and some fun SQL to get a funnel of our users:

@funnel = User.find_by_sql("SELECT DATE_TRUNC('month', created_at) AS month,
             COUNT(id) AS registered,
             SUM(CASE WHEN verified_at IS NOT NULL THEN 1 ELSE 0 END) AS verified,
             SUM(CASE WHEN confirmed_billing_at IS NOT NULL THEN 1 ELSE 0 END) as confirmed
FROM users
GROUP BY DATE_TRUNC('month', created_at)
ORDER BY month")

connection.execute
The other way to execute SQL is with connection.execute. This returns a database specific object, for example a PGrecord for my postgres database. You can then iterate over that object like any standard ruby array of hashes. The hash key will be the SQL header. Here’s an example where I calculate the # of new user signups per week:

sql = "SELECT COUNT(*), DATE_TRUNC('week', created_at) FROM USERS GROUP BY DATE_TRUNC('week', created_at)"
results = ActiveRecord::Base.connection.execute(sql)
results.each do |foo|
  puts "#{foo['date_trunc']} #{foo['count']}"
end

Number 1 lasted
one month till concrete killed it
replaced by apple

The second was ill
the screen ignored my fingers
known, fixed by apple

3rd time not the charm
microphone refused my voice
replaced in the morn…

…and 12 ticks later
four met it’s watery end
slipped into toilet

five wins the award
for longest in possession yet
replaced with 3G

six took a world trip
last seen lounging in hong kong
someone got lucky

just replaced today
seven was a white 3gs
crack in case near switch

now 8 is restored
hopeful that it shall live on
at least till next one

That would be William Shakespeare or Stephen Sondheim.

The IT team and developer team look more and more like the Sharks and Jets or the House of Montague and Capulet every day.

Check out some quotes from VMworld this week:

The SpringSource CTO is on stage, hopefully to explain this. Unfortunately people start to leave as soon as they see code. — Virtualization.info Day 2 liveblogging

And just one from the many on twitter:

Can’t help notice the number of attendees leaving the keynote as VMware demos SpringSource :-( — @markbowker

Over the past 40-50 years, the world has evolved an order to the IT world.  Developers create, IT deploys, Ops manages, and we’ve got vendors that cater to each.  Each silo has it’s own jargon, procurement process, goals, etc.  Each has it’s own self-reinforcing feedback loops strengthing the status quo, from press to analysts, to the vendors themselves.  IT is a $1.66 trillion business.  Down from previous years.  Trillion. Larger than 50% of the annual US budget.

The most brilliant thing VMware has managed to do is introduce an amazing new (or really old) technology, without disrupting the process in any way.  Each silo still gets to work the way they have in the past.  Macro processes remain in place.  Developers still code.  IT still provisions stuff.  Ops still manages stuff.  Some of it just is running on other stuff now.  VMware enables IT to deliver what they’ve been promising for years.  Finally, IT teams are able to deliver servers at a pace, reliability and capabilities that they’ve promised for decades.  For once, IT is a rock star!  Frankly, if you’re IT team isn’t using some kind of virtualization, you may want to look real close at that team.

PaaS, though, is a whole different beast.  When you start talking about letting developers code AND auto-deploy, it begs the question: what are those IT guys gonna do?  Sure, some still need to be around.  But not 1 for every 50 vms.  If this actually catches on, it might get as low as 1 for every 500, or even 5000 VMs.  That’s disruptive to the very people who’ve built the 1.66 trillion business.  Of course they’re gonna walk out of the room – what possible value is in it for them?

Show that same demo to a room full of Java developers though – say at the next JavaOne (er, oracle world?  What is the conference for Java developers these days?), and see what kind of reaction you get.  If our experience with ruby developers is any indication, I’d bet they’ll get a standing ovation.

For those following along at home, as Olivia and I have been developing Zed9, I came across this amazing PaaS cloud provider, Heroku. The second I heard about them, I was totally entranced. In a nutshell, they make deploying, scaling, maintaining and living with Ruby based apps trivial. For those who have lived through the application deployment lifecycle before, you know how huge this is. I knew I needed to work with them. And today marks the close of my first month anniversary with Heroku.

It’s been a great first month – with over 30,000 applications deployed, it’s a product managers dream. We have engaged passionate users, and a team who started day one understanding just how important listening to customer feedback is. I’m ecstatic to be part of the team, and help make the cloud talk and vision a reality here.

As easy as using Heroku is, setting up DNS seems to be one of the trickier parts. Heroku has some decent instructions, but the dirty secret is that their required config is actually in violation of the DNS RFC.  While I’m sure they’re trying to fix this, I’ve been running this for a few months and it does actually work, email and all.

Show don’t tell, so here’s a 2 minute screencast that walks through the process of setting it up.  For those who want the one piece of magic: when you want to setup your domain to point to heroku so that http://yourdomain.com actually works, the trick is in the “host” field of the CNAME, put yourdomain.com. <— NOTICE that there is a trailing period there.  Put your domain name, plus a period at the end of it, and you should be good.

Configure Go Daddy DNS & Heroku from Oren Teich on Vimeo.

A nasty bug on ZED9 I’ve been avoiding the past few days was solved with a stupid single line.

In Rails, when you have a has_many relationship, it’s great and easy to be able to iterate through all the items. Assume you have models as such

class Foo < ActiveRecord::Base
    has_many :bars
end

class Bar < ActiveRecord::Base
    belongs_to :foo
end

So you go ahead and iterate through them:

@foo = Foo.find(:last)
@foo.bars.each {|f| f.your.logic.here}

If for anyreason, you’re counting on the ordering of those items, watch out! I was iterating through GPS coordinates, comparing them with the first one. SQL databases make no promise on ordering if you don’t ask for it, and every now and then I was getting totally insane results. Turns out the DB was feeding me the GPS coordinates in reverse order sometimes.

Luckily, rails makes this very easy. Just specify your ordering in the model:

class Foo < ActiveRecord::Base
    has_many :bars, :order => 'time ASC'
end

With that one line, I fixed half a dozen outstanding bugs. Mostly posting so I’m not stupid in the future.

Last night’s VC Taskforce Elevator Pitch Roundtable was quite the eye opener.  

10 CEOs each get up and present their elevator pitch in 90 seconds.  A panel of 5 VCs then spent 5 min in Q&A, followed by a 2 min critique and scoring from 1-5.  The rest of us – around 50 people total – are just fly’s on the wall.  

Two clear groups emerged: the obscure but competent idea, and the arrogant bastards.  In this group, we had  3 to 6.  Twice as many arrogant bastards.  Both the decent and the TERRIBLE pitches were really useful for me, and I pulled out a few points for any elevator pitch.  They may seem obvious, but apparently 66% of people don’t grok em, so here they are:

1. Start off with what you do.  Hi, my company is X, and we do Y.  Right up front.  Don’t set things up.  Don’t make excuses.  Don’t joke.  Don’t try and ask questions to “connect” with your audience.  Just say what you do.  NO ONE did this. Not one. At least 50% of the pitches the VC’s needed to ask during the Q&A, “Just what is it exactly?”. The rest it emerged, but it was like pulling teeth!  
2. Explain what job people hire you to solve.  Put another way, why should we care, and why does the user care?  You probably think this is obvious, and maybe 5% of the time it is, but you’re the expert, no one else is, so dumb it down.  
3. Make no assumptions.  OK, this should possibly be #1.  One guy spent the entire 90 seconds talking about “this”.  He even pointed to a physical device.  I thought it was a cell phone when he flashed it.  So 90 seconds he’s talking about bringing wireless to cell phones, and I’m thinking he’s a complete idiot.  Turns out “this” is actually a video camera.  And no one today has a wifi enabled video camera.  Ohhh….!  A bad idea still, but at least I know what he’s talking about now.  Don’t assume anyone understands the opportunity.  Don’t assume the market is clear.  It’s a hard line to balance, you don’t want to explain the basics that they get.  Understand the audience.
4. Never get defensive.  Ever.  Ever.  Ever.  If you’re getting questions, it’s a good thing!  Answer them.  SOB #1 last night immediatly assumed that the VCs were idiots, didn’t get what he was pitching, and responded to one very valid question about the company name by proudly stating that his previous company sold for $XXX million dollars.  Followed by explaining why he knew things the VCs didn’t, but couldn’t explain them.  He got all 1s. 
5. An elevator pitch doesn’t include financials, sales plan, etc.  The point is get em hooked so they can ask for more info.  What is the idea/product, the opportunity, and the problem you’re solving.  That’s it.  5 of the presenters wasted 30+ seconds talking about various internal details.

Some other random notes of interest:

• One of the VCs was concerned about a 3 year break-even.  Thought it was too long by far.  Someone, please show me what % of VC funded companies break even in 3 years, let alone 2 or less.  I’d guess it’s vanishingly small.
• One presenter dressed in jeans a t-shirt and sneakers.  Turns out, he had one of the best pitches, if not the best.  But in my mind, that initial disconnect on clothing mattered.  He overcame it admirably, but why start at a disadvantage?
• If I were a VC, there wasn’t one single idea that was interesting to invest in.    

As I’ve been developing Zed9, and reflecting on what we’re doing at Replicate, I’ve spent a lot of time trying to make sure I’m building the “right” thing.  So what’s right?  One theory is “Build it for yourself”.  As recently discussed by Tim Bray, John Gruber, and 37 Signals, the basic idea is: unless you’re a very strange individual, make a product you use all the time and chances are others will too.  You can focus on what you know, and make something great.  Build for yourself suffers from survivorship bias - it’s a necessary and helpful precondition, but far from sufficient.  There are countless products built for their creator that never made it anywhere.  The very fact that you have the skill set to create a product to solve your needs means that you ARE NOT LIKE the vast majority of your potential user base.  Using yourself as the prototypical customer can lead to dangerous assumptions.  What’s easy for you isn’t for others.  What’s obvious to your user could be totally obscure for you.

In creating any product, there’s always a disconnect.  You, as the product manager, startup founder or whatever, are either trying to solve a product you’ve personally experienced, or one you’ve seen.  Either way, it’s based on your experience.  And as much as we all like to scratch our own itches, selling to yourself is a hard way to make a living.  Many times you’re not creating this product to make a living, it’s an artifact of some other focus.  That’s certainly what happened with Rails, and I believe for Gruber with Markdown as well.  And when that happens, GREAT!  We all love the serendipitous success.  But what happens when you’re setting out from the start to actually make money on your product?  

That’s where the MVP comes in.  MVP stands for “Minimum Viable Product“.  Closely related to agile practices, the idea is put together the real minimum product to get customer feedback, and use that to validate and move forward.  Sometimes, your MVP can be as simple as a slide deck or even just an adword.  If people click and sign up for a waiting list, it’s probably a good sign that they’re interested, and it’s worth pursuing.  Don’t build a 2 month alpha if a prototype will suffice.  Don’t build a prototype if a marketing landing page will suffice.  The key here, of course, is the word “suffice”.  The MVP is all about getting customer feedback.  And apathy is the death of any feedback process.  We know what to do with negative results (try something new!), with positive (do more!), but what about no results?  

Apathy is really the scariest thing that a product owner can experience.  The lack of feedback gives us nothing to hold on to.  We start to breath our own fumes, going in circles.  What are the main causes for apathy? Three stand out: 

• Not a large enough sample size. We’re aiming for early adopters with any MVP based feedback.  If we’re lucky, they make up 5% of our target population.  Depending on your marketing and engagement practices, this means you might need to kiss 1000 frogs just to get 10 qualified responses.  Let’s take the ad-words example.  Your ads have a 2% CTR, and you expect 10% of the visits (a high number) to translate into actual feedback.  To get 10 actions, that’s 100 CTR, and 5000 impressions. Want 100 user feedback base?  Now you’re talking 50,000 impressions!  Dealing with contacts?  Assume you can get a 15% conversion rate and you’ll still need to talk to 66 people just to get 10.  This is probably one of the biggest issues many startups face.  From day 1, you need to be VERY aggressive about talking to as many people as you can to increase the population base.
• Didn’t actually make a minimum product. This is probably the first place any engineer will go.  ”Clearly, if we just add feature X and Y, THEN they’ll understand what we’re doing, and give us feedback”.  This is the most dangerous path to go down.  It’s the one where you eventually throw away the MVP, because you’ve never satisfied with the minimum.  It’s one I’m personally struggling with on Zed9 right now in fact.  We’ve launched a minimal product, that at the least gives the user some interesting rudimentary comparative analytics.  It’s different from other offerings, but not earth shatteringly so.  It hints at where I’m taking it.  I’ve had some interest, but not droves beating down my doors.  I keep thinking “if I just add this feature, then I’ll get 100x more people interested”.  This way madness lies.  You need to set clear metrics, and hear from prospects.  Go back to step 1 – not a large enough sample size.
• Not solving a broad/interesting problem. At some point, you need to call it.  Remember, the point of an MVP is specifically to find out IF IT’S GOING TO WORK.  It MAY NOT.  Hell, it probably won’t.  If you’ve talked to enough people, and still not getting a decent response, it just might be time to move on to the next idea.  If you keep going, you’re not building a product, you’re satisfying a hobby.  A hobby that may well turn into something interesting, but for now a hobby.

For me, I’m redoubling my efforts on increasing the sample size.  Before I spend a few more long nights and weekends adding all these cool features, I’m going to go out into the field, and talk to people.  I’m heading out to some bike and running stores during the weekday, to talk to the sales guys and see what HW is moving, what tools they recommend, and their thoughts on the viability of the product.  I’ll be joining some group rides, checking out the local meetup groups, and just talking to people about their problems today.  I’ll let you know how the next few weeks go!