Deadlines – they have the word dead in them for a reason

For years I shipped boxed software. Quite literally, boxed. At Sun, we had a guy who’s job title was ‘boxologist’. He figured out what we could fit in to what kind of box and how to package it up. Not to do it, not to arrange the actual hard work of logistics. Just to figure out what could fit into a box. That’s it, a full time guy, in 2006. (Not 1996. 2006!) (Possibly one of the reasons Sun didn’t do so well, but I digress.)

Back in those funny days of boxed software, we’d ship things once every 2-3 years. It was a symbiotic fail: our customers couldn’t test, QA and deploy new versions faster, and our product development couldn’t develop, QA and release any faster. As the end of a release came up, it was a mad rush of work to get features in. The months before feature freeze were insane panic of forcing tons of things together in a dependency tree that makes a redwood look small.

Of course box or download: it’s no different. When you have a long release cycle, regardless of the delivery method, you have a deadline. A deadline means you are either in or out. At some point, call it ‘feature freeze’, or ‘the final iteration’, you’ve got to package up what you have and actually get it to the customer. This process, the packaging, takes components from across your product, cutting across features and teams. Often the only common theme here is the deadline. Sure, there’s probably an overall mission – introduce massive awesome feature X, clean up bugs Y, but the reality is vast swaths of code will have nothing to do with this mission. They will be the various cruft that builds up over time and NEEDS to get out to customers.

Deadlines are nasty things. Miss them, and you are in a world of hurt. Either you push the deadline for everyone else back to fit in, or you’ve got to wait till the next go around. Pushing back the deadline only leads to madness. Once you start pushing, it’s almost impossible to stop that other team over there who can now add in their feature that just missed. Till that feature has as bug, and pushed the deadline even further back. And then someone else will have something to put in. Once you cross the line and push your deadline, you’re in slipping-for-years mode.

Iterations – because you want more deadlines?

The rational decision is to stick to the deadline, and push to the next release. With packaged software, maybe you’ll wait a few years for that next release. That just ratchets up the pressure around hitting the deadlines 1000 times, making the bugs, the confusion, and the unnatural acts that much worse. The obvious solution is to shorten the release cycle. Release more often, say once every 6 months instead. Why stop there? With cloud delivery, we’ve removed the pain of updating from the customer. A well developed cloud app allows for transparent updates without an acceptance loop from the customer. This means there are no more constraints on how often you can release. Do it every month, or heck, every week.

However, if you’re talking about releasing every X period, “You’re Doing It Wrong” ™. By definition, you’ve still got deadlines. Sure, they’re smaller, but they are there. All the same problems discussed above continue to exist, toned down, handled more frequently in smaller chunks, but fundamentally unchanged. You have still forced your company to make tradeoff decisions based on timelines.

Developers have Flow – Products have Momentum

Individual creatives are aiming for flow. Cranking out code, art, words. Just hammering away. In one session of good flow state I get more done than in weeks of fits and starts. Everyone has their own path to getting to flow. Some put on the headphones, wear a hoodie, and disappear into their computer. Others go to a coffee shop, let the white noise wash over them, and get into the zone. Sometimes though, the conditions are right, but the project isn’t willing.

Blog posts, software development, homework: we are all familiar with the experience of working incredibly ‘hard’, but just not getting anywhere. 20 hours in, and you just don’t feel like you have a sense for how to even begin to tackle the project. Flow has it’s own conditions. Without clear understanding of goals, you aren’t getting anywhere. Goals don’t come from flow. They come from somewhere else. When you’re creating, working on something innovative, they often seem to come from the subconscious. When you’re stuck, and don’t have a clear sense of the goal, if you’re lucky, you walk away. You take a break. Maybe for 30 min. Maybe for a month or longer. And something happens. A shift. A new angle. A comment. Inspiration. You can pick the project back up again, and now you’ve got traction. You can push just as hard and make some real progress. You personally can get into flow. Your project can pick up momentum.

Momentum is fickle. It’s not something you can predict, store, save and spend. Some projects are real bastards – they get stuck just when you think you’re making the most progress. Sometimes you need to put it down and pick it up 6 times over 13 months before you finally cross the finish line. Some projects get momentum from the second you sit down, and one night later you’ve launched.

Continuously avoid the death march.

Working against a project’s momentum is about the most painful work experience I know of, and the standard name express it well: the Death March. Deadlines mean death marches. Maybe small ones. Maybe huge ones. But if you’ve got a deadline, you’ve incorporated into your project ‘death march by design.’ The shorter the cycle, the smaller the potential, but even 1 week release cycles have a mini-death march built in.

Continuous deployment gives you the option to free yourself from the death march. As soon as you have a feature/bug fix/whatever, ship it. Don’t wait for an artificial iteration to pass. If it works, if you are ready, if you’ve managed the lifecycle of the change, ship it. If you’re not ready, don’t. Releasing is hard work. You need to make sure it works. You need to test it with a segment of your customers. You need to think through all the permutations and impacts of your change. Slow down. Take the time to make sure you’re doing it right.

Don’t push things that don’t have momentum. You’ve got 1000s of projects to work on, and 100s that are all amazingly valuable. Pick the one that’s ‘ready’, work on it till you’re not making progress, and then walk away. Take on something else. When it’s ready, you’ll know. The work you put in will deliver results. You’ll be energized. It’s obvious once you’ve experienced it. It’s liberating.

Slow in, fast out

Here’s where the magic is. Your customers have no idea what your development process looks like. They have no idea what you’re picking up and putting down. They see what you release. That’s it. If you’re following the momentum, you probably have a few projects in process at all times. Pick one up, put it down. Pick a different one up. Put it down. When and only when a feature is ready, push it out. This only works if you’re deploying all the time. Not weekly, ideally not even daily. Deploy when it’s ready. This lets you focus on just one thing. When feature Z is ready, ship it. No dependencies. Remove the complexity across projects. Once you’ve done that, you can walk away from projects at any time. There’s no fear that if you miss the deploy next month, you’ll have to wait 45 weeks till you can get it out again. You can get it out when it’s ready. Tomorrow or next year.

There’s a word for this: you are pipelining your development process. You will likely be releasing features significantly more frequently to your users, specifically because you’re willing to walk away when they aren’t ready. This point bears repeating: by specifically trying to slow down, by not pushing things, by juggling a number of projects at once, you will actually ship to your customers more value, more often.

Thanks to Adam Wiggins, Rich Miller and Mark McGranaghan for reviewing and providing feedback on this blog.

I'm a bit obsessed with hacking my tools selection for optimum... um... not productivity. Enjoyment? Satisfaction? Self-actualization?

Regardless, I play with a lot of various tools. The past few months I've been pretty stable so it seems like a good year-end time to share.

Launcher: Alfred.app. It's clean and fast. I don't use any special fancy features, but I bought the powerpack to support the developers anyway.

General Content Sync: Dropbox. I purchased the 50GB level. I have all my actual docs here, but don't have my media or anything crazy large like that. Usinng 18GB of my 50 right now. I right now keep my docs unencrpyted, but have at times used encrypted FS tools. Not sure what the best thing here is.

Todo Management:  Cultured Code Things. I use my todo list in a quasi-GTD kind of way. I try to offload anything I know I need to do into there, keep my inbox 0, etc. I don't usually have a ton of todos - I factor aggressively and throw things away. Usually I've got 20-50 things max, including the 'someday' items that I really just delete months later. I've tried every todo thing I could get my hands on, and ultimately settled on Things for one main reason - it's quick entry dialog box.

A todo manager is useless if you don't get things into it. For me, this means having a single key-stroke access at all times. I've setup my computer so control-option-space brings up this awesome dialog box. You can see in this case it includes a snippit of text from this blog post. If I'm in mail, it'll include a link to the message. If you're on a webpage, it includes the URL. You never lose context this way.

This one feature has driven almost all of my other tool selection.

Up until recently, the lack of a working sync solution with Things has killed me. They've become more open in their beta program however, and it seems like just about anyone can get in to the beta by dropping them an email. Now that I can cloud sync between my iPhone, iPad, 2 work laptops (don't ask) and home iMac I'm very very happy.

Mail: Mailplane.app. I've found gmail with keyboard shortcuts to be far and away the fastest way to manage email. If you don't know the keyboard shortcuts, go take a week and learn them. I've tried sparrow and mail.app and neither can hold a candle on speed to just gmail. The thing is, I don't like just having a tab open. I like to be able to cmd-tab over to my email. Or fire up alfred and just type something to switch. This means I need a dedicated applicaiton. For a few years I used fluid.app which mostly works awesome. However, it doesn't support the all-important things quick entry discussed above. So fluid.app out, and mailplane.app in!

Notes: nvALT (fork of Notational Velocity). Notational Velocity is revelatory. Just go try it. It's harder to explain than use. I use the nvALT fork because it supports Markdown preview. Otherwise it's identical. I store my files as plain text in a dropbox folder so I can access my notes on-the-go through Elements.

In case you're wondering about notes vs. todo. Notes never have actions. It's literally just notes. I often will review my notes, and create tasks in things based on what I've written.

Calendar: Google Cal & BusyCal. Google is my canonical source of calendars. I actually have 2 calendars - one on heroku.com, and one on my personal teich.net domain. Work things go on the work cal, personal things on the personal. Yes, this means my free/busy is never correct. For me, I do 99% of my meetings on business, so it's fine. For some people, this may not work at all.

For desktop access, I'm currently using BusyCal. It's not perfect, but it's decent. Meh.

Password Management: 1Password of course. It works. It syncs with dropbox across all my devices. Done.

Backups: I have 2x 1TB drives. I keep 1 plugged in at work at all times, the other at home. My MBA Time Machine backups when I leave it alone for a bit. Sometimes I go a few weeks without backing up. Living life on the edge! Every month I swap the drives home/work.

Screenshot Management: LittleSnapper. I can't believe how much I use this. At least 2x a week. I use it a poor-mans-OLAP DB (take screenshots of data over time to compare). I use it to capture customer sites. I use it to get nice long screenshots instead of just what's on screen (since all the plugins like awesome screenshot seem to never work for me). And it keeps from cluttering my filesystem. I only use this for full app screenshots, not small snapshots. For that I just use the built in system capability (shift-command-4 space).

File Sharing: Cloud.app. 90% of my cloud.app usage is for sharing small screenshots over IM. Take a screenshot, and BAM you've got a URL to it: http://cl.ly/1I1y3j043V0o0G2p3Q1i It just works. Of course you can share bigger files by dragging them to the menubar as well. It's just awesome.

Hope this is helpful for anyone out there curious about what I'm doing these days. Love to hear what you're doing too.

Getting a once-size-fits-all conferencing solution failed us, so I looked for a specific use-case approach.

Each week, we have an all-hands meeting. Anywhere from 1-5 people will present a topic. Sometimes it's just talking, othertimes there is a demo, slides, or looking over the shoulder while surfing. We can have >50 people on in the room and on the phone, plus a bunch more who can't make it and want to see a recording. People on the phone or in the room can be speaking.

The Audio

First priority is getting great audio for all involved.

The SF office

Instead of trying to mic up the room so anyone can talk at anytime, we went with handheld mics to maximize the quality of the one person who is actually speaking at a time. Yup, you've got to pass the mic around now. It actually helps focus the meeting, and hugely improves the audio quality.

These mics have an XLR output. That means you can't plug them into your computer like normal. Apogee Duet to the rescue. The duet is a mac-only interface that has XLR inputs and outputs which will come in handy in just one second.

To output audio from remote people, we needed a speaker. I went with a pair of close-out M-Audio DSM1. Though technically near-field speakers, they work out just fine for the huge room they are in. The speakers have only XLR inputs so they are also hooked up to the Duet. 

The Conference Service

Now that we've got great audio input and output, we need a service to match it. I stumbled across http://www.hidefconferencing.com/ which is just what the doctor ordered. It's a standard dial-in conference system, with one HUGE benefit: people on Skype can dial directly from Skype and get full bandwidth gorgeous audio. 

Protip: make sure to configure the line in the web interface to DISABLE the entry notifications, and put everyone in mute mode by default when they dial in. They can then press *2 to mute/unmute their line, and you won't get the annoying background noise issues.

Screensharing

For screensharing, we're using GoToMeeting. Works fine.

Hooking it up.

First step, fire up skype, and try it out. It's working great! Wait, only one mic is working. Turns out that sykpe only sends one channel of audio, the left one. Any inputs in the right channel are discarded, and that's my second mic.

I really didn't want to buy more hardware, so after some digging Wiretap Anywhere came to the rescue. Wiretap lets you setup virtual audio devices and hook up input and outputs. In my case I setup a patch from the Duet to a mono output, and then configured skype in the preferences to use that as it's input.

From there, starting a call is just a matter of dialing in to a special # from Skype. I added my # to the skype address book, and now we are 1 click away from starting an audio conference that actually works.

Recording it for later

I had hoped that the built-in screen recording feature of quicktime would work, but it doesn't pick up the audio from the remote participants. Enter screenflow. It's a great screen recording program, and with a built-in driver will even record computer audio.

Right now the setup has local particpants on the left channel, and remote on the right which sounds weird. Will have to play with mono mixes for the future.

For playback, we went with Vimeopro. After the recording is done, we can use the built-in export to vimeo feature of screenflow. We put it up password protected, and share it internally.

Next steps

Automation. Right now there are a few two many moving pieces, specifically the recording process. I'm hoping to get someone here at Heroku to applescript this whole process.

So far the one call we've done this way has been amazing. The recording is easy to follow, the participants were happy. A bit early, but it may be problem solved. 

Last I checked, we're all humans. That means for certain types of information exchange, nothing beats verbal communication. As in voice. That's easy when it's just yourself (nothing wrong with talking to yourself) or a few others. Grab a room and chat away. But what happens when some of those people aren't close? And worse, what happens when there are a lot of them?

The driver for all our efforts is audio quality. When we have 20 people on the phone and 20 people in the room, it's been impossible for anyone on the phone to hear anyone in the room. Many phones we've tried are half-duplex, so if someone on the phone starts talking and gets on a roll, there's nothing you can to do break in.

Other requirements include screen-sharing and ideally recording for later playback.

We've tried everything from free to $10K+++, here's notes on what we've done so far:

• Speaker-phone & conference dial in: We started with a simple polycom  and http://ww.freeconferencecall.com/. For more than 2 people on either end, the results where terrible. As we got >10 people involved, locally or remote, it became impossible for phone participants to understand anything. We tried many different combinations of conference dial in and speakerphone (including polycom's most expensive desktop phones with remote mics around the table), but nothing was decent, let alone good. We tried campfire's conference line (powered by twilio), expensive dial in #s, free ones, we tried them all. POTS audio quality just isn't good.
• Skype: To address the audio quality, we started using skype with a blue snowball. This proved to be great audio for small groups but the participant management is a killer. One person needs to dial out and add each participant. If someone drops, it's up to the conference organizer to add people back in. For half a dozen people in the room plus 2-3 on the phone, this is the easiest and best solution by far. As the # of remote grows beyond 3 or 4, the management makes this impossible.
• Super speaker-phone & conference dial in: Next, we upgraded one of our rooms to a full-on pro system. Polycom Vortex EF2280 & EF2241 + 4 shure table mics + 4 shure orchestra mics hanging from the ceiling. Foam acoustical treatments on the ceiling.  I'm not sure it's any better than a standard polycom. This is a very challenging install, and it may be terrible because of the setup we have. 
• GoToMeeting for screen sharing & audio. Works really well for screensharing, but the audio quality is dissapointing. Specifically, it seems to have gain issues when using the VOIP connection on a Mac. The exact same hardware setup with Skype sounds amazing, but with GoToMeeting sounds quiet and unacceptable.

Lessons learned so far:

• Audio quality is the #1 priority. Without that, we're wasting time.
• Individuals need to dial-in. We can't have an organizer dial out.
• POTS (plain old telephone service) probably isn't high bandwidth enough
• Spending tons of money on 'pro' solutions so far hasn't seemed worthwhile.

All is not lost. See the next post.

6 months in at my job, and the past two weeks I started feeling odd; I've been busy, cranking out more than ever, but feeling a bit burned out.  At first I chalked it up to working hard, and enjoyed my time-off for the holidays.  But I came back, and didn't feel that same passion.  What's going on?  I doubled down, cranked out more than ever, and buried myself in working hard. Last week, an office mate was going over our expenses, and asked me about a really LARGE charge from Google.  Big enough I'm embarrassed to even post the number here.  I logged in to adwords, and sure enough we were hitting our daily cap every day, and that cap wasn't low!   Some history - Two months ago, the question came up: are our users representative of the market overall?  Since we've done no marketing to date and all our users come from blogs and word of mouth, we weren't sure if they were self-selecting in some non-representative way. We had the idea to get a more random sample of users by using adwords - set up a campaign, and see how the usage of people from adwords compare with our general user base.  We ran the campaign, leaving google to it's auto-bidding thing.  We grabbed a bunch of keywords, set a super high daily cap for the experiment, and watched it closely.  After 3 weeks, we were running about $20/day in ads, and getting 1-2 signups a day.  Then I forgot about the account. Jump back to last week, and we have our huge bill.  I quickly shut adwords down, and moved back to getting stuff done off my todo list.  A day later, in passing, I mentioned the adwords SNAFU to James, one of our founders.  His immediate question: "At least we have a bunch of people - how do they validate our hypothesis?"  SMACK.  Kick to the head.   I had forgotten the whole reason we ran the experiment in the first place!  I got so caught up in doing things, I'd missed the whole point. At a startup, regardless of your position, you're there because you're a self starter.  You like setting your own priorities, figuring out the right items to focus on.  The key is, not to get trapped in the work you're doing.  For me, it's great to get success stories out, push some marketing activities, get customer feedback, push forward the features through engineering, and drive process improvements.  That's what I do.  But why?  And are they the right things to do now to answer those goals.  For me, the answer was no. I've now got a clear set of goals I'm working towards: Better define and optimize our funnel.  Understand our customer profile.  Validate our hypothesis around customer segmentation.  I've still got my list of todos (Things rocks). Now they're hyper focused on working towards those goals, not just working to checking off todo items. I'm so excited to go into the office tomorrow, I can't wait.

I've been working on some stats tracking.  This means a lot of complex queries, and sometimes ActiveRecord just doesn't cut it.  Doing it the ruby way (or at least, the ruby way I know how to do) would often mean queries > 30 seconds.  A simple SQL query could get me the same data in <100ms.  For some reason, I was having a really hard time figuring out how to do direct SQL queries though. After banging my head, here's what I figured out. There are two ways to directly query the DB with AR: find_by_sql and ActiveRecord::Base.connection.execute. They each have their place, and I use both. find_by_sql find_by_sql is super easy, as it actually returns an ActiveRecord object. This means you don't need to do anything unusual. It works on an object, and works great when you're selecting fields from that object specifically. Here's an example of how I use find_by_sql and some fun SQL to get a funnel of our users: [rails] @funnel = User.find_by_sql("SELECT DATE_TRUNC('month', created_at) AS month, COUNT(id) AS registered, SUM(CASE WHEN verified_at IS NOT NULL THEN 1 ELSE 0 END) AS verified, SUM(CASE WHEN confirmed_billing_at IS NOT NULL THEN 1 ELSE 0 END) as confirmed FROM users GROUP BY DATE_TRUNC('month', created_at) ORDER BY month") [/rails] connection.execute The other way to execute SQL is with connection.execute. This returns a database specific object, for example a PGrecord for my postgres database. You can then iterate over that object like any standard ruby array of hashes. The hash key will be the SQL header. Here's an example where I calculate the # of new user signups per week: [rails] sql = "SELECT COUNT(*), DATE_TRUNC('week', created_at) FROM USERS GROUP BY DATE_TRUNC('week', created_at)" results = ActiveRecord::Base.connection.execute(sql) results.each do |foo| puts "#{foo['date_trunc']} #{foo['count']}" end [/rails]

Number 1 lasted
one month till concrete killed it
replaced by apple

The second was ill
the screen ignored my fingers known,
fixed by apple

3rd time not the charm
microphone refused my voice
replaced in the morn...

...and 12 ticks later
four met it's watery end
slipped into toilet

five wins the award
for longest in possession yet
replaced with 3G

six took a world trip
last seen lounging in hong kong
someone got lucky

just replaced today
seven was a white 3gs
crack in case near switch

now 8 is restored
hopeful that it shall live on at
least till next one

That would be William Shakespeare or Stephen Sondheim. The IT team and developer team look more and more like the Sharks and Jets or the House of Montague and Capulet every day. Check out some quotes from VMworld this week:

The SpringSource CTO is on stage, hopefully to explain this. Unfortunately people start to leave as soon as they see code. -- Virtualization.info Day 2 liveblogging

And just one from the many on twitter:

Can't help notice the number of attendees leaving the keynote as VMware demos SpringSource :-( -- @markbowker

Over the past 40-50 years, the world has evolved an order to the IT world.  Developers create, IT deploys, Ops manages, and we've got vendors that cater to each.  Each silo has it's own jargon, procurement process, goals, etc.  Each has it's own self-reinforcing feedback loops strengthing the status quo, from press to analysts, to the vendors themselves.  IT is a $1.66 trillion business.  Down from previous years.  Trillion. Larger than 50% of the annual US budget. The most brilliant thing VMware has managed to do is introduce an amazing new (or really old) technology, without disrupting the process in any way.  Each silo still gets to work the way they have in the past.  Macro processes remain in place.  Developers still code.  IT still provisions stuff.  Ops still manages stuff.  Some of it just is running on other stuff now.  VMware enables IT to deliver what they've been promising for years.  Finally, IT teams are able to deliver servers at a pace, reliability and capabilities that they've promised for decades.  For once, IT is a rock star!  Frankly, if you're IT team isn't using some kind of virtualization, you may want to look real close at that team. PaaS, though, is a whole different beast.  When you start talking about letting developers code AND auto-deploy, it begs the question: what are those IT guys gonna do?  Sure, some still need to be around.  But not 1 for every 50 vms.  If this actually catches on, it might get as low as 1 for every 500, or even 5000 VMs.  That's disruptive to the very people who've built the 1.66 trillion business.  Of course they're gonna walk out of the room - what possible value is in it for them? Show that same demo to a room full of Java developers though - say at the next JavaOne (er, oracle world?  What is the conference for Java developers these days?), and see what kind of reaction you get.  If our experience with ruby developers is any indication, I'd bet they'll get a standing ovation.

For those following along at home, as Olivia and I have been developing Zed9, I came across this amazing PaaS cloud provider, Heroku. The second I heard about them, I was totally entranced. In a nutshell, they make deploying, scaling, maintaining and living with Ruby based apps trivial. For those who have lived through the application deployment lifecycle before, you know how huge this is. I knew I needed to work with them. And today marks the close of my first month anniversary with Heroku. It's been a great first month - with over 30,000 applications deployed, it's a product managers dream. We have engaged passionate users, and a team who started day one understanding just how important listening to customer feedback is. I'm ecstatic to be part of the team, and help make the cloud talk and vision a reality here.

As easy as using Heroku is, setting up DNS seems to be one of the trickier parts. Heroku has some decent instructions, but the dirty secret is that their required config is actually in violation of the DNS RFC.  While I'm sure they're trying to fix this, I've been running this for a few months and it does actually work, email and all. Show don't tell, so here's a 2 minute screencast that walks through the process of setting it up.  For those who want the one piece of magic: when you want to setup your domain to point to heroku so that http://yourdomain.com actually works, the trick is in the "host" field of the CNAME, put yourdomain.com. <--- NOTICE that there is a trailing period there.  Put your domain name, plus a period at the end of it, and you should be good. Configure Go Daddy DNS & Heroku from Oren Teich on Vimeo.