Community Stewardship 101

Companies are the stewards of the community NOT the manager... it was something that was deeply ingrained into every single employee’s psyche at eBay. (eventhough it may not seem like it from the outside :) ) Apparently that lesson is being learned everywhere these days : youTube, eachnet, myspace, digg... the latest is that of facebook.

There are a lot of random posts on the controversy but only two actually offered some solutions and suggestions from the company/product management angle : Ed Sim & Ken Norton...

I wrote about it before on so called Web 2.0 Community Management but I think Stewardship is probably a better word for it...

A/B testing of new features is actually really really hard (like Ken said). You have a few options (not mutually exclusive)

1) Blind A/B

2) Opt-Out A/B

3) Opt-In A/B

None of them are perfect and in the end the community might still scream and yell.

1) Blind A/B

This just doesnt really work for community centric businesses... peoeple are just so attached and familiar to the site you need to be extremely transparent, otherwise you are going to get support calls (people think the site is broken) or another riot on your hand. Google/Yahoo/Amazon can do this because they have huge amount of “blind” traffic coming through the site... Google especially is already a black box so people dont expect it to be transparent in the first place. For eBay and, my guess, social networking sites; it will not work.

2) Opt-Out A/B

This is a little better in preventing a riot but you better make sure the A(control group) portion is siginifcantly bigger. Even so, you are going to get huge self selection issue because people will notice the “opt out message” and act very differently (such as clicking around a lot more and become a lot less task oriented). The portion that got sent there by default might still cause a rukus because they believe these features have a huge sunk cost so they will be released eventually anyways - perhaps with tweaks - and you know what... he/she is probably right...

3) Opt-In A/B

This will get you more input and participation from the community but in the end, the results are not going to be high fidelity... or even directionally correct. Those who opt-in either are early adoptors, have an ax to grind, or explorers. Furthermore, whatever metric you are testing for, selection bias will make it almost irrelevant. In many ways, this is the ONLY option community based sites have to be partipatory in releasing new features... another thing to do to reduce bias is to run the A/B for LONG time so that you at the very least filter out the riff-raffs who are digging around for dirt (reporters?) and see if your community either adopts its slowly but surely or segments significantly. (or never at all!).

For companies without conversion metrics (like facebook) the above tests are even harder... you can survey people at the end of the new experience but expressed and behaviorial feedback is very different. Furthermore, not until they have explore the majority of their OWN use cases for the site, their opinion might change if the feature is rollout permenantly. Retention/repeat visit/page view metrics are not linearly correlated to “satisfaction”, thus the effect of these features might be delayed or compounded negatively by some other factor down the road. And ofcourse relying soley on “feedback” creates self-selection bias on top of self-selection bias.

Furthermore what do you do if 5% of the population is hugely against the new feature but the silent majority (based on metrics) seems to be inconclusive or positive?

Llastly, as Adam Nash likes to say to me... what IF your most vocal community is the dying majority (as in The Innovator’s Dilemma - also brilliantly point out by Ken Norton). What IF you end up beholden to a segment of the customer base indefinitely and if that segment for some reason slowly shrinks? You do not have the strategic alternative of serving a new segment and grow for the fear of losing your current customer/user base. Many industry giants have died off this way...

(Side note, if facebook rolls back this feed feature, some startup should build a social network purely on the idea of broadcasted status... they will attract/self select those who dont mind and they will also know that the incumbents will not encroach. And as I suspect, as the web matures evenmore, the who idea of realtime updates will become more common place/desirable).

There are 20+ variables to evaluate when trying to roll out a new feature smartly to a community of users, there is no right answer for all cases, this is what makes this whole web 2.0 thing harder than people think .