Because real community ROI can be hard to measure (so we use proxies for it like cost-per-impression comparisons with advertising, or offsets for physical event costs), we often tend to measure the 'health' of our communities by the level of activity.
New members, posts-per-member, churn (new members - members who are inactive or quit) are fairly typical metrics that I've used in looking at communities.
But for myself, I've also been obsessed with 'quality' as a community metric; both in terms of how 'upward-norming' the community is, and how well the community fulfills the intent of the community sponsor. I'm not sure how to quantify it, but I think we all know it when we see it.
'Upward-norming' is simple; it's the willingness of community members themselves to intervene and react to typical online bad behavior; trollish posts, ad-hominem, off-topic or abusive posts. When the community members themselves are willing to enforce positive community norms, you have a relatively stable community. You can, however, also get a stable negative community, where abuse is itself a norm. The problem with that is that such a community will drive away everyone who isn't interested in a hostile free-for-all, which makes the traffic and attention generated from such a community a lot less valuable.
I've been looking at local review sites a bit this month, and in talking to friends about them, and it's been interesting to see the back and-forth about Yelp. My wife uses it extensively (but she also plans most of our outings) - I tend to go to things that I hear about through mainstream reviewers or through friends (a benefit of having a whole lot of friends).
She and one of her friends were having a back and forth about plans and her friend said:
I rarely Yelp anymore. Yelp has changed a lot, it’s not really about food anymore, more about young people talking about getting drunk and acting dumb. I'm back on Chowhound more now, since it really is about food.
And poking around in comments about Yelp, I noticed a decent number of complaints along the same lines.
Now, to keep perspective, Yelp's traffic is still growing fast, so it's not like occasional complaints about quality are going to shut the business down. But...they do present a vulnerability to competitors (see: Chowhound) and a long-term risk to the brand.
For sponsors of communities, the point I'm trying to make is that it's perfectly possible to have a community that's firmly in the green on the typical metrics - traffic, membership, sentiment, number of posts, churn - and is still having no or even a significantly negative impact on the brand.
If the brand is about sharing tips on good restaurants and bars, but the traffic is dominated by posts about how much fun it is to go out, get drunk, and hook up - you'll get lots of traffic, the sentiments will be positive - but in terms of meeting the brand goals, you'll be failing.
What's the answer?
Lead the community (leadership is more important to communities than management) by setting out a clear statement of what the community is for, both for the sponsor and for the members (and those had better be aligned...) and by calling out behavior that adds to the community purpose, and behavior that detracts from it.
I think you're absolutely correct.
In my time moderating the blog comments at JewishJournal.com, I felt it necessary to limit certain members to X number of comments per day.
These were the nitpicking pedants who were quick to attack spelling mistakes, grammar, typos, etc. as well as the trolls and haters who apparently enjoyed online flame wars as a recreational activity.
The throttling of 'power posters' helped the tone of the discussion immensely, although the immediate, short-term effect was a decrease in activity.
The long-term effect -- before my efforts were over ruled by ignorant higher ups -- was to elevate the discussion and make the blog comment space more welcoming, nurturing and interesting.