People move on because they seem to assume that this is incompetence at work. Search is sooo 1.0, right? It’s been solved, and we’re just fumbling the execution, right? You usually get some sort of ironic joke along the lines of “wow, it is so easy to find stuff out there on the public Web, and here with all our resources, we can’t even do search right.”

And then the conversation tends to move on to more obviously “2.0″ things like blogs, wikis, how to increase participation, and my personal pet peeve: annoying moaning about “culture change.”

Hold on. Rewind. Let’s go back to search and think for a moment. I have a theory here, and I’d like to see if all you smart E 2.0 guys agree. I have reached a radical conclusion: broken search is the problem, but fixing search is not the solution. Search breaks behind the firewall for social, not technical reasons.

How Search Breaks Behind the Firewall

Let’s start with the blindingly obvious, and then draw some weird conclusions. Here are the most common reasons that come up:

1. Icebergs: Too much stuff that “ought to be shared” is on people’s desktops, or sitting in emails in attachment form.
2. Digital Landfills: There’s still tons of those random fileservers all over the place, where people simply dump files. They “ought” to put it into content management systems.
3. It ain’t a net: Web-based content is just easier to crawl. You have all those nice, friendly sociable things like robots.txt files that tell crawlers where to go and not go. Intranets are much ruder. It’s hard to take a proper inventory of everything that’s not on the searchable “grid” even it is nominally “online” in some form. Corporate IT “ought” to clamp down on undisciplined asset sprawl.
4. Those pesky formats: PDFs, PPTs, Excel spreadsheets, random proprietary databases with query interfaces.
5. Permissions: Lots of need-to-know systems with non single-sign-on protections. No way you can give your friendly Intranet spider ALL the skeleton keys, right? Corporate IT “ought” to enforce corporation-wide single-sign on.

Sounds like a purely technical problem doesn’t it? Get it all SOAn up (apologies for the terrible pun), create the holy grail “email attachment killing” technology (or alternately, figure out a way to make email “socially searchable”).

Why Search Breaks Behind the Firewall

Okay, I’ve been leading you to what I believe is the wrong solution. Everything above is merely a symptom. You could put the most talented engineers onto the problem, spend billions, and still end up with a mess.

You see the problem is not technical. It is social.

The key here is all those “ought” phrases.  Who’s recommending the “ought”? Who is being recommended as the “ought to do something” constituency? There’s no such thing as a person or organization without an agenda within a corporation. If the IT czar is a psycho with crazy ambitions, do you really want him/her to have all the skeleton keys? Is the person saying “ought” things about email attachments a librarian trying to make his/her job easier? Or somebody who feels cut out of certain conversations? Or somebody who is genuinely pointing out a non-political problem with sloppy sharing?

We often forget that the public Web is a very democratic place. Authority is democratically earned. PageRank is democratic. We link to stuff we like, authors we like. Google’s algorithms mine a vast democratic vote. Stuff on the public Web is online primarily because people WANT it online. They WANT it to bubble up. There are no big devious intentions. On the public Web, popularity equals power in very direct and simple ways.

Now step behind the firewall. What do you see?

1. Every single email ends up being an act of political judgment. To, cc and Bcc are three words about which I could write an entire book.
2. Need to know and organization charts/cascade patterns beat democratic content popularity and “Word of Mouth” information travel hands down. This surprises people: we often remark about how much work gets done at the watercooler. But this does not mean more gets done that way than through formal channels. The formal channels carry 80% of the communication. The reason we focus on the watercooler is that most of the politically sensitive stuff travels that way.
3. On the Web, people read people they like. Inside the firewall, everybody is constantly trying to figure out who’s important, who has the money, whose stock is up, whose stock is down. Whose coat-tails to ride. Which fool to suffer gladly for the moment, which battle to pick.  In other words there is very low correlation between information flows and friendship networks. You may pay most attention to people you hate or fear. Your best friend may be your lunch buddy from another department, and while you may be on each other’s blogrolls outside work, you have no connection inside and don’t vote up each other’s information contributions. Oneworkgroup may be trying to stay under the radar in skunkworks mode, while another may have a reason for wanting a dog-and-pony influence roadshow this year.

Forget blogs, forget wikis. Quite often intranet bloggers and wiki champions are the zero-influence noise signals (and I say this despite being one myself!). Most of the powerful people won’t blog. Let’s say a bright young intern writes a killer strategy brief/analysis of the company’s positioning as an internal blog, which gets a lot of comments and chatter.  Sure, an alert VP might spot it and hire the kid and groom him/her (okay I’ll admit it, that happened to me!). But look at what that does to search.  That page might well top the rankings by normal “democratic” search logic for the word “strategy,” but truth be told, the most important strategy documents are probably some low-distribution emails/meeting minutes, or some very carefully judged piece of officialese that comes out as a fait accompli, when it is useless to everybody else. Most people take one look at it, and yawn, but it is designed to send the right coded, and tactically timed signals about who has what piece of turf, to the few people who can interpret it. This document may get buried.

There are even rules. Like actual government, go-to-jail rules, that prevent the most important stuff from being shared at all beyond corporate officers, let alone discussed.

Some of the polyannish types might claim that this implies the organization is broken and that there is a need for more transparent, open, democratic governance.

I say, HELL no! This stuff may sound horrible, but large companies are not democratic for a reason. They exist to make money, not as laboratories for empowerment or social engineering.  The authority structures, communication flow patterns, emergent models of what’s hidden and how and why, versus what’s open, are all designed to work. Sure, you can find a lot of alternative models that are starting to work, but let’s not throw out the old without understanding how and why it works.

Which brings me to my main point. Search is political.

Search is Political

We often forget that Web 2.0 is built on top of Web 1.0. It assumes functioning search as part of the canvas.  Google and other search majors attempt strongly to maintain a “search neutrality” that matches “Net neutrality.” You may not like to hear this, but “search neutrality” is a really bad idea inside the firewall.

Inside the organization, the main form of authority and control is based on patterns of information sharing. We assume that function is accomplished by permission controls on content management systems and the like. Not true. Most information flow control is not based on restricting access, but limiting the very knowledge that a certain piece of information even exists.

And this is a good thing. If a lunatic in some toxic silo has evil designs on another department, then even the knowledge that a certain document exists may need to be hidden from him/her. There may be very good tactical reasons for burying some information in a complicated email thread, rather than being shared with clarity. Harsh but true: there are many good things that get done because someone was smart enough to cover their actions with nominal disclosure (”but it was in that email I sent 3 weeks ago, don’t act surprised.”)

Yes, such toxic politics can be really bad for an organization, but the solution is not to deny that it exists, but be smart about managing it.  Ben Horowitz has a brilliant recent post about what to do about minimizing politics from the CEO point of view, but it is really everybody’s job to be sophisticated about this stuff. And shoving a democratic search engine into your information ecosystem is a pretty naive thing to do.

A large organization is a contentious place. Alignment isn’t some idealized state of harmony to aspire to. It is a decision-by-decision evolving reality as power and influence shift through the organization chart. And again, this is a good thing. If things are going badly, you want different organizations to debate and express dissent, negotiate and if necessary, yes, hide information. You might even want them to work against each other for a bit of fairly-structured internal competition if you are not sure who has it right.

Public Web models of search blunder into this intricate and complex piece of machinery like clueless bulls in a china shop. This is the reason why they very rarely get much traction. It is in almost nobody’s interest to have powerful enterprise-wide search in place. This may sound perverse, but the ability to penetrate the opacity of organizations is to a certain extent the right test of whether you deserve access at all. If you can get to it, you are smart enough to use it wisely. It’s not a great system, but it’s better than naive search.

Instead of Search

I’ve been thinking hard about this problem, and I don’t think enterprise search can or should be “fixed.” The fundamental social and information flow assumptions of “search” need to be deconstructed and reconstructed for the enterprise.  Local/silo search within single sites/assets is fine. Enterprise-wide search in its naive form is a terrible idea.  And it is actually a good thing it is being done badly. Doing enterprise search well will provide a LOT more visibility to the most useless parts of the information ecosystem within an enterprise, and provide more incentives to hide the important information. Search infrastructure can provide a false sense of security, openness, transparency and “being informed.”

What can we do instead of search? I have some ideas, but they are still at a very early stage. But I do know that a smart discovery function is essential before the rest of the E 2.0 suite can transform organizations. The Web 1.0 agenda is still unfinished within the enterprise, so let’s be cautious about moving on to the E 2.0 agenda. Plumbing before furniture.

I am curious about how many people agree with my diagnosis or have alternative ideas/mental models here. Apologies if I’ve offended any huge advocates of enterprise search.

(Anyone know of any interesting forums/email lists about this issue? If not, I might be interested in starting one. Email me if you are interested, and I’ll start some sort of Google group or something if there’s enough interest).

Venkatesh Rao is an Entrepreneur-in-Residence at Xerox, and the manager of the Trailmeme project. He blogs at Ribbonfarm.com. These views are his own, and do not reflect those of his employer.

If you’re an Enterprise 2.0 entrepreneur, you’ve got until August 30 to Twitter pitch #e2conf-lp Paige Finkelman of the US Enterprise 2.0 Conference your ‘Launch Pad’ contest entry for consideration by the jury (which includes me), and broader audience votes.

The contest, to be held at the West Coast US edition of the show in November, is open to all Enterprise 2.0 vendors of any size that have launched or announced something new in 2010.

Paige spells things out above and you can find more details on the E2.0 blog, with rules and key dates listed on the official Launch Pad site.

Innovation is the lifeblood of any space, and you may find yourself on stage with Paige in November if your Enterprise 2.0 offering makes it to the finals, providing you with valuable exposure.

The concept of a Twitter pitch is a powerful one: simply send a tweet containing the hash tag #e2conf-lp and your contest entry from your Twitter account to get into the running.

Here’s a video archive of past finalists…I’m looking forward to seeing the new crop of entries and ideas…

Thanks again Oliver!

One week remains for enterprise 2.0 / collaboration vendors to Twitter pitch #e2conf-lp and tell the E2.0 team and Launch Pad Jury what you’re up to in 2010. The Launch Pad contest is open to all enterprise 2.0 companies - big or small - that have released or announced something new in 2010. A new product, a new version, a new partnership or integration point - so long as the announcement was made this year, it counts.

You can find more info on the E2.0 blog, and rules and key dates are listed on the official Launch Pad site.

Craft those 140 characters carefully and let her rip! Twitter Round 1 closes on Monday, August 30th. Looking forward to your submission, and best of luck with the contest!

Google did say it would continue development of Wave, but instead seek to integrate it into its other applications. As a model for future collaboration, Wave did make a wave. But its greatest impact will be to drive co-authoring capabilities into other collaboration applications.

Who qualifies?

The contest is open to all enterprise 2.0 companies - big or small - that have released or announced something new in 2010. A new product, a new version, a new partnership or integration point - so long as the announcement was made this year, it counts.

How do I enter?

Well, things start off rather simply. Beginning August 9th, 2010, you have until August 30th, 2010 to make your Twitter pitch to the Launch Pad hashtag: #e2conf-lp. We want to hear, in 140 characters or less, what makes you Launch Pad worthy.

Then what?

After Twitter submissions close on August 30th, myself and the esteemed Enterprise 2.0 Launch Pad Jury will vet the entries and choose our 8 favorite. Those 8 will then be asked to create a 3 minute video demoing their product. We’ll open up a community vote on September 20th, 2010. The four videos with the most votes will be asked to demo on the keynote stage at Enterprise 2.0 Santa Clara on November 10th, 2010. The contest culminates with a live audience vote to select the Launch Pad winner. More info on rules and key dates here.

Any cool prizes?

Of course. In addition to the exposure our Launch Pad qualifiers receive with their involvement in the program, our Final Four receive a special publicity package to help get as much buzz as possible out of the contest.

To get a taste of what our previous winners have submitted, take a peek at our video archive.

Looking forward to your Tweets starting August 9th!

Microsoft last week announced a connector between Microsoft Outlook, and profile information in Facebook. The service works by matching a user’s e-mail address to their Facebook profile. So if one of your contacts in Outlook is on Facebook, you will see whatever information is publicly available from their profile within Outlook (or whatever information you can access if you are “friends”). Microsoft previously announced a similar integration between Outlook at LinkedIn.

Microsoft’s move creates new challenges for organizations trying to balance the need to embrace the world of social software with concerns over security, compliance, privacy and productivity. Our 2010 benchmark of over 200 companies shows that 40% block access to public social sites such as Facebook, Twitter and YouTube, but often are forced to back off blanket bans due to employee demand or business justifications to participate in public social communities. Meanwhile, only 23% have a formal social strategy.

Allowing employees to engage with public social networks can provide real benefits in terms of building personal relationships with customers, partners, and suppliers, but of course carries risk and must be implemented with respect to information protection requirements (See Socialware’s recently released Guide to Facebook Social Networking Compliance).

We continue to spend a lot of time working with our clients to try and help them balance the need for openness with the reality of governance. Enterprise managers should take efforts by Microsoft and others to poke holes in the social firewall as further justification for a proactive enterprise social strategy.

We encourage all who submitted, all who plan on attending Enterprise 2.0 Conference Santa Clara, and anyone interested in Enterprise 2.0, to review the submissions, and vote for their favorites. Submissions are searchable by category, speaker or keyword, and votes received by each session will be viewable by all participants. Sessions advance to the “Crowd Favorites” stage based on community votes, after which they will be reviewed by the E2 Advisory Board. Submissions selected by the Advisory Board will then proceed to the final “Selected Sessions” stage. Selected speakers will be contacted in August.

We look forward to discovering the topics and sessions you want to see incorporated in the E2 Santa Clara 2010 Conference Agenda. If you have any questions, feel free to contact me at mfarrell@techweb.com or (415) 947-6250.

From speaking with various companies of different shapes and sizes (and from owning a small business, Chess Media Group) I can say that while Enterprise size organizations are interested in collaboration many small businesses (and some mid size as well) are not.  Again, this is an observation but it seems as though smaller businesses are more interested in business management tools as opposed to collaboration tools.  The business I run works with a handful of people and the issue that we run into is that we are using multiple tools to do things such as share documents, invoice clients, email market,  project manage, time track, and keep track of contacts.  We don’t have trouble collaborating on things that needs to get done but we do run into challenges when it comes time to using multiple tools to run our business.

It wouldn’t make sense for many smaller organizations to use Sharepoint, Jive, Cisco’s Quad, or a host of other “collaboration” tools on the market today, but what about an overall business management tool (there are several possible candidates that I am aware of)?  Something that allows me to access Gdocs, Mailchimp, Freshbooks, integrate my contacts, and do anything else I need to manage my small business from one platform/interface?

Larger organizations with hundreds or thousands of employees have specific departments that handle these tasks and their challenges are based around finding the right people and information to get their jobs done in the best way possible.

So let me ask you, are business management tools more suited for small and medium size businesses whilst collaboration tools are more relevant for larger companies?  What have you seen?

Jacob is the Principal of Chess Media Group, a Social CRM and Enterprise 2.0 consultancy.  Jacob also blogs on Social CRM and Enterprise 2.0, you can find him on Twitter @JacobM.

The deadline for the Call for Papers for E2 Santa Clara 2010 has been extended to Friday, July 2nd. We’ve partnered with Spigit to vet our submissions, and you can begin the process here.

A few important things to remember:

• Sign in and create an account to make or review submissions.
• If you have an account from the E2 Boston 2010 Call for Papers, it will work here.
• Submissions can be tagged with one of our pre-populated technology areas or a unique topic of the submitter’s choice.
• All submissions made will be immediately viewable on a discussion forum.
• Submitters will be allowed to edit their submissions based on any feedback received, and can also attach files with supporting materials to submissions.
• Submissions will be searchable by technology area, keyword or speaker

After the Call for Papers submissions period closes on July 2nd, an official voting period will begin July 6th through July 16th.  The submissions with the most votes will be reviewed by the Advisory Board and considered for inclusion in the Enterprise 2.0 Conference. Please note, top number of votes does not mean automatic inclusion in the conference. We want the E2 community to collaborate, give us their most exciting ideas and experiences, and tell us what they want to see in Santa Clara in November.

Have any questions about submitting? Manuela Farrell put together a great post with more info on the the process. She is also a great resource to help answer your questions.

We look forward to your submission!

I thought the E 2.0 gang would appreciate a pointer to a new book by Rob Salkowitz, Young World Rising, where he examines the bottom-up revolution being created by young entrepreneurs in parts of the world with a young and growing working-age population. I have previously talked about Rob and his work on the interplay of demographics, generational effects and 2.0 technologies (in my SM vs. KM post, and in my review of Rob’s previous book, co-authored with Dan Rasmus). This is easily his best work so far.

I posted an interview with Rob on the Trailmeme blog, with links to the book. Rob shared some fascinating views on technology and demographics, and I have included a brief introduction to his work for those who are new to this important subject. Check it out.