People move on because they seem to assume that this is incompetence at work. Search is sooo 1.0, right? It’s been solved, and we’re just fumbling the execution, right? You usually get some sort of ironic joke along the lines of “wow, it is so easy to find stuff out there on the public Web, and here with all our resources, we can’t even do search right.”

And then the conversation tends to move on to more obviously “2.0″ things like blogs, wikis, how to increase participation, and my personal pet peeve: annoying moaning about “culture change.”

Hold on. Rewind. Let’s go back to search and think for a moment. I have a theory here, and I’d like to see if all you smart E 2.0 guys agree. I have reached a radical conclusion: broken search is the problem, but fixing search is not the solution. Search breaks behind the firewall for social, not technical reasons.

How Search Breaks Behind the Firewall

Let’s start with the blindingly obvious, and then draw some weird conclusions. Here are the most common reasons that come up:

1. Icebergs: Too much stuff that “ought to be shared” is on people’s desktops, or sitting in emails in attachment form.
2. Digital Landfills: There’s still tons of those random fileservers all over the place, where people simply dump files. They “ought” to put it into content management systems.
3. It ain’t a net: Web-based content is just easier to crawl. You have all those nice, friendly sociable things like robots.txt files that tell crawlers where to go and not go. Intranets are much ruder. It’s hard to take a proper inventory of everything that’s not on the searchable “grid” even it is nominally “online” in some form. Corporate IT “ought” to clamp down on undisciplined asset sprawl.
4. Those pesky formats: PDFs, PPTs, Excel spreadsheets, random proprietary databases with query interfaces.
5. Permissions: Lots of need-to-know systems with non single-sign-on protections. No way you can give your friendly Intranet spider ALL the skeleton keys, right? Corporate IT “ought” to enforce corporation-wide single-sign on.

Sounds like a purely technical problem doesn’t it? Get it all SOAn up (apologies for the terrible pun), create the holy grail “email attachment killing” technology (or alternately, figure out a way to make email “socially searchable”).

Why Search Breaks Behind the Firewall

Okay, I’ve been leading you to what I believe is the wrong solution. Everything above is merely a symptom. You could put the most talented engineers onto the problem, spend billions, and still end up with a mess.

You see the problem is not technical. It is social.

The key here is all those “ought” phrases.  Who’s recommending the “ought”? Who is being recommended as the “ought to do something” constituency? There’s no such thing as a person or organization without an agenda within a corporation. If the IT czar is a psycho with crazy ambitions, do you really want him/her to have all the skeleton keys? Is the person saying “ought” things about email attachments a librarian trying to make his/her job easier? Or somebody who feels cut out of certain conversations? Or somebody who is genuinely pointing out a non-political problem with sloppy sharing?

We often forget that the public Web is a very democratic place. Authority is democratically earned. PageRank is democratic. We link to stuff we like, authors we like. Google’s algorithms mine a vast democratic vote. Stuff on the public Web is online primarily because people WANT it online. They WANT it to bubble up. There are no big devious intentions. On the public Web, popularity equals power in very direct and simple ways.

Now step behind the firewall. What do you see?

1. Every single email ends up being an act of political judgment. To, cc and Bcc are three words about which I could write an entire book.
2. Need to know and organization charts/cascade patterns beat democratic content popularity and “Word of Mouth” information travel hands down. This surprises people: we often remark about how much work gets done at the watercooler. But this does not mean more gets done that way than through formal channels. The formal channels carry 80% of the communication. The reason we focus on the watercooler is that most of the politically sensitive stuff travels that way.
3. On the Web, people read people they like. Inside the firewall, everybody is constantly trying to figure out who’s important, who has the money, whose stock is up, whose stock is down. Whose coat-tails to ride. Which fool to suffer gladly for the moment, which battle to pick.  In other words there is very low correlation between information flows and friendship networks. You may pay most attention to people you hate or fear. Your best friend may be your lunch buddy from another department, and while you may be on each other’s blogrolls outside work, you have no connection inside and don’t vote up each other’s information contributions. Oneworkgroup may be trying to stay under the radar in skunkworks mode, while another may have a reason for wanting a dog-and-pony influence roadshow this year.

Forget blogs, forget wikis. Quite often intranet bloggers and wiki champions are the zero-influence noise signals (and I say this despite being one myself!). Most of the powerful people won’t blog. Let’s say a bright young intern writes a killer strategy brief/analysis of the company’s positioning as an internal blog, which gets a lot of comments and chatter.  Sure, an alert VP might spot it and hire the kid and groom him/her (okay I’ll admit it, that happened to me!). But look at what that does to search.  That page might well top the rankings by normal “democratic” search logic for the word “strategy,” but truth be told, the most important strategy documents are probably some low-distribution emails/meeting minutes, or some very carefully judged piece of officialese that comes out as a fait accompli, when it is useless to everybody else. Most people take one look at it, and yawn, but it is designed to send the right coded, and tactically timed signals about who has what piece of turf, to the few people who can interpret it. This document may get buried.

There are even rules. Like actual government, go-to-jail rules, that prevent the most important stuff from being shared at all beyond corporate officers, let alone discussed.

Some of the polyannish types might claim that this implies the organization is broken and that there is a need for more transparent, open, democratic governance.

I say, HELL no! This stuff may sound horrible, but large companies are not democratic for a reason. They exist to make money, not as laboratories for empowerment or social engineering.  The authority structures, communication flow patterns, emergent models of what’s hidden and how and why, versus what’s open, are all designed to work. Sure, you can find a lot of alternative models that are starting to work, but let’s not throw out the old without understanding how and why it works.

Which brings me to my main point. Search is political.

Search is Political

We often forget that Web 2.0 is built on top of Web 1.0. It assumes functioning search as part of the canvas.  Google and other search majors attempt strongly to maintain a “search neutrality” that matches “Net neutrality.” You may not like to hear this, but “search neutrality” is a really bad idea inside the firewall.

Inside the organization, the main form of authority and control is based on patterns of information sharing. We assume that function is accomplished by permission controls on content management systems and the like. Not true. Most information flow control is not based on restricting access, but limiting the very knowledge that a certain piece of information even exists.

And this is a good thing. If a lunatic in some toxic silo has evil designs on another department, then even the knowledge that a certain document exists may need to be hidden from him/her. There may be very good tactical reasons for burying some information in a complicated email thread, rather than being shared with clarity. Harsh but true: there are many good things that get done because someone was smart enough to cover their actions with nominal disclosure (“but it was in that email I sent 3 weeks ago, don’t act surprised.”)

Yes, such toxic politics can be really bad for an organization, but the solution is not to deny that it exists, but be smart about managing it.  Ben Horowitz has a brilliant recent post about what to do about minimizing politics from the CEO point of view, but it is really everybody’s job to be sophisticated about this stuff. And shoving a democratic search engine into your information ecosystem is a pretty naive thing to do.

A large organization is a contentious place. Alignment isn’t some idealized state of harmony to aspire to. It is a decision-by-decision evolving reality as power and influence shift through the organization chart. And again, this is a good thing. If things are going badly, you want different organizations to debate and express dissent, negotiate and if necessary, yes, hide information. You might even want them to work against each other for a bit of fairly-structured internal competition if you are not sure who has it right.

Public Web models of search blunder into this intricate and complex piece of machinery like clueless bulls in a china shop. This is the reason why they very rarely get much traction. It is in almost nobody’s interest to have powerful enterprise-wide search in place. This may sound perverse, but the ability to penetrate the opacity of organizations is to a certain extent the right test of whether you deserve access at all. If you can get to it, you are smart enough to use it wisely. It’s not a great system, but it’s better than naive search.

Instead of Search

I’ve been thinking hard about this problem, and I don’t think enterprise search can or should be “fixed.” The fundamental social and information flow assumptions of “search” need to be deconstructed and reconstructed for the enterprise.  Local/silo search within single sites/assets is fine. Enterprise-wide search in its naive form is a terrible idea.  And it is actually a good thing it is being done badly. Doing enterprise search well will provide a LOT more visibility to the most useless parts of the information ecosystem within an enterprise, and provide more incentives to hide the important information. Search infrastructure can provide a false sense of security, openness, transparency and “being informed.”

What can we do instead of search? I have some ideas, but they are still at a very early stage. But I do know that a smart discovery function is essential before the rest of the E 2.0 suite can transform organizations. The Web 1.0 agenda is still unfinished within the enterprise, so let’s be cautious about moving on to the E 2.0 agenda. Plumbing before furniture.

I am curious about how many people agree with my diagnosis or have alternative ideas/mental models here. Apologies if I’ve offended any huge advocates of enterprise search.

(Anyone know of any interesting forums/email lists about this issue? If not, I might be interested in starting one. Email me if you are interested, and I’ll start some sort of Google group or something if there’s enough interest).

Venkatesh Rao is an Entrepreneur-in-Residence at Xerox, and the manager of the Trailmeme project. He blogs at Ribbonfarm.com. These views are his own, and do not reflect those of his employer.

I thought the E 2.0 gang would appreciate a pointer to a new book by Rob Salkowitz, Young World Rising, where he examines the bottom-up revolution being created by young entrepreneurs in parts of the world with a young and growing working-age population. I have previously talked about Rob and his work on the interplay of demographics, generational effects and 2.0 technologies (in my SM vs. KM post, and in my review of Rob’s previous book, co-authored with Dan Rasmus). This is easily his best work so far.

I posted an interview with Rob on the Trailmeme blog, with links to the book. Rob shared some fascinating views on technology and demographics, and I have included a brief introduction to his work for those who are new to this important subject. Check it out.

Soldiers, Privateers, Mercenaries and Web Technology

One of the fun aspects of Web product development is that you get to think up usage scenarios, and interesting personas to go with them. A few months back, an old classmate, who now works for a big Wall Street firm, emailed asking if we had an enterprise version of trailmeme available. We don’t, and at that point we didn’t even have fully-thought-through ideas; we only had some poorly-defined conceptual vaporware for potential enterprise markets. But the email got me thinking, and a startling thought hit me: the old idea that you develop a Web 2.0 product for consumer/SMB users and harden/evolve it for enterprise users is changing very rapidly. This is because the enterprise itself is changing very rapidly, due to the emergence of three very different IT-user personas, who demand very different future enterprise IT strategies. The three user personas I’ve defined are soldier, privateer and mercenary. Depending on which one dominates the future of work, enterprise IT could evolve in three very different directions. Which means the enterprise design for products like trailmeme could evolve in radically different ways.

One of the more radical conclusions in the article is that perhaps there is no real need for a separate category called “Enterprise 2.0″ software/IT, given the way the workforce (and as a result, the enterprise) are changing, with blurring boundaries and increasing numbers of people employed in non-traditional models.

Read the full post at here

The Problem of Enterprise Document Integration

A typical large-scale Intranet is a cheerful mess of Websites running on different platforms, often from entirely different generations and technology paradigms. Within Xerox for instance, we have an ancient 1.0 level default Intranet (sorry any Xerox IT guys reading this, but you know it is true!), several repositories running on our homegrown Docushare CMS, some Sharepoint installs, and of course, a whole bunch of special-purpose local workgroup sites running on systems ranging from Confluence to Mediawiki. This is as it should be — a few slowly-evolving sensible enterprise defaults, complemented by some faster-moving local solutions for local problems. Besides this general content management infrastructure, like every large corporation, we naturally have all those specialized platforms for functions like elearning and HR.

This information environment though, doesn’t lend itself to easy organization and re-organization around changing and often transitory purposes. How, for example, would you pull together all the online resources needed to help employees from a newly acquired company orient themselves? Or give your employees an easy way to explore all the information related to a particular market?

The traditional 1.0 answer (which we know does not work) is “portal.” Portals are simply too hard to set up, too slow to adapt, and too centralized (I often rant about “Portalitis” — multiple silos each wanting to centralize information around themselves, leading to a proliferation of portals with a lot of redundancies). In reality, valuable information exists all over in a distributed way, and is constantly shifting.  Most of us are reduced to emailing bunches of cut-and-paste links and attachments to each other for every situation.

This is the problem of enterprise document integration, EDI: The problem of rapidly assembling and sharing an organized set of online resources to suit a given situation.

I don’t claim to have the complete, definitive answer, but I do think the idea of a “trail” is one potential solution. So here’s my quick soapbox pitch, before I get back to discussing the broader EDI problem. Keep in mind that what I am describing below is the simpler consumer version that we already have out there. The stuff isn’t yet ready to take on full-fledged EDI.

What is Xerox Trails?

Trails allow you to collect, organize and present (in short, “blaze”) collections of Web content using the user experience metaphor of a “trail” (as in hiking). You navigate a trail using a “Trail Map” view and a “Following” view. Show is easier than tell, so here is an example trail I created on our destination site, trailmeme.com, about the “Social Media vs. Knowledge Management” debate I triggered a year and a half ago, organizing the original article and some of the most interesting responses. Here is a screenshot of the “Trail Map” view of the trail (Click here to go to the live, clickable map: you’ll need to have Flash enabled in your browser).

And here is the “Following” view (think of it as a “ground-level/street-level” view). The idea is to support navigation with the sorts of signs you might see on a highway or a hiking trail at the ground level, by framing the original source with a useful navigation overlay. Click here to start “following” the trail.

This is the basic user experience. We also have a plugin that implements this experience on WordPress blogs (our project blog has this enabled), and we have a similar plugin for MediaWiki in the works. These allow you to create the “trails” user experience on your own site. Both are open source (GPL) and if you are interested in supporting trails on other platforms, talk to me.

And of course, since we are Xerox, there is a print and publishing angle to all of this. Trails (when appropriate) can be sequenced and converted to instant PDFs with one click (currently only supported on the WordPress and MediaWiki plugin version).

So that’s the technology itself.  Now let’s get back to the EDI question.

The Varieties of EDI Problems

Here is a quick half-dozen variants of the EDI problem that occur to me, that frequently appear in the enterprise. One of the big things I am interested in hearing from you guys is: are these the right EDI use cases? Are there better/more important ones?

1. A bid team for a major and complex enterprise sale collaborating to rapidly assemble existing resources into a raw material package to support the proposal writing
2. An HR/eLearning department creating an ad-hoc, just-in-time “course pack” out of stuff already scattered across the intranet, to support a course on the eLearning system
3. A workgroup involved in competitive analysis gradually putting together a “market view” picture by putting together internal material, analyst reports, press releases, competitor news and so forth, into a sort of constantly evolving online intelligence dossier
4. Creating an ad-hoc org chart out of wherever people choose to post their Intranet (or even public) profiles (even for companies that have a single social network/expert locator system, the ordinary model of organizing with tags and a couple of matrix dimensions reflecting the static organizational structure, rarely reflects the real org chart of ad-hoc teams, workgroups, dotted-line reporting and so forth)
5. Creating a weekly or monthly newsletter out of press coverage of your company
6. Creating a collection of information for a new “community of practice” type group, without creating yet another silo with redundant copies of information

Some of these problems, of course, can be solved with other technologies, including internal social bookmarking or collections of RSS feeds in a truly SOA-enabled Intranet. Where trails add value is where there is a need to add some meaningful sequencing (“read this before that”), so that the collection becomes a guided narrative of sorts. Some questions that my research team and I are interested in include:

1. What are the most critical EDI problems and pain points?
2. When the existing solutions to these problems work poorly, how do they fail?
3. Lost opportunities: what problems are being finessed or not faced at all, due to the difficulty of the problem?
4. How transient are information organization needs? How much organization lasts for days, versus months, versus quarters, versus years?
5. How dynamic are information collections, how often do they change during active and quiescent phases of the respective workflows?
6. What sort of security and permissions architecture constraints impact EDI?
7. What regulatory and “need to know” constraints impact EDI?

Invite Code and an Invitation to Dream

One of the smart things Xerox does is organize “dreaming sessions” between researchers and customers (who otherwise rarely meet).  That’s kinda what I am hoping to create here, virtually. If there is enough interest and I am able to scrounge together a budget, we might even do a face-to-face dreaming session at some point.

So here you go, here is how you can experiment with the technology in its current consumer incarnation:

• Go to the login/registration page of trailmeme
• Request an invitation to register with the coupon code: e2blog

You should see the invitation in your email in 10-15 minutes. If you don’t, check your spam folder. Once you register, you can create your own trails. The code will work for the first 100 users who sign up. We’ll release more batches of invites soon.

Keep in mind that this is NOT a commercially launched Xerox product. It is a labs project that is in research beta, so yes, some things (okay, several things) are kinda klutzy at the moment. But I hope you will be won over by the sheer coolness of it all.

There is a FAQ, and quick-and-dirty screencasts to help you follow/blaze trails. You can keep up with our updates  on the project blog at blog.trailmeme.com

So play around, create some trails, and feel free to post any comments/feature suggestions/high-level conceptual thoughts on EDI, anything you like. You can also email me directly at venkatesh.rao@xerox.com if you want to share private comments or just want to connect and talk about this.

If you are itching to use this inside the firewall, and you happen to run WordPress inside your org, you can of course try the plugin right away.

And I’d appreciate any reblogs. Feel free to share the code too.

Venkatesh Rao is a researcher in the Xerox Innovation Group. His personal blog is at ribbonfarm.com. While his posts on this blog usually come with the disclaimer that the opinions are not that of his employer, in this case, that is obviously not true.

McAfee and the Left Brain

McAfee’s book is dry, to the point and somewhat formulaic, but it gets the job done. That job is to serve as ammunition for E 2.0 evangelists, change agents and executive sponsors everywhere.  The book is a well-balanced mix of anecdotes and concepts, with a couple of new acronyms thrown in (ESSP: emergent social software platform). Astutely, McAfee starts with a personal story that gently encourages skeptics to identify with his own trajectory. He recounts how he was personally skeptical of the claims of the Web 2.0 movement until he decided to look up “Skinhead” on Wikipedia (the idea being that as a contentious flame-bait topic, it would stress Wikipedia’s claims of being able to produce level-headed articles on contentious subjects). Wikipedia passed the test, won over McAfee, and presumably, at this point, the book will sort of draw the skeptical reader into adopting a more open mindset.

From there, the book proceeds systematically to a chapter-length answer to the basic “what problems does this solve?” question asked by passive aggressive blockers and nay-sayers every evangelist must face, titled “vexations and missed opportunities in group work.”

We then get a brief history (too brief, as we’ll see) and a treatment of how E 2.0 does what earlier generations of Enterprise groupware and Knowledge Management (KM) failed to do (yay! for my early KM vs. SM thesis). The usual laundry list of design principle differences are trotted out, starting with the basic distinction between folksonomies versus top-down imposed taxonomies.

After what are essentially “neutralize the nay-sayers” ammunition chapters, we get a more positively-framed chapter listing the benefits. That concludes Part I. If you are using this as your tactical evangelism manual, by this time you should have convinced your quarry of the basic value proposition, and have him or her prepped for the deployment discussion.

Part II proceeds with a higher-level neutralization task: the roadblocks to deployment and operationalization. A particularly valuable section is “red herrings” — the huge list of idiotic objections that slow adoption (arising rarely from genuine ignorance, and usually from basic risk aversion, inertia, posturing by self-styled “critics” of flavors of the month, and intentional passive aggression). This bit should help newbie evangelists avoid particularly vicious and tortuous bunny trails and arguments that go nowhere, with people who don’t intend on moving, and who don’t need to be moved. Then we get a sketch of the the legitimate concerns, and a wise caution to prepare for the long haul. I was pleased to see a strong bit of advice to avoid the ROI question and focus on “are we moving?” as the core operational strategy question (the book does not address business model strategy at any length).

The book concludes with two chapters titled “Going Mainstream” and “Looking Ahead” which are roadmap fodder for the unfortunate middle managers everywhere tasked with creating those powerpoint slides for senior exec briefings. “Looking Ahead” has one nice bit about a “Model 1″ (goal-directed/waterfall) vs. “Model 2″ (open-ended/agile) approaches, and should help evangelists set the right expectations.

Overall, the book is a workman-like tactical manual aimed at driving change. That does make it somewhat tedious and boring to those of us who’ve already been down this road and learned the lessons the hard way, but it is critical to understand that this sort of book is essential for the army of second-generation evangelists out there (think Marines vs. Army), to make sure they are solving the right problems, and solving them right. A big part of this is knowing which battles to pick, and knowing who must be persuaded and who needs to be sidestepped.

If I have one criticism, it is that the book is almost too cynically level-headed and focused on being the ammunition needed by the frontline warriors. Even frontline folks have souls and need some inspiration and a sense of wonder at what we still don’t understand, to keep them going.  Rather than focusing on converting the resistance through sparking genuine “Aha!” moments, the book focuses on helping believers win, outmaneuver the resistance, and push through their agenda. Winning hearts, minds and souls isn’t something the book will help with.

The  book will also not help those who already have frontline experience in their search for a broader philosophical synthesis and intellectual stimulation. That book is the Fraser/Dutta one.

Fraser/Dutta and the Right Brain

Throwing Sheep in the Boardroom is a very different book. It eschews all talk of mundane practical issues except where such talk helps to make broader social-psychological or historical points. The book also does not suffer fools gladly, and is written at a high, nearly academic level. I suspect (and I am being pessimistic here) that the book will simply be beyond the comprehension of the majority of bright-eyed and clueless evangelists who just want to whine about (ugh) “culture change.” While their efforts may be propped up and rendered safe in the short term by good practical manuals like McAfee’s, to operate at higher levels of sophistication, understanding at this level is needed. Most will not achieve it. They will be doomed to energetic and misguided evangelism that will do more harm than good.

With that caveat, a look at the book. I have less to say here since it is a dense, closely-argued book that is anything but formulaic.

The overall narrative attempts to make a single broad point: Enterprise 2.0 tools up-end power structures in radical ways, and the higher-level social psychology questions that matter are all about the various tensions that play out in individual power struggles and authority battles.

To this end, the book situates the E 2.0 debate in a much broader context, drawing its inspiration from the power struggles between the “hierarchical” monarchies and “horizontal” Knights Templar in medieval European Christendom. There is a nuanced discussion of power, status, authority (ascriptive as well as real), and how these evolve when there is a struggle between hierarchical and network modes of organization. There are deep discussions of social identity, privacy, loyalty and friendship. Where McAfee and others take ideas like ratings at face value, Fraser and Dutta dive deep into the motivations driving rating behavior.

The book is very loosely structured at an overt level, but is very tight at the sentence level. There are three parts titled Identity, Status and Power, each with 5 chapters. There are no crutches for those with weak stomachs. No anecdote boxes, no graphs, no bullet lists of key concepts and talking points. Instead, each chapter is a single densely argued essay that dives into a few tricky themes. The only concession to “popular” accessibility is a single well-chosen cartoon at the start of each part. Don’t expect to easily translate this book to Powerpoint.

It is a keep-up-or-shut-up book that takes no prisoners. While not scholarly in the strict sense, there are plenty of references to the sort of original literature that most readers would never think of reading. Probably the best thing about the book is that it pointedly avoids annoying Kumbaya talk about how social software is all about world peace and Burning Man sensibilities. It is not, it is about the same old ornery self-interested, calculating individualists that power is always about. Again, I feel vindicated (see The Unsociable, Radically Individualist Soul of Social Media)

The overall effect of the book is that it delivers a brilliant and much-needed boost to the level of the discourse. This is the level at which we should be talking at today. This is the level at which fresh insights are still available. This is the level at which the conversation needs to operate if what McAfee calls the “long haul” challenge is to be met. Unfortunately, I think maybe 10% of the people who need to be talking at this level will actually be able to do so. I don’t mean to be elitist here. This has simply been my personal experience. Enthusiasm for a cause, and competence in supporting it are not the same thing.

Which Should You Read?

Ideally, you should read both. Use McAfee’s book to help you fight your battles. Even if you already know all that stuff and have gone beyond, the book is the ammunition you need. Put a little bit of your own soul into the battle though.

The Fraser/Dutta material is far more dangerous and subversive. If you really get what is being talked about, you should be cautious in selecting who you discuss the ideas with. The ideas openly frame the conflicts the right way, and only those who have been won over heart and soul will be prepared to engage you at this level.

If you can read and follow McAfee, but find that Fraser/Dutta leaves you confused and fumbling, you might want to gently step back from what you are doing before you do too much damage.

Venkatesh G. Rao writes a blog on business, innovation, and other stuff at www.ribbonfarm.com, and is a Web technology researcher at Xerox. The views expressed in this post are his personal ones and do not represent the views of his employer.