It seems to me that, in this post, when you say a “firm”, you actually have in mind only one of its formats: hierarchical organization with centralized control and vertical chains of command. In the past 30 years (if not longer), there emerged a lot more hybrid models of the firm, which combine properties of hierarchies with markets.

Examples: a) industrial clusters (Marshall was the first who described them, long time ago, and today most famous industrial clusters are Hollywood, Silicon Valley, and Italian fashion districts). Clusters are composed of tightly connected firms which operate in spatial proximity to each other, and which pull in – often via informal ties – outside talent in on a project basis.

b) Then, there’s Toyota factory, which is a hybrid between hierarchy and a network organization, having its units functioning separately but also being loosely horizontally connected with each other.

c) Digital firms (digital marketing, production, and design shops) are designed as “heterarchies” (a term introduced by David Stark), with horizontal organization, lateral accountability and distributed authority. Their organizational form is directly related to digital technologies they are using – and possibilities of organizing, codifying, storing, and accessing knowledge that they offer. Today, a “digital shop” is a quite prominent – and successful – organizational format.

d) And then, there are examples as Xerox Park and Gore factories, which also experiment with organizational formats alternative to hierarchical firm. Consulting firm McKinsey&Co., more than a decade ago, devised a system of codifying and tagging its knowledge in order to make the most out of it. (For this, and other examples, see “The Biology of Business: Decoding the Natural Laws of An Enterprise” by Brook Manville).

All these innovative organizational formats emerged as attempts to overcome the shortcomings of a traditional hierarchy in organizing and accessing its own knowledge. Some, if not most of the above, are pretty successful in doing so. What they all have in common is decentralization of knowledge management. So, while hierarchical firms are, as you say, “no longer efficient and effective means for collecting and sourcing innovative ideas and insights”, I would not extend this conclusion to all firms.

And then, hierarchical firms are still the most efficient organizational forms for conducting routinized tasks.

There is something more important here, though (and I am sure you already know this): the boundaries between a firm and its environment were never really closed (as Granovetter and Marshall have shown). John Seely Brown and Paul Duguid analyzed the firm in relation to its environment as “networks of practice” and “communities of practice” (where communities of practice exist within a firm, and networks of practice span throughout firms). This is even more true today, with a networks and communities of “amateurs”. who are easy to find and mobilize. More famous examples are P&G, Dell, Netflix, Starbucks, etc. are some of the most common examples, and I am sure that you know about many, many more.

This kind of open organization found its way in advertising, too – which was a traditionally “proprietary ideas” based business: http://www.nytimes.com/2009/10/29/business/media/29adco.html?_r=1

What I find most interesting here is that firms still exist in order to organize this crowdsourced knowledge and make it part of its own production process. Communication costs are really high in crowdsourcing communities, and they need some sort of “control” (at the end of the day, firms crowdsource in order to improve their own production/knowledge/innovation, that is, they use crowdsourcing towards measurable results.) So, at the same time you’d want to increase communication and to decrease communication costs.

In contrast, a purely open source community, Linux, is a messy and often sub-optimally efficient organizational form – especially because it is a community – and not a firm. Here, communication costs are high, which results in less than optimal production efficiency (products are released quickly, but are full of “bugs” that need to be consequently corrected).

So while I am sure that in your everyday work you still encounter more of hierarchical firms than any other formats, organizational innovation like the one you describe actually started decades ago. We can only hope that digital technologies will accelerate the process.
Finally, my question “how do you know what you are looking for before you find it?” is not the one of how we make unexpected connections (which social media facilitate, as you rightly observe), but how to recognize – among all of them – the “right” solution? That is, there is a double uncertainty here: first, there is ambiguity (which is interpretative uncertainty, how do you know that what you bump into you can use for solving your problem), and then, there is “ambage” (Harrison White’s term for vagueness in social pattern). Organizational response to ambiguity is to rapidly generate a “best fit” to environmental challenge: to create a flexibly organized, soft-assembled team of highly skilled agents that can recombine their diverse knowledge in order to respond to ambiguous task (and social media and crowdsourcing software are critical here). Organizational solution to “ambage” is, however, more complex: it requires knowledge of “who knows what” – i.e. it requires a rapid search, detection and mobilization of resources required for the solution of the ambiguous environmental challenge. That is, sometimes, we just don’t have time to rely on the organizational dynamics of pure crowdsourcing.