Tapestry.net?

[rising_moon]

There's a SMOP here somewhere.

I've been thinking for a while about the challenges -- and potentials -- of multithreaded conversation.

Recent LJ upheaval, at least in the SCA communities, has illustrated that several nodes of conversation can be active and engaged, but over time tend to favor (with people-energy and bandwidth) the incendiary over the moderate, whether the hot topics are productive or not.

LJ encourages multithreaded conversation -- however, the conversations are complicated by friends-locks and -lists. I don't read SCA_east, but I hear about it. In all likelihood, there are several recursive conversations about topics introduced there, all over LJ.

Bear with me... we're talking about threading per se, not a particular topic in SCA_east.

LJ is also notable in that it encourages Memes. I mean "meme" in the historical sense: a practice repeated within a culture. alexx_kay summarized a while back that any particular "lj meme" like a poll or list of favorites is not a meme; rather, the practice of introducing a poll or list is a meme. My buddy did it, now I repeat it. Meme.

I've also noted (led by experts in the field) that the foundation of Collective Intelligence being laid on the internets in the form of Wikipedia, Craigslist, Y! Forum, and basically any open-source content aggregation system, is doing quite well providing first-level information on a wide range of topics.

These systems encourage online community-building and information-sharing. They permit more-or-less single-threaded commentary on limited topics to varying depths: deep for Wikipedia (check the History tab on the Davis Square article) and shallow for Craigslist, for example.

Layer on top of these notions (recursive multithreading, memes, obsessive engagement, community-building) and Filter in the question raised in a bunch of articles I've read recently which, sifted, seem to call for a global/international idea collection, aggregation, ranking, and reporting system to help Solve the World's Problems(TM). See:

• "Toward Ecosophy" http://thearchdruidreport.blogspot.com/2009/02/toward-ecosophy.html


• Seed Magazine, Feb 2009, page 72, "Scientific Multilateralism"


• "Can We Exploit Collective Intelligence for Collaborative Deliberation? The Case of the Climate Change Collaboratorium" http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1084069

So I've been wondering. If a global database of ideas, and the threaded commentary that builds and stretches the ideas, will help save the world, then:

The system would need thread-node analytics and realtime alerts to point commenters to an existing topic and summarize the node state. The system would need disk space, translators, and a whole lot of other things to work right, but what's really missing, the SMOP (thank you again, justjanus) is the analytics: how to make sense of the inpouring ocean of data.

What else? What are the challenges?

Is this kind of thing even possible? I've seen the fury and fire generated by a dumbass LJ post.

Can we harness it and use the fire for good?

Comments

[dreda.livejournal.com]

I have nothing to say immediately except that I am totally besotted with your big sexy brain.

[slyppi.livejournal.com]

What she said. Completely smitten.

[shadowswords.livejournal.com]

First milestone shall be reached when the SMOP correctly links above comment to above post.

[rising-moon.livejournal.com]

Ha!

[wiley-b.livejournal.com]

I have a rather obsessive interest in community building using software, I've been doing little else professionally for the last decade or so. For the last week or so I've fallen into a design trap created by another recent LJ post, which thoroughly displayed the benefits of being able to operate (relatively) anonymously within a community.

We seem to have come to about the same place from two different directions, though my focus was more on the community aspect than the collaborative problem-solving bits. The sticking point, as usual, is people. If we do away with them, things become surprisingly easy. *ahem*

Rather, directing people to the places where their interests and knowledge can contribute while providing the tools to help focus their interactions positively (and filtering out the noise). To that end, a lot of my recent focus has been on how to present textual data in a variety of contexts to maintain engagement and help with information gathering and reference. The other area that I think would be useful to your problem is facet-based community management and arbitration, which allows users varying degrees of anonymity, without losing the context of the conversation. Which is just a fancy way of saying "giving every user pseudonyms, one of which is pseudo-anonymous". I have some rough ideas on user arbitration and community standards enforcement, as well, but some of this is actually easier to code than summarize.

You folks should come over for dinner some time, I'll steal a white board from work. ;-)

[wiley-b.livejournal.com]

Also, whatever solution you come up with should feature Siftables (http://www.ted.com/talks/david_merrill_demos_siftables_the_smart_blocks.html), because they're just that cool.

[rising-moon.livejournal.com]

Yes! They are totally awesome.

(The kid pointing blocks at the screen? The word games? Unbelievably cool.)

[miekec]

Oh my. That is one of the most excellent GUIs I've seen. Just unbelievably awesome. (traipsing into this LJ via hissilliness)

[rising-moon.livejournal.com]

The sticking point, as usual, is people. If we do away with them, things become surprisingly easy. *ahem*

Hee!

The noise-filtering and direction capability is exactly what I was trying to describe -- the "facet" sounds similar to the element I was calling a "node" for want of another term. I like both, for different reasons. :)

Yes: maintain engagement, consolidate arguments/comments on a given topic, and filter out noise, and allow the cycle to repeat. I'd suggest that the facets/nodes also somehow breed additional facets and nodes, allowing ideas to germinate and then pollinate.

[wiley-b.livejournal.com]

That sounds like good fun. A system where each user could have multiple facets (personas), and each post (node?) could have multiple facets as well (possibly on a per-paragraph or per-bullet basis) allowing users to easily create idea mash-ups by combining bits of text from varied sources instead of copying and pasting. The value here is that if the original source material gets updated, the user has the option to update to the latest version or fork it to make their own updates. Allow people to vote on a per-facet basis, slap a 3D version of the CoolIris interface on it and you're good to go.

Upon further consideration, what I just described is really just a wiki with article competition, cross-threaded discussions and entries, and a karma system. I have to admit, though, the idea of an LJ/Wiki/Subversion hybrid makes me a little faint. It would probably be pretty good for writing legal documents and tech specs, though.

[rising-moon.livejournal.com]

Yes, yes, yes. That. Everything you said after "really just". That and a place to stand, and we can move the earth.

Dinner. Whiteboard. :)

maybe the Dalai Lama's right?

[digitalt.livejournal.com]

we can all solve our problems together if we work on them respectfully together?

I guess he'd know.

Re: maybe the Dalai Lama's right?

[digitalt.livejournal.com]

that was not T. Guess who? :-/

Re: maybe the Dalai Lama's right?

[rising-moon.livejournal.com]

Ha! I can make a broad leap at this answer... :)

And yes, indeed: I agree with the DL in this as in many things. I am hopeful about the prospect of combining all of our brainpower, and would love to help find a way to do that democratically. In the "divine chaos" sense.

[learnedax.livejournal.com]

Hmm. While I think there's a lot of interesting potential in such a system, I'm somewhat dubious about the efficacy of collective intelligence for solving thorny world problems, at least without careful cultivation.

I saw an interesting essay a while back (that I cannot furnish a convenient link to, sadly) about the places where collective intelligence worked and where it failed. The author used a predominantly statistical basis for his points, and they seem difficult to argue with at a basic level: if, within the pool of people whose intelligence you draw, the average person has a better than 50% chance of recognizing The Right Thing, a larger pool will trend towards rightness... but if the chance is less than 50%, the trend is the opposite. Hence, it's easy to use collective intelligence to construct the list of U.S. presidents, and very very hard to use it to construct a solution for world peace.

So, if you want to solve the world's problems this way, what you need is a way to filter your contributing pool such that it only includes those who are likely to come up with great ideas, and recognize great ideas in others - I don't know how to do that, but once the odds are above 50%, you should be golden.

(And, to pick a nit, I disagree mildly with your usage of "meme". Memes are ideas more than they are practices, e.g. "Magenta is not a color" is a meme, because it's a notion that people pass on, but "List 25 random things about yourself" is not, because there's no unifying belief to it, although there are memes loosely associated with it. The notion that things people pass around on LJ are memes is, itself, a meme, though I would argue that it's cancerous.

But, moving to an actually useful point, a system that can recognize a meme as it propagates, and somehow encourage propagation of more Correct memes, is, I think, exactly what you're seeking to build. With just, as they say, a SMOP.)

[rising-moon.livejournal.com]

careful cultivation

Yes, this is the SMOP I speak of: how to categorize, sort, and simplify an incoming stream of information. Very knotty, that.

You've rightly identified an issue (50/50 correctness) that I had hoped to solve with ranking and node consolidation, but likely can't. To wit: where 50 people are certain that they are right, the other 50 voices will be lost in the clamor between sides. And if the idea is picked up for further discussion, the 50/50 assessment continues up the chain of interested parties and you end up with two votes of equal weight. Or, worse, you end up with one successful bad idea with a million comments behind it.

That issue brings to mind the popularity problem: the most interesting or compelling ideas garner a lot of weight in votes and discussion, but might not actually be helpful to saving the world; they might just be shiny.

At any rate I would hope that the magical analytics would list that bad idea alongside the others as the summary of its component comments and analysis, with some kind of statistical marker next to it indicating the number of converging threads. Good, Bad, or Middling, the idea would bubble up whole.

Hmm. Thank you.

[Re: Meme... I thank you for your mildness and disagreement. (Although we do agree with each other, with respect to the popular use of the word "meme" in an LJ context.) Memes are ideas and practices, and other things that propagate, like inherited genes, in the way of fashions and tunes. At any rate I believe the discussion will proceed much more enjoyably over a glass of port. I will be happy to oblige.]

Did someone mention analyzing conversation? (part 1)

[metahacker.livejournal.com]

So it sounds like you're looking for topic extraction / recognition technologies, coupled with some sort of spider, some sort of "relevance" dial (i.e. turn it up for more specificity, down for more volume), and some way to get enough context to actually jump into the conversation once you find one.

This is a grand idea. I want this, ten years ago. Finding out who was working on what was the bulk of getting my degree. Likewise, finding stuff on the web

The first bit (topic recognition) I worked on for a while, some years ago, before punting as too hard for the time available. It's a hard project, and is under active development at places similar to (but not necessarily) my old employer. It's decently well funded, because folks like the NSA really would dearly love to be able to do it. They want alerts like that so when someone is talking about, say, renting a van and also where to get some fertilizer, they can find out and then figure out who and when and where and how realistic. This was part of the motivation behind Carnivore and other such large-scale projects to collect data.

It's hard because no one can agree on what a "topic" is, and when you look at conversations closely, people don't stay on anything like a topic. We wander all over, and come back, and it's "close", but it's not really something you can summarize. I'm always interested to note what people put in the titles to their posts or comments, because that's the human equivalent to summarizing/defining the 'topic' -- and we all know how mediocre the link between post title and actual-topic-discussed is. There was some nice work on forward- and backward-looking-centers that I thought could be applied to chaining together a topic, but it hasn't panned out so far.

It's hard because it depends on the underlying knowledge of human connections and common sense and whatnot. There are some big projects tackling this hard problem, too -- CYC being the most famous, but the stuff that cmouse is involved with over at MIT is another case (Common Sense Computing Initiative). This is great for figuring out word sense, and word distance, and whatnot, but in point of fact people have been working on "meaning distance" for a really long time now (it was part of my *advisor's* thesis, back in the 70s) and it needs a breakthrough. The breakthrough may be "more data" or "more computing power".

...cont., I ran over the comment limit.

Re: Did someone mention analyzing conversation? (part 2)

[metahacker.livejournal.com]

There's some decent work on coming up with computational methods for grinding through raw text and getting results. The one that springs to mind is Latent Semantic Analysis, which tries to cluster documents based on the words in them. Summarizing massively and slightly erroneously, LSA figures out what context each word is likely to appear in, builds a high-dimensional space where each important context is a dimension (and the probability of appearing in that context is a document's position along that axis), and then throws away the uninteresting dimensions. This gives you a rough "place" for each document, and then you build clusters based on things that are near each other.

The nice features are that it's relatively easy to implement, and gives you 'nice' results (i.e. is completely automated). The downside is that it's just flat-out wrong a lot of the time, because the words which surround a word don't actually do a great job of determining its meaning; it's better than a straight word search, but not enough. There's some additional cool work recently by Tom Griffiths to try to improve accuracy.

On another front -- one of my good friends is now working at CCI, so maybe you know Josh I...? His degree was on augmenting conversation to improve collaborative decision-making, by measuring how much information is shared, and how much needs to be shared (in the sense of disagreement). He might have some interesting thoughts on the subject. (And needs to get a forking LJ already.) He used to draw large graphs of conversation, showing how topics would get brought forward, subside, return, mutate, etc. Extremely complicated. And this was conversation in the very limited, task-oriented "game" that we'd built, with a quite limited number of things to talk about.

So that's a few ways of looking at it. Hope it helped expand your thoughts on the topic. (Uh. Pun not intended.) I can keep going if you want...

Re: Did someone mention analyzing conversation? (part 2)

[metahacker.livejournal.com]

Oh! Hunh. Josh Tenenbaum's at CCI. So, the Griffiths work above? Colliding with the CCI work. ;)