July 27, 2007

"Grub" crawler and Wikia Search, and working for free

News: Search Wikia Takes Steps To Crawl; Acquires Grub

"Wikia, Inc., the for-profit company developing the open source search engine Search Wikia, has acquired Grub, a distributed crawler platform, from LookSmart."

Now, let's follow the money:

Specifics of the deal were not revealed, though it is part of a larger advertising deal between Wikia and LookSmart which was announced last week.

Under the deal, LookSmart will provide text and display ads in Wikia's freely hosted wiki communities, and eventually on the Search Wikia site, Wales said. Ads will be sold by Wikia on either a cost-per-click (CPC) or cost-per-thousand impressions (CPM) model. Inventory not sold by Wikia will be back-filled by ads from LookSmart's distributed ad network.

So ... the ad-provider will give some old searching technology (I downloaded the Linux version, and it seemed to be from December 2002). The legions of free citizen-workers-for-no-money will immediately improve this, again, donating their skilled labor. The company will run ads on the system the free workers build. That's called democracy?

Bonus Link: Beware The Online Collective - Jaron Lanier

The Web 2.0 notion is that an entrepreneur comes up with some scheme that attracts huge numbers of people to participate in an activity online - like the video sharing on YouTube, for instance. Then you can "monetize" at an astronomical level by offering a way to bring ads or online purchasing to people in your gigantic crowd of participants. What is amazing about this idea is that the people are the value - and they also pay for the value they provide instead of being paid for it. For instance, when you buy something that is advertized, part of the price goes to the ads - but in the new online world, you yourself were the bait for the ad you saw. The whole cycle is remarkably efficient and concentrates giant fortunes faster than any other business scheme in history.

By Seth Finkelstein | posted in wikia-search | on July 27, 2007 04:16 PM (Infothought permalink)
Seth Finkelstein's Infothought blog (Wikipedia, Google, censorware, and an inside view of net-politics) - Syndicate site (subscribe, RSS)

Subscribe with Bloglines      Subscribe in NewsGator Online  Google Reader or Homepage


It's still eyeball monetisation.

The work that people do still belongs to them (to be asserted in case of contention).

I'd be careful to avoid suggesting that the 'free labour' is being exploited. It's the provision of the space and facilities to allow the public to collaborate on public works that attracts eyeballs - and consequently the eyeballs that are exploited (not the work).

Posted by: Crosbie Fitch at July 28, 2007 05:06 AM

> and they also pay for the value they
> provide instead of being paid for it

Actually, a YouTube user is "paid" with a free service. They can upload their video, they don't pay for conversion, for the embedded player, for bandwidth, for server CPU costs, for storage costs. That's the deal on which voluntary participation is based, and in fact it wouldn't work any other way because people wouldn't use the service if it didn't give back to them (even "data mining/ aggregation games", like the HumanBrainCloud.com game, are played because they're entertaining, not because people necessarily like data mining).

Posted by: Philipp Lenssen at July 28, 2007 01:14 PM

Seth, this troubles me as well. Jimbo is essentially using free software for his efforts. However, the software is developed largely by volunteers – and that's just the software, not the data.

I'm not sure how long he can keep that up, but the fact of the matter is they get enormous piles of cash, which they don't pay taxes on. And as you said, there's an exploitation angle as well.

Posted by: Alex J. Avriette at July 28, 2007 04:50 PM

An Interesting concept. make money by getting a community of people built the site for you. and then place adds. Sounds like a great way to make money!

Posted by: Todd Harrison - la headshot Photography at July 28, 2007 05:45 PM

Crosbie: Ah, but it's not a "public work" (except in the sense of working the public ..). It's a for-profit entity, a corporation.

Philipp: People wouldn't sell flowers at the airport for cults either, if it didn't give back to them somehow.

Alex: It's definitely a challenge. But Wikipedia actually doesn't get enormous piles of cash. It's finances are public. It runs on a million of so, most of which goes for hosting expenses.

Todd: Welcome to "Web 2.0" :-)

Posted by: Seth Finkelstein at July 28, 2007 05:51 PM

Seth, any member of the public can legally copy MediaWiki and Wikipedia. In this way the public work belongs to the public. All that Wikipedia own is the URL, the hosting resources, and the goodwill of users and contributors (which can be lost if abused).

The same applies to any other site that apparently exploits the public to create the content it hosts. The moment the site even thinks of pretending it owns the public work it hosts is the moment the public should start thinking of seizing back its work.

Posted by: Crosbie Fitch at July 29, 2007 05:05 AM

Saw this in the groklaw newspicks

Seth's blog is worth $142,264.08.
How much is your blog worth?

Now if they actually offered to buy it at that price it would be interesting.

I don't who Dane Carlson is or if he is a scammer but the simple fact that he is selling blog monetisation tips tells me that concepts such as freedom, democracy, sharing and other nice things aren't what everyone sees in blogs.

Posted by: tqft at July 30, 2007 05:44 AM

Crosbie: And the Google-juice. You forgot the Google-juice. That is NOT owned by the public!

tqft: I think that's an estimate based on the sale of Weblogs, Inc. But comparing it to me is like comparing big-city real-estate to some backwoods hovel.

Posted by: Seth Finkelstein at July 30, 2007 04:17 PM


That is simply goodwill by another name. Wikipedia do not own the Google-juice they merely enjoy it, because the public work in its accredited location enjoys the Google-juice.

Should Wikipedia fall out of favour and Spikiwedia become the new definitive destination, then Spikiwedia will get the Google-juice. The juice follows the information like paparazzi follows celebrity, it is not owned by the site that it flows to, though perhaps it could be harnessed to some extent.

However, look at the other end of Google-juice. This is a public benefit. The public are rewarded by Google's direction, and the public's gaze to publicly rewarding content commensurately rewards the owners of the sites that host such content.

There is no free lunch here.

In exchange for providing the information that the Google user seeks, Google and the website (that the user's been referred to) obtain the user's eyeballs (briefly).

If the public want the Google-juice, there's nothing stopping them providing their own resources rather than take advantage of a proprietor's.

Posted by: Crosbie Fitch at July 31, 2007 06:31 PM

So...advertisers pay Google to build software for me to use. I pay Google with very personal information about myself that Google then gives to the advertisers who pay them. Then the advertisers decide if they want to talk to me and Google arranges the conversation.

Web 2.0, do I have it???

Ooops...Forgot about the right of my soon to be ex-wife's attorney or Alberto's DOJ to demand my personal info for free whenever the hell they really want to screw me.

Is it possible to protect myself from my Ex and Alberto while at the same time allowing the AD men and the Google geeks to get paid???

If Wikia can do something like that I'm interested, otherwise I'll put the wikia bookmark next to my lycos bookmark.

Posted by: wtf at August 1, 2007 03:42 AM

Even if users of Google were anonymised, they'd still pretty much identify themselves by their search behaviour. Identity comes from interaction, not by dint of the body you posssess.

So, yes, Google is obtaining information about users (irrespective of whether it can or wants to identify their corresponding bodies). However, as long as that can't be used to incriminate the person (even given a search warrant), then I don't think there's significant leakage of private information going on here. At worst it's like being naked at a masked ball in a nudist colony.

However, given states tend to be very interested in 'body management', I suppose that they'd be unable to resist exploiting techniques to identify bodies from search records even if users were pseudonymised.

So, yup, I'll concede that it may be best if there was no correspondence between searches and the users who requested them.

This still doesn't mean publicly created works unfairly exploit the public.

Posted by: Crosbie Fitch at August 1, 2007 04:14 AM

"Even if users of Google were anonymised, they'd still pretty much identify themselves by their search behaviour. Identity comes from interaction, not by dint of the body you posssess."

I think the firefox extension 'trackmenot' is an interesting way for users to protect themselves from be being tracked by search behavior. Not perfect but it helps with plausible deniability and the infamous security through obscurity.

"So, yup, I'll concede that it may be best if there was no correspondence between searches and the users who requested them."

It be nice if there was some way to establish a sort of random master key that works once or something but could not be used again...kind of like Schrödinger's cat...it works fine for making educated guesses and then linking you to personalized advertisements but once you try to observe the private characteristics of the user...poof you interfere with the key and lose everything about them. hmmmm...

"This still doesn't mean publicly created works unfairly exploit the public."

I didn't going there. :)


If I pay illegal immigrants 10 cents on the dollar to build a public road and then the public pays me a dollar for road...have I exploited the immigrants? If instead I pay them nothing...am I off the hook for exploitation? Its an interesting one...

Posted by: wtf at August 1, 2007 06:28 AM

First replace 'illegal immigrants' with 'members of the public who happen to be passing by' and then answer the question.

Any unfairness probably occurs in your original question because the immigrants are unable to equitably exchange their labour in a free market.

Posted by: Crosbie Fitch at August 1, 2007 07:43 AM