December 11, 2007

DEBUNKING "Google Hijacked" - The Sky, err, The Internet, Is NOT Falling!

[I wrote this for a mailing list, before the story started spreading all over the usual places. I didn't even get through there [sad face image]]

Regarding Lauren Weinstein's post on "Google Hijacked -- Major ISP to Intercept and Modify Web Pages"

This is apparently not quite the danger it may appear at first glance.

The product at issue, PerfTech, seems to have been around AND USED for a while, for example:
Code Amber Utilizes PerfTech to Reach ISP Customers
February 2, 2005

"Code Amber ( and Wide Open West (WOW!) Internet and Cable last week delivered an Indiana Amber Alert to customers in the neighboring state of Ohio, enabled by a product deployed in WOW!'s network that allows the Internet provider to deliver bulletins directly to the screens of its browsing subscribers."

A look at shows this is hardly a stealth application - they tout advertising-insertion as a *feature*, for subsidized ISP services.

Also, is one file with an example using *Google* ... dated March 26, *2004*.

Now, it strikes me as a very obnoxious product. But I'm so tired of the "The Sky, err, The Internet, Is Falling!" paranoia every time an ISP or teleco does something, anything, that can be twisted into service for the buzzwords of Net-you-know-what.
Again, can't we be better than that?

By Seth Finkelstein | posted in cyberblather , google | on December 11, 2007 01:48 AM (Infothought permalink)
Seth Finkelstein's Infothought blog (Wikipedia, Google, censorware, and an inside view of net-politics) - Syndicate site (subscribe, RSS)

Subscribe with Bloglines      Subscribe in NewsGator Online  Google Reader or Homepage


I think you need to think a bit more about this

ISP risk their common carier status by changing the content - I bet someone will sue if this practice becomes common.

Theres also tradmark and passing off issues.

Posted by: Maurice at December 11, 2007 06:49 AM

ISP's are not common carriers. It's a complex legal issue.

Are there potential legal issues here? Sure. There always are. But this is being presented as some sort of expose, whereas the company has been kicking around and the stuff in use for years.

Posted by: Seth Finkelstein at December 11, 2007 07:12 AM

You're mistaking blogging for breaking news. In this case, blogging is serving the role of distributed collaborative research -- ammunition gathering, if you will. When people find evidence that, in their minds, supports their cause, they blog about it. To the extent others think it supports their cause, they re-blog about it. Eventually, it gets woven into the fabric of the Internet (colloquially referred to as "Google") sufficiently that when people who can actually use this information go looking, they'll have no problem finding it. Sometimes this is more consolidated (think Groklaw), but often it's quite widely federated.

Complaining about their writing in breathless, "sky is falling" tones is akin to complaining about the effectiveness of a fire company because they went with green paint on their fire trucks instead of the classic red.

Posted by: Mark Murphy at December 11, 2007 07:33 AM

I see very little "distributed collaborative research", and quite a lot of hype and paranoia.

You might think about why that's the case.

Posted by: Seth Finkelstein at December 11, 2007 07:44 AM

Scott Rosenberg has a post on something similar but hardware based that's used to target ads, or that's the pretext to scan a user's Net communications.

Deep packet inspection and the new ad targeting -

Watching What You See on the Web
New Gear Lets ISPs Track Users and Sell Targeted Ads;
More Players, Privacy Fears
December 6, 2007

This is spying. Simply. Spying. Your making light of it doesn't diminish the potential malevolence of it. The sky may not be falling but seemingly each day it gets a little more gray. A little more murky. And one day we'll be thinking that's the way it should be. We have no right to privacy and corporations can take whatever they want from us because they have control of a public entity, whether it be public roads and lands or the public communication spectrum. We're owned. All our bases belong to them. But the sky isn't falling.

Posted by: Amos Anan at December 11, 2007 10:09 PM

"Amos", that horse left the barn when Google did such ad-targeting scanning for Gmail - and I thought that was a real issue, since it was being done extensively and routinely, with a lot of potential for abuse.

Posted by: Seth Finkelstein at December 11, 2007 10:19 PM

ISPs not common carriers - I believe you but would like to understand it some more

Any good references to where it might be explained at least some?

Legalese doesn't scare me.

More significantly - any court decisions that back that up? Just because a law is written doesn't mean a judge will agree that it is valid.

Posted by: tqft at December 12, 2007 06:01 AM

Perftech uses URL redirection and Rogers is changing html code of the page. There IS a big difference between those 2.

Posted by: anonymous at December 12, 2007 07:35 AM

I agree seems there is a lot of paranoia.

Although on the other hand, if you'd asked me if ISPs would lie about the NXDOMAIN responses a while back I'd have laughed as well.

Carrying advertising in such a format is a bad idea, as it would allow the advertisers to do cross site style attacks if not monitored correctly. But that is a general issue with advertising on the web, as Microsoft found out when the advertising on the home page they used for some UK versions of IE, was trying to compromise the users systems.

But I think the relevant point, fiddling with others content, is a bad idea. If they really need to notify users of something they should email them - or perhaps intercept the whole page - half and half is not an ideal compromise.

Posted by: Simon at December 12, 2007 02:46 PM

tqft: Here's a good page, by a law site, explaining ISP vs. common carrier issues
It's a well-known topic in communications law.

anonymous: I believe you're mistaken.

Simon: A problem is that many users don't read their email as frequently as would be necessary.

Posted by: Seth Finkelstein at December 13, 2007 12:46 AM

Seth, that's not a very good page on the common carrier issue.

A good page would mention things like the FCC Computer I && II in ancient history, and move up to Brand X in recent history.

That's just off the top of my head. Sorry this isn't a more in-depth comment.

Posted by: chilled effect at December 14, 2007 03:06 PM


1) Users not reading their email is not the responsibility of the ISP. Why not create a custom-widget loadable by the users, that's much m ore effective and less intrusive.

2) How does showing that the product has been around a while do anything to debunk the fact that this is the start, or the defining start, of something much larger?

3) You've done little to debunk how technology like this will prevent breakage of other existing technologies that use the HTTP protocol. Do I need to rebuild my applications deal with someone modifying the end result of my content? I've seen this technology in action while testing sandboxed user control on a live network, I've seen it break a multitude of applications.

Posted by: Kevin Morgan at December 15, 2007 05:18 PM

chilled effect: I suspect the material you're talking about is too detailed for a first approach to a topic.

1) Custom widget? You're kidding. The idea is to reach as many people as possible.
2) It shows the sky hasn't fallen yet, thus debunking the idea that the sky is about to fall.
3) You never have had a blanket guarantee that content won't be modified under any circumstances, and this specific example - THIS ONE, THIS CASE, RIGHT HERE - is likely well within reasonable service provisions
(to be tedious, because I know it's coming, it's possible to have an unreasonable example - but this one is reasonable).

Posted by: Seth Finkelstein at December 15, 2007 05:32 PM

[SF - I made this into a guest post on its own]

Posted by: Brett Glass at December 16, 2007 05:34 PM

Seth, if you don't think this is a problem, I suggest you go back and re-read one of the Internet's foundation papers: "End-to-End Arguments in System Design" by Saltzer, Reed and Clark.

Also read the archives of the end2end mailing list by Bob Braden, and the minutes and archives of the various IAB and IETF workshops on "middleboxes".

If, after all that, you still can't see that it is an EXTRAORDINARILY bad idea for ISPs to modify packets that aren't addressed to them, then I won't try to argue any further. My philosophy here is "Don't get mad. Get even".

If I can't switch to a more enlightened ISP that doesn't mutilate my traffic (e.g., because the ISP in question has a monopoly on local broadband) I will encrypt as much of my traffic as I possibly can through that ISP, at least with SSL/TLS and preferably with IPSec. This will put an immediate end to "deep packet inspection", port blocking, TCP RST insertion, and Rogers/Perftech-style content modification, among many other increasingly heavy-handed ISP practices.

And I'll work as hard as I can to promote the same to as many Internet users as possible by writing, speaking and distributing free, open source, turnkey encryption packages.

There's even a business opportunity here: a packet relay service for users afflicted by brain-dead ISPs like Rogers. The user tunnels all their traffic to the server, which in turn provides them with unrestricted, transparent access to the rest of the Internet. This is of course less efficient than the ideal approach of getting the retail ISPs to leave their grubby hands off my packets, but if it's necessary, then we'll do it.

The main challenge is overcoming the users' presumption that they can't do anything to stop ISPs from doing whatever they want. Users have an exceptionally powerful defense of end-to-end transparency in the form of encryption. All that's needed is to make them aware of it and help them to use it.

And if all this encourages the ISPs to stop mutilating user traffic "because they can", that will be fine with me.

Posted by: Phil Karn at December 18, 2007 02:20 PM

Phil, there's theory, and there's practice. No real-world system is completely true to an abstraction. I don't think this is a great application myself, but I also don't think it's any sort of deep threat either. In fact, I think the manipulative use of the application as a political football is far more troublesome.

Posted by: Seth Finkelstein at December 18, 2007 02:41 PM

Seth, after looking at some more of your comments, I would like to suggest that before you read the paper by Saltzer Reed and Clark, you should read an introductory text on Internet protocols and architecture. Stevens' series of books is a good start.

Anyone who can't see a difference between a) Google adding ads to web pages they create themselves and b) Rogers corrupting HTTP packets sent through them from another server probably doesn't know very much about how the Internet works.

Posted by: Phil Karn at December 18, 2007 04:05 PM

Phil, if you want to say this is an ugly technical hack, I might go along with you. If you want to say it's violating a sacred trust, from which all good-thinking people should react in horror, I don't see it. There's a big different between those two.

Posted by: Seth Finkelstein at December 18, 2007 06:34 PM

Seth, I ask again: have you studied the Internet architecture and understand how the protocols fit together?

This is much more than an ugly hack (though it is certainly that). It has the potential to cause some very serious problems.

Who would have thought that an "ugly hack" as simple as returning address records for non-existent domain names would have caused so many problems? But it did.

That was nothing compared to what PerfidyTech, oops, Perftech and Rogers are doing here. Look at all the uses to which HTTP is already being put *other* than sending HTML for rendering on a browser for human eyes. Downloading software security patches, for one.

Perftech's steaming pile of crap was written by people with no understanding of (or at least respect for the importance of) the Internet architecture and the importance of end-to-end transparency to keep it all working. It is obviously proprietary -- no self-respecting open source programmer would work on it. So we have no idea exactly how it would munge any particular HTTP stream. So are you prepared to say that Perftech cannot possibly cause a serious problem with any conceivable use -- present or future -- of HTTP over port 80?

And where is it even written that you MUST speak HTTP over port 80? As you will know if you've ever studied this stuff, consenting hosts are free to speak whatever they wish over any TCP or UDP port they wish. After all, the network has no legitimate reason to even look at a transport header, much less modify the user data that follows it.

I know of only one way to avoid such risks: not muck in the first place with packets that aren't addressed to you.

But as I said, encryption will put a stop to this nonsense quite effectively. It's ironic, really. We developed IPSEC assuming that third parties (industrial spies, rogue government agents) would be our adversaries. We never considered that our own carriers would be our adversaries. But here we are, and it'll also defend against them just fine.

Posted by: Phil Karn at December 19, 2007 05:30 AM

Let's put it this way: Some ISP owners seem to disagree with you, and the fact that this has been IN USE - let me stress that, it's not new, it's not something utterly untested - seems to argue against your concerns in practice.

Software security patches should be checksummed.

When we get to an ideal world, where everyone is a geek communicating with encryption all the time, life will be different. But that's a long way off.

Posted by: Seth Finkelstein at December 19, 2007 01:00 PM

Indeed many ISPs do disagree with me. That's why it may well be futile to try to reason with them, just as it was futile to try to reason with the 19th century's railroad robber barons. That's why the government eventually stepped in with common carrier legislation and antitrust legislation. The communication companies have managed to shake off common carrier status, but their greed (and their belief that no one can stop them) will eventually do them in. The more the ISPs act like the robber barons of old, especially when there is no meaningful competition in the local broadband market, the sooner we'll have network neutrality legislation.

In the meantime, we can fight back. The ISPs do what they do because they think they can get away with anything. Just look at PerfidyTech's website if you don't believe they have big plans for this stuff that go far beyond inserting "helpful" messages about running into your monthly data transfer limits.

As I said, encryption is the users' strongest countermeasure, just as it's already the most powerful tool for users to avoid censorship in even more repressive countries. We just have to educate end-users as to its potential and make the necessary tools available and easy to use.

Oh, by the way, checksums won't help protect software patches if every single attempt to fetch them over HTTP results in a checksum failure due to your ISP inserting some unexpected and unwanted crap into what is supposed to be a transparent byte pipe between you and the server you thought you were talking to.

I don't know you, Seth, are you just somebody who takes ridiculously indefensible positions to start arguments?

Posted by: Phil Karn at December 21, 2007 04:07 AM

Phil, given that a reasonable ISP owner just said there's no problem, and this stuff has been around for years, I don't think my position is "ridiculously indefensible".

I'm having bad flashbacks to too many old arguments and much time I wasted, that I now regret :-(.

Posted by: Seth Finkelstein at December 21, 2007 03:11 PM