True -- Google's good boy

Don Sambandaraksa

True -- Google's good boy

January 24, 2014

Over the past week, it has been interesting to monitor True’s fixed line internet in the wake of the transparent proxy caching incident that delivered doctored Google ad Javascript, an act that their official PR line denied ever happening.

I was contacted by a number of users who shared their own experience with the annoying BCDuplicator pop-ups. The oldest I could verify was dated in September 2013 who documented his experience on the popular expat webboard ThaiVisa.

In most cases, True’s standard response was to send someone in, reset the router, and then blame the user for having a virus on their computer.

To recap, True’s transparent proxy was serving up a doctored version of Google ad’s JavaScript file pubads_impl_32.js from one of Google’s partner.googleadservices.com servers, 173.194.126.122, but not from any of the others on the 173.194.126.0/24 subnet.

Instead of delivering Google’s targeted ad frameworks, the .js file was much smaller (7 kb instead of 59kb) and was hardcoded to deliver certain ads instead of Google’s.

Fresh installs of Windows and Linux using Chrome and Firefox have exhibited the symptom as has Android.

The delivery of the doctored file ceased on Saturday, 18 January, hours after it was blogged by a Twitter user by the name of @_jacobFish. Jacob has since taken down his blog and has disappeared, but that was only the beginning of the story.

It would seem that either True solved the problem that weekend or the attacker decided to stop the attacks and lie low. Emails shared with me from one of True’s customers however indicated that even by mid-week, the telco was still unable to replicate the issue and was asking users for help in doing so.

However, one network engineer set up a couple of scripts to repeatedly download pubads_impl_32.js from the relevant address and what happened surprised many.

True stopped caching not just that particular file, but it stopped caching all dynamic content, all JavaScript files. Only .css and static HTML files were still being cached by True’s transparent proxy. This is a reasonable thing to do given the high international bandwidth costs and had minimal impact to most users, unlike their ham-fisted attempts at caching dynamic content.

Engineers must have wondered on and off if the attack was at the DNS server and for intermittent periods throughout last week, had turned off their own DNS resolver and instead forwarded DNS queries to Google’s DNS servers. Perhaps it was a DNS poisoning issue in the transparent proxy rather than a proxy issue per se.

A net neutrality dream come true? Some of my sources hailed it as a victory for a free internet all thanks to this unknown attacker.

The flip side was that resolving through Google DNS threw CDN networks into disarray. CDN depends on local resolving to identify the fastest local server and overnight True became a slow extension of the USA with content being delivered the long way round.

True sent me an email assuring me that the problem had been solved and I was ready to write about all this good news and how they had grown up, but then they had a relapse.

My contacts who had set up that script had expanded it to access content from the 100 most popular websites. The script would sound an alarm if the latency changed, essentially to act as an early warning system for True turning their transparent proxy back on, and turn it back on they did.

On Tuesday, they were back at it, caching javascript left right and center apart from Google domains such as ajax.googleapis.com, google-analytics.com and ytimg.com (YouTube).

So the victory was short lived.

The original attack was simple - replace Google’s ads on just one of many servers with your own and make money. Had the ads not been annoying pop-ups perhaps nobody would have noticed. But the attacker had to get greedy and deliver more profitable, annoying pop-ups.

My network engineer friend said that what probably happened was that Google was probably angry at what True had been doing, allowing someone else (through lapse security) to syphon advertising revenue. So True was given a spanking and told to stop caching Google dynamic content, which they complied with.

The real question that nobody seems inclined to answer is whether the perpetrator has been caught and if not, does he still have access to True’s network?

All of this comes at an interesting point in time when the US is debating the loss of net neutrality. Thailand has never had net neutrality. Indeed, until a few years ago, IIG was a monopoly run by state telco CAT telecom but at least that part was deregulated. It is a scary reminder of what ISPs can do without net neutrality. Advertising revenue can easily be hijacked. Streaming can be disabled for competitors. Any OTT service can be blocked.

Indeed, that is already happening. Only if you are a giant like Google or Naver can you force the ISP to behave properly. Anyone else, any minnow, and they can stomp you out of existence without a second thought.

I wonder if anyone even is taking notice.
 

Thumbail image from server side store: 
Don Sambandaraksa
VMware's Casado talks about the goal of redefining a new architecture for network security
Market needs reliable infrastructure, pro-innovation policies, effective competition to mature
It's either that or handset subsidies, apparently
Vendors ignoring the basics of stability, good battery life, easy navigation
Even Android is being adapted for the new product segment
A rundown of the new smartphone OS challengers vying to be the third platform
Ericsson has been in Asia for more than 100 years and will continue to drive technology and services leadership in order to bring the best mobile experiences to end users

 

Telecomasia.net full website

© 2012 Questex Asia Ltd., a Questex Media Group company. All rights reserved. Reproduction in whole or in part is prohibited. Please send any technical comments or questions to our webmaster.