Tag Archives: transcoder

Verizon Using Novarra's Transcoder

Posted on November 13, 2008 by Dennis Bournique

Verizon Logo The largest US mobile carrier, Verizon Wireless has started funneling traffic between Verizon feature phones and the web through a transcoder from Novarra. Verizon calls the service "Optimized View" and has added promotional information and a FAQ to their customer website There is also a page on the carrier's developer site which has links to a opt-out form and to a PDF document detailing the rules that the transcoder uses to determine which sites to transcode.

Given the disastrous effects that VodaFone's UK roll-out of Novarra had on mobile sites and services, I'm rather apprehensive about this. In the PDF, Verizon/Novarra say that they won't change the User Agent header or transcode sites that have submitted an opt-out request or have a URL corresponding to one of these patterns:

*.mobi, m.*, mobile.*, avantango.*, wap.*, iphone.*, <domain>/m/*, <domain>/mobile/*, pda.*, wireless.*, wml.*, xhtml.*, <domain>/m/, <domain>/gmm/, <domain>/portable

For sites that have not opted out or do not use one of the mobile URL patterns, the transcoder will change the User Agent and a number of other headers to ones mimicking a desktop browser. However the site will not be transcoded if it meets any of the following conditions:

It uses one of the following mobile specific DTDs:
- <!DOCTYPE html PUBLIC "-//OMA//DTD XHTML Mobile 1.2//EN" "http://www.openmobilealliance.org/tech/DTD/xhtml-mobile12.dtd">
- <!DOCTYPE html PUBLIC "-//WAPFORUM//DTD XHTML Mobile 1.1//EN"
  "http://www.wapforum.org/DTD/xhtml-mobile11.dtd">
- <!DOCTYPE html PUBLIC "-//WAPFORUM//DTD XHTML Mobile 1.0//EN"
  "http://www.wapforum.org/DTD/xhtml-mobile10.dtd">
- <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML Basic 1.1//EN" "http://www.w3.org/TR/xhtml-basic/xhtml-basic11.dtd">
- <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML Basic 1.0//EN" "http://www.w3.org/TR/xhtml-basic/xhtml-basic10.dtd">
It uses one the following MIME types:
- Application/vnd.wap.xhtml+xml
- Text/vnd.wap.wml
It has a self referencing link rel tag:
- <link rel=”alternate” media = “handheld” href=mymobilesite.com/>
It sends a "Cache-Control: no-transform" header.

I don't have a Verizon account but I did do some limited testing in a Verizon store. I used a Motorola V950 which runs the Openwave 6 browser. I mostly tested my own sites. The transcoder seems to follow the rules in the PDF. None of my sites that use a mobile URL pattern, Doctype or no-transform were transcoded. A couple of test pages I created as copies of mobile pages but with none of the Novarra trigger conditions where transcoded even though they were under 10KB.

I didn't really test the quality of the transcoded pages but noticed that a two line navigation bar is added to the top and bottom of the page and pagination is used. I thought the transcoded version of howardforums.com was reasonably usable, but my test sites, which were under 10KB, were needlessly split into two pages.

According to Verizon, the transcoder does offer users the ability to view the original version of any transcoded page by clicking a "Turn off Optimization" link in the bottom navigation footer. As a user, I consider this a very desirable feature which all transcoders should implement.

Secure HTTPS sites are transcoded, except for "banking sites". I wonder how Novarra identifies banks? Users are warned that their security may be compromised when visiting a non-banking secure site through the transcoder. I didn't try any secure sites so I don't know what this warning looks like.

Transcoders are the bane of mobile web developers. This one will probably not be as disruptive has the Vodafone UK one was because Novarra has learned some lessons from that debacle and is doing a better job of detecting mobile sites and also because the development community is more aware of the transcoding problem and how to work around it. There should be no impact on m. and .mobi sites and others with a recognized mobile URL pattern. Sites that use a mobile doctype and don't do browser detection will also be unaffected. However, sites on .com domains that rely on the User Agent or other headers to optimize for specific handsets need to either get on the white-list or start checking the "X-Device" headers where the transcoder is putting the original values of the headers it's modified as follows:

Original Header	Renamed Header
User-Agent	X-Device-User-Agent
Accept	X-Device-Accept
Accept-Charset	X-Device-Accept-Charset
Accept-Encoding	X-Device-Accept-Encoding
Accept-Language	X-Device-Accept-Language

The Verizon transcoder can be identified by the Via: header which it sends, "1.1 Novarra (Vision/7.3)
X-Mobile-Gateway: Novarra-Vision/7.3 (VZW; Server-Only)"

The Vodafone transcoder broke many content (ring tones, game, application) download sites. I don't see that happening with this one because Verizon already blocks almost all off-portal downloads.

The problem with transcoders changing headers is that it forces millions of mobile sites around the world to change their code in order to keep the same functionality. It's an annoyance for capable mobile web developers who somehow find out that another transcoder has been deployed somewhere in the world. The real problem is with sites that aren't actively maintained or whose developers don't actively follow the doings of the W3C or monitor the mobile development mailing lists, forums and blogs. Which, I suspect, is the majority of sites worldwide.

Verizon and Novarra claim that their transcoder partially follows the recommendations of the W3C Content Transformation Guidelines. However those guidelines are still a work in progress and the issue of whether transcoders should alter the User-Agent except in a very limited number of cases such as when a site returns no content when presented with a mobile browser User-Agent is still under discussion. It would be far better for the health of the mobile web if transcoders did not alter the User-Agent and other headers as recommended by the Rules for Responsible Reformatting: A Developer Manifesto a document that evolved from discussions on the wmlprograming Yahoo Group and which has been signed and adopted by several transcoder vendors, but not by Novarra.

Posted in Mobile Web Development | Tagged Novarra, transcoder, Verizon mobile content transformation | 94 Comments

Microsoft to Use InfoGin's Transcoder

Posted on September 9, 2008 by Dennis Bournique

InfoGin Logo InfoGin announced today that they have signed an agreement to provide data transformation services for Microsoft's Live Mobile Search. Microsoft had been using an in-house transcoding engine for Live Search.

InfoGin has one of the best of the current crop of transcoders. It partially converts Adobe Flash content by rendering Flash animations as animated gifs and allowing users to follow URL links within Flash.

What I like best about InfoGin though is that they do not alter the HTTP User Agent request header like most of the other transcoder vendors. Mobile content sites use this header to identify specific handsets and customize content for them. Modifing the User Agent can have disasterous effects on the usability and revenue of effected mobile services.

InfoGin was the first transcoder vendor to sign Luca Passani's Rules for Responsible Reformatting: A Developer Manifesto which provides a set of guidelines to help transcoders coexist with the existing mobile ecosystem.

I spoke with InfoGin's Eyal Rosen at yesterday's Mobile Web MegaTrends and he told me that signing the Manifesto had helped the company close deals with carriers. Congradulations to InfoGin, the Microsoft deal is huge and validates InfoGin's approach of working with rather than against the mobile web development community.

Posted in Content Adaptation | Tagged Infogin, Microsoft, mobile content transformation, transcoder | Leave a comment

Opting Out of Transcoding

Posted on August 27, 2008 by Dennis Bournique

: Logo courtesy of the W3C

Remember the furor that erupted on the web and the wmlprogramming Yahoo group over transcoders? Specifically, the ones that Vodafone UK and other carriers implemented with the goal of making non-mobile web pages more usable on mobile handsets? The issue was that these services had the (hopefully unintended) consequence of degrading or completely breaking numerous mobile web sites and services. Ring tone and game downloads no longer worked, some mobile web sites didn't load or displayed malformed content. Sites that relied on being able to detect mobile browsers and deliver optimized content no longer worked as designed because the transcoders removed or changed the HTTP headers that identified the browser, user's language settings or preferred content types.

Developers and content providers upset with the disruptions to their sites and business models rallied around Luca Passani, co-creator of the WURFL mobile device repository to formulate a document, "Rules for Responsible Reformatting: A Developer Manifesto" which suggested guidelines that transcoders should follow to avoid disrupting existing mobile sites and services. The "Manifesto" was generally well received in the mobile web development community and was also endorsed by three of the major transcoder vendors InfoGin, Volantis and Openwave. Openwave provides the OpenWeb transcoder used by Sprint.

The furor had died down, but now it's been rekindled by the W3C's release of a draft of its "Content Transformation Guidelines" recommendations, which like the Manifesto, describes best practices for mobile transcoders. The Guidelines are open for public comment until Sept 16. Anyone can comment by sending an e-mail to [email protected] (note that your e-mail address will be published on the web - I recommend that you use a disposable e-mail address!). All comments can be viewed at lists.w3.org/Archives/Public/public-bpwg-comments

Their are some clear differences between the two documents. The W3C Guidelines doesn't go as far as the Manifesto in limiting what is acceptable behavior by transcoders, particularly with regard to changing HTTP headers. The Guidelines also place a larger part of the responsibility for interoperability on mobile web developers and publishers rather than transcoder vendors. All of which understandable given that the members of the W3C, including the Mobile Web Best Practices Working Group, are primarily representatives of commercial organizations including mobile carriers, transcoder vendors and large web publishers while the Manifesto comes out of the mobile web developer community.

There are lively discussions about the W3C Guidelines going on both through the comment process and on the wmlprogramming Yahoo group. Luca and others have proposed changes to the Guidelines that would bring it closer to the Manifesto. It remains to be seen if these suggestions will make it into the final document.

In reading both the Guidelines and the Manifesto, I noticed that neither said anything about allowing end-users to opt out of transcoding. I think it is essential that all transcoders offer ordinary users a way to disable the transcoder, either temporarily for the current site or page, or globally for all sites.

In spite of everyone's best intentions, transcoders will sometimes break sites that would otherwise be usable on a given handset. If you have ever used Skweezer, Mowser or the transcoders that Google, Yahoo, AOL and Microsoft Live use to reformat web content returned from search queries, I'm sure you've noticed that some sites come up blank, broken into one line pages or missing essential parts and functionality when transcoded. Fortunately those services are opt-in, no one is forced to use them, and except for Yahoo and AOL, they offer links to the original version. With the carrier transcoders there is often no way to opt out, leaving users no alternative if the transcoded version of one of their favorite sites is broken.

Secure sites pose a special problem for transcoders as they can not be transcoded without decrypting the the secure content, potentially exposing users to "man in the middle" attacks. Users need to be able to opt out of transcoding when doing online banking or shopping when account information or credit car numbers are being passed.

I've left a comment with the W3C arguing for making user opt-out a requirement for all transcoding proxies, I won't bore you by reprinting it here as it essentially restates the above. You can read it in the W3G BPWG archives.

The issue of transcoding, is a very important one for the future of the web on mobile devices. Take a few minutes to skim the Guidelines and the discussions at the W3C and wmlprogramming and if sufficiently motivated, email comments on the W3C Guidelines to [email protected] by the Sept. 16th deadline.