Search Engines and Privacy
Marc Roessler
marc@tentacle.franken.de
March 28, 2001
Abstract
This paper analyzes the impact of using
search engines on the user's privacy.
Several privacy related issues were found,
such as search engines leaking the query
string to advertising companies or sharing
user identifiers with other sites. This
paper will show the way this is
accomplished as well as possible counter-
measures to be taken by the user to
preserve her privacy.
1. Introduction
Search engines may be one of the most widely used and most important
services the World Wide Web offers.
Concerning privacy search engines could be described as a 'data bottleneck'.
Companies usually need to analyze lots of access statistics to create a
profile for a single user. Yet when using a search engine the user herself
submits condensed and accurate information on her topics of interest. This
makes search engines an interesting target for profiling and advertising
companies.
Many users do not understand why leakage of information not containing any
personal data such as email addresses or real names is considered
a threat by privacy concerned users. However the tiny bits of information
which get dropped every now and then form a trail which can be used
to trace users, i.e. the user is not anonymous any more but pseudonymous.
Being pseudonymous means that it is not known who the user really is, but it
is known that it is her. The problem is that giving personal data such as
an email address or a name only one time will make all past and future
pseudonymous actions personal actions which can be linked to the user's real
identity.
Also in the computer security sector it is common practice to disclose as
little information as possible as every tiny bit of information might provide
valuable information to an attacker trying to break into the system. There
exist attacks exploiting bugs in web browsers yielding access to the user's
system, and knowledge of the exact browser and OS version will simplify the
task of the attacker.
After discovering that a major search engine had started to use redirects
enabling the search engine server to log which of the search hits were
actually visited a closer look at several popular engines was taken.
The follwing search engines were analyzed:
www.altavista.com
www.excite.com
www.google.com
www.lycos.com (only www.de.lycos.de tested)
www.hotbot.com (www.hotbot.lycos.de, www.hotbot.lycos.com)
www.webcrawler.com
www.yahoo.com
www.metacrawler.com
www.looksmart.com
www.directhit.com
www.goto.com
www.go.com (same as www.infoseek.com)
www.mamma.com
Banners and advertising links (href) were ignored. Only privacy related issues
concerning regular use of the search engine (submitting a search string,
viewing the results, visiting found webpages) or occuring without user
interaction were examined. An example for an event occuring without user
interaction is the automatic loading of an image.
Be advised that the fact that banners and advertising links were not analyzed
does not say anything about whether information is leaked when clicking them.
In fact it was found that the search query string is often passed to the
advertiser when clicking an advertisement.
For the code snippets presented in this paper unimportant parameters such as
border, height, width etc were omitted. The search hits used for illustration
were randomly selected without paying attention to the company or site they
pointed to.
Several techniques that have an impact on the user's privacy were found to be
in use. There also were some strange side effects which may or may not pose a
threat to the user's privacy. They are mentioned nevertheless so the reader
can judge for herself.
One of those 'effects' is caused by the use of images instead of buttons for
submitting html-forms. If the image is clicked the coordinates of the
click will be transmitted along with the data of the html form.
If the button is not pressed but the form is completed by pressing Enter
in the query string input field the x and y parameters will usually not appear
in the form data submitted. Several browsers were tested, the following table
shows what parameters are sent along with the form data for each browser when
either clicking or submitting by pressing Enter:
Enter Click
Netscape Nav. 4.76 no x/y params x/y coordinates
MSIE 5.01 no x/y params x/y coordinates
Opera 5.02 no x/y params x/y coordinates
Lynx 2.8.3rel.1 x/y: zero -
If lynx is used and the form is sent by pressing the submit
image replacement the string '&x=0&y=0' will be appended to the form
data. This string is very unlikely to occur by clicking the image using
Netscape, MSIE or Opera.
This means that a probabilistic approach for determining whether a GUI based
browser or Lynx is being used is possible. This works even if the HTTP
headers are rewritten using a simple filtering proxy.
Another problem is the fact that the URL of the search results page usually
contains the query string. Contrary to what one might expect parameters are
included with the URL in the referer field of the HTTP header (Netscape Nav.
4.76, MSIE 5.0 tested), which means that the query string will be leaked to
every site visited. This means that all advertisers displaying banners on the
search results page and all visited sites will know the search string if
referers are not filtered by an intermediate proxy.
Another strange piece of code related to ad.preferences.com looks as if
it was used to measure the time needed for loading the search result pages.
At the beginning of the page an image tag such as
is placed. At the end of the page there is some request to a program or
script named 'ping' at the preferences.com site:
In combination with user identifiers (such as cookies) in theory this can
be used to measure the speed of the user's internet connection.
Another thing which was noticed were the extremely long expiry times of
some of the cookies. For instance 31-Dec-2030 23:59:59 seems to be a bit
far fetched.
Disclaimer:
No guarantee is given concerning correctness or completeness of the information.
Giving correct and up to date information is almost impossible due to the
fact that the HTML templates of search engines are highly subject to changes.
When in doubt browse the html source yourself.
This paper proposes no claims whether the mentioned companies do use the
gathered information, it only shows that they have access to the
mentioned data which means that they are able to store and use it.
Some of the mentioned data can be used to improve the service provided to
the customer, e.g. by sorting search results by the number of users which
followed them, thus making the search more effective.
2. A look at some search engines
2.1 www.altavista.com
Uses cookies: Yes.
Cookie Example: AV_USERKEY=AVS037fdee1e2ac000a20203c0051635
Cookie Expiry: expires=Tuesday, 31-Dec-2013 12:00:00 GMT
The query string ('privacy on the net') is leaked to ad.doubleclick.net,
possibly including IP address if no proxy is used.
This happens without any user interaction by including a layer located at
ad.doubleclick.net, passing a few parameters inluding the query string.
Chris Brenton [1] quotes an unnamed person on the 'ord' parameter seen above.
According to this person the ord value includes the value of the cookie
issued by Altavista, which means that Altavista and doubleclick are sharing
user identifiers. He or she showed that the value of the ord parameter
converted to hex (e.g. by using perl -e 'printf "%x\n", ;') is part
of the Altavista cookie. This was in December 1999.
This has been checked again and it turns out Altavista obviously has modified
the 'syntax' of its cookies. They are much longer now and the ord value
is at least not trivially contained in the cookie. Of course this proves
nothing, the ord value may still be part of the cookie in some way.
Altavista does not link search hit URLs directly but via a redirector
located on the Altavista server. This (in theory) enables Altavista to tell
which search hits actually were visited. In conjunction with cookies and the
query strings this is nice ground for a database.
Privacy on the Net: Practical Issues
Note that this redirect is hidden from the user if JavaScript is enabled:
to the user the URL will be shown as 'http://www.tamos.com/privacy/' at the
bottom of the browser window.
The present server hostname (www.altavista.com) is inserted by the browser
automatically. In effect '/r?r=' will become 'http://www.altavista.com/r?r='.
The "Sponsored Listings" are linked in a strange way. The user gets passed
back to Altavista, which passes her to www.goto.com, which in turn resolves
the xargs string and redirects the user to the URL she intended to visit.
Example:
Siegesoft Protects Your Net Privacy
It is unclear if the xargs string is only a key used for looking up the URL
in a database or if it contains the URL itself. It may as well contain
cookie data or even the search string.
An image is used as the submit button. This could be used for the mentioned
probabilistic browser statistics.
2.2 www.excite.com
Uses cookies: Yes.
Cookie Example: UID=E37F3193E55CE9D0
Cookie Expiry: Expires=Fri, 21-May-2004 02:25:38 GMT
Cookie Example: registered=no
Cookie Expiry: Expires=Fri, 21-May-2004 02:25:38 GMT
Search hits are linked using redirects.
Privacy Site
Just as with Altavista.com, the real URL is hidden from the user.
The id and pos parameters seem not to be related to tracking users.
The id is different for each search hit, it is probably some database key.
The pos field contains the position of the link on the page (first, second,
third, .. ).
The search form is quite strange. The query is submitted using a redirect:
There are no direct links to the search hits. Instead the obfuscated mechanism
mentioned before is used for redirection. Note that the mentioned 'pin'
(sessionid) is transmitted as well:
Siegesoft Protects Your Net Privacy
2.13 www.infoseek.com, www.go.com
These are now one site (infoseek is now go.com).
A few weeks ago they still used Cookies and lots of horribly obscured
JavaScript code which transmitted referer, browser, OS, screen
resolution/color depth, plug-ins, language settings, cookie preferences,
search engine keywords, JS enablement, number of visits, paths taken and time
spent on sites and pages (quoted from http://websidestory.com/privacy)
to stats.hitbox.com, another domain of websidestory.com.
Now this site is powered by www.goto.com.
Cookies are from www.goto.com.
Uses cookies: Yes.
Cookie Example: sessionid=NPCHJNIAAJ4HFQFIEOSAPUQ
Cookie Expiry: -
Cookie Example: UserID=ABCF5D2AF79E36E9
Cookie Expiry: expires=Fri, 18-Mar-2011 17:02:02 GMT
The user gets 'pinned' with a session ID which is not only present as a cookie
but passed around between the different pages of a site.
Further querys will carry the same session id: