Troubleshooting

A higher quality score means more revenue per click, so it’s important to keep the quality of the results and traffic as high as possible. Here are some tips to help improve quality and prevent problems.

Error messages are covered in the result set section.


Using Cache-control to Prevent Ad Caching

To ensure that ad results are not cached by proxy servers sitting between the site servers and end users, please include Cache-control:no-cache in all HTTP response headers that are sent to end users. Failing to include this text will affect the relevance and quality of ads served.

Traffic Management

Automated or ‘synthetic’ traffic can cause errors in search and click traffic for both paid and non-paid engines. This is normally an issue when bots, crawlers or other software accesses the site checking the site or exploiting the search engine. Some crawlers or legitimate bots will have little to no effect, however other malicious bots or crawlers can negatively impact the partner site and cause errors in reporting and revenue recognition. This can have a negative impact on the sites monetization and the quality score for the site and its overall traffic.

To identify synthetic results, various checks have been put in place by Infospace and the content providers. The IP address and user agent of the end user are checked to ensure that the information given for who clicks the result matches that of who requested the result, for example. Consequently the user agents must match the standard formatting requirements specified by W3.org. Traffic that doesn’t meet this minimum requirement may be rejected by the search engine or counted as invalid traffic as opposed to valid clicks, paid or otherwise.

A ‘robots.txt’ file can be added to the site to restrict where legitimate crawlers and bots will search. Unfortunately, malicious bots will likely not respect the settings in the robots.txt file and may still attempt to access pages that are set as off limits to bots.

Valid/Invalid User Agents

Even when passing complete user request headers, formatting issues can cause XML errors or spikes in spam related metrics. Complete user request headers must be URL encoded and passed into the ‘x-insp-user-headers’ field of the API request header.

Web crawlers or bots may include a URL and/or e-mail address so that the Webmaster can contact the operator of the bot. Browsers will typically not pass this sort of information in their user agent headers.

Example of a Valid User-Agent value

  • Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)

Examples of Invalid User-Agent values

  • Mozilla/4.0+(compatible;MSIE+7.0;Windows+NT+5.1;+FunWebProducts;.NET+CLR+1.1.4322;.NET+CLR+2.0.50727)\r\nhttp://www.partnersite.com/results.php?keyword=remove+antivirus+pro+2009
  • User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
  • User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Spam Indicators

A variety of factors are used to identify patterns of bad traffic or spam.

  • Repeated requests from the same IP within a short period of time
  • Repeated requests from the same user agent with a short period of time
  • Repeated requests for the same keyword within a short period of time
  • An end user IP matching the server IP
  • A high frequency of keywords that are ‘deep’ URLs to specific pages
  • Poor quality score, refer to your Partnership Manager for quality score metrics

Back to: Implementing Features