In Search Of Readers: A Brave New World For Researchers

Anne Crassweller, Judy Rogers and Don Williams, NADBank

Worldwide Readership Research Symposium Valencia 2009 Session 6.1

Setting the Stage

Online panels are becoming the standard choice for researchers today. The temptation to use them is overwhelming as they are a less expensive alternative to traditional telephone and face-to-face research and the results are available sooner. Are they the right choice for all research projects?

There is no question that considerable effort is being devoted to building better panels. Commonly, however, the “virtues” of panels are demonstrated not by their intrinsic measurement superiority but by their speed and low cost vis à vis other sampling and data collection modes and/or by deficiencies in other types of surveys (e.g., falling response rates in telephone studies because of consumer avoidance techniques and new technologies masking geography etc as well as the growth in cell-only households).

Conversely, online panel research limitations are often overlooked (convenience samples, coverage bias etc.) in favour of cost and timing attributes. As broadband Internet access increases, coverage error and potential bias for online surveys will likely decrease but not eliminated.

Market research practitioners and users have long recognized that focus groups or shopping mall intercepts are not suitable tools for answering some types of questions. The same applies to online panels. They have their place in the toolbox but purveyors and users are engaged in an ongoing discussion about which types of questions they are best able to answer. Can they replace studies based on a random sample of respondents within a defined universe in which the probability of selection is known?

There remains considerable support for the position that when quantitative estimates such as market shares are required, they should be derived from studies that rely on random probability sampling rather than online panels (Chakrapani,2007) Nonetheless, like other services, newspapers are under increasing pressure to be “with it” (a.k.a. be on the web), to reduce data collection costs (a.k.a. be on the web), and to increase reporting ease and speed (a.k.a. be on the web).

Since 1986 newspapers in Canada have relied on traditional telephone data capture techniques to estimate readership of daily papers in major markets, to build profiles of readers and to set market parameters and pricing for advertising. Millions of dollars of advertising revenue depend on readership and profile estimates of how many and who reads specific daily newspapers. How are print media and advertising mavens going to determine when or if to move to online panels?

As Canada’s newspaper audience measurement agency, the Newspaper Audience Databank Inc. (NADbank) has been monitoring developments in online panel research for some time. In 2006 the organization embarked on a journey to determine if its annual study could be moved from a modified RDD telephone methodology to a web-based survey. The results of this parallel online survey using the TNS Canadian Facts’ web-access panel in Toronto were reported to this symposium in 2007.

The results from the first study showed that the demographic profiles of respondents in the online panel differed from the population as a whole and the telephone sample. As well as demographic differences, there were variations in general media behaviour as well as the primary metric being investigated: readership of daily newspapers.

As only one online panel was included in the test, there was no way to assess the extent to which profile differences were a function of idiosyncrasies of the single panel and/or were linked to online panel data capture per se.

In the fall of 2007 a second, larger scale study was undertaken to further explore the differences between telephone and online protocols; inter-supplier consistency and the potential use of a web-based survey outside of the Toronto market.

Study Objectives and Design

The overall purposes of the research were as follows:

  1. To determine the extent to which online results resemble or differ from the current modified RDD telephone survey methodology used to estimate newspaper audiences in Canada’s urban centres;
  2. To examine the extent to which there is inter-supplier consistency using independent online panels; and
  3. To explore the feasibility of undertaking online panel surveys in small and medium sized Canadian markets.

NADbank partnered with TNS Canadian Facts, Ipsos Canada, Synovate Canada, and Decima-Harris for this second study. NADbank collected five data sets that purport to measure the same phenomenon: readership of daily newspapers in three Canadian cities (Toronto, Québec City and Halifax).

From more than 2000 completed surveys per supplier among Toronto residents to between 400 and 630 in the smaller centres, each of the four firms implemented online panel studies that mimicked, to the extent possible, the ongoing modified RDD telephone readership study.

The four research companies were provided with identical instructions for project design, implementation, weighting and projection. To the extent possible, the survey instrument, survey scheduling procedures, weighting and projection techniques matched those used in the current NADbank telephone survey methodology. To ensure study-to-study consistency, coding of responses to the occupation measures in the four online surveys was performed by TNS Canadian Facts using the same principles and categories the firm uses to code occupation in NADbank’s traditional telephone survey. More details about the methodology are also available from NADbank (under separate cover).

The three cities were selected to represent different market conditions:

  • Toronto: a large metropolitan area with multiple English language daily newspapers;
  • Halifax: a small metropolitan area with a limited number of English language daily newspapers;
  • Québec City: a medium sized metropolitan area with two French language daily newspapers.

Survey materials for Toronto and Halifax were in English; those for Québec City were in French. Targets for completed surveys differed substantively across the three cities (Toronto, 2000; Halifax, 400; Québec City, 500). A common data file incorporating all suppliers’ data in a consistent format was created. This file was used to generate tabulations with identical specifications from supplier to supplier, thereby ensuring that outputs among the online studies and to the telephone survey were directly comparable.

The identity of the four online study suppliers has been masked. In no particular order, the firms were labelled with the first four letters of the alphabet (Panel A, B, C, D). The NADbank telephone readership study is the benchmark against which each online supplier’s data are assessed.

Key Findings

Single Panel Consistency – 2006 vs. 2007 in Toronto

Stability of year-to-year results is an important consideration for currency measurement and a key issue when evaluating the value of a new protocol. Before looking at the multi-panel data the consistency between the two studies conducted with the same supplier panel; TNS Canadian Facts, was reviewed.

NADbank data are weighted to match age-within-gender estimates of the population 18 years of age and over (by market) based on projected census data.

Table 1 shows comparable distribution for the selected demographics that are not captured in the weighting protocol for both the telephone samples and online panels; all conducted by TNS Canadian facts. The findings presented in Table 1 are not restricted to the selected variables but indicative of other “not-weighted” demographics from the 2006 and 2007 studies in Toronto. With some notable exceptions, the patterns hold constant within the data collection modes from year to year but differ between the telephone and online methodologies.

TABLE 1

COMPARISON OF SELECTED SAMPLE AND PANEL DEMOGRAPHIC COMPOSITION 2006 AND 2007

weighted

TELEPHONE ONLINE PANEL

2006 2007 2006 2007

Some Post Secondary Education 31% 32% 46% 44%
University Graduate + 38% 37% 33% 33%
Manager/Professional 20% 22% 18% 18%
Blue Collar 12% 13% 9% 9%
Adult Only Households 60% 60% 66% 45%
English as a Mother Tongue 61% 61% 77% 78%

General media habits provide an overview of the behaviour of the respondents in each group and harbinger of the newspaper readership results. Table 2 shows time spent in hours per week with each medium and the reach of any daily newspaper yesterday, on Saturday and readership online (weekly). As noted with the demographics, there is good measure of consistency for the top-line measure of time spent by medium (as well as reach by medium – not shown here); a seductive outcome early in the evaluation process.

TABLE 2.

COMPARISON OF MEDIA HABITS IN TORONTO 2006 AND 2007 – weighted

TELEPHONE ONLINE PANEL

2006 2007 2006 2007

HOURS/WEEK        
Daily Newspapers 4.2 4.5 5.8 5.7
Radio 13.0 12.4 13.8 13.7
Magazines 3.7 3.1 4.9 4.2
TV 13.9 13.0 21.4 20.6
Internet 13.9 13.4 22.3 22.1
REACH        
Any Paper Yesterday 47% 46% 56% 48%
Any Paper Saturday 43% 40% 52% 44%
Any Paper Online Past Week 19% 21% 29% 28%

Stability begins to breakdown with the Any Newspaper readership results such as any paper yesterday (Table 2). Table 3 illustrates the further degradation of consistency for specific newspaper readership. It should be noted, though, that the general decrease in Saturday readership and increase in weekly online readership between 2006 and 2007 on the telephone (Table 2), is consistent with results in all other markets measured in the telephone in the study.

TABLE 3

READ YESTERDAY IN TORONTO – TELEPHONE VS. ONLINE PANEL 2006 AND 2007

TELEPHONE ONLINE PANEL

2006 2007 2006 2007

READ YESTERDAY        
Toronto Star 23% 22% 31% 24%
The Toronto Sun 12% 10% 18% 14%
The Globe and Mail 9% 9% 11% 8%
National Post 5% 5% 7% 7%
Metro 9% 11% 14% 12%
24 hours 8% 8% 15% 12%

Multi-Panel Results

  1. Demographics

As mentioned, once the data had been weighted, each online panel supplier matches “exactly” the telephone benchmark for age and gender distributions in each city. Nonetheless, there are considerable fluctuations in the actual (unweighted) distributions by age and gender from online panel to panel. While it can be argued that “weighting” remedies differences between panel respondents and the actual market, further exploration suggests that this may not be the case. For Toronto, the unweighted results for age and gender are shown for the telephone sample and each of the panels. [Table 4].

TABLE 4

COMPARISON OF GENDER AND AGE IN TORONTO

WEIGHTED ACTUAL – unweighted

TELEPHONE TELEPHONE PANEL A PANEL B PANEL C PANEL D

Completed Surveys   2063 2001 2214 2112 2029
GENDER            
Male 49% 43% 52% 36% 45% 50%
Female 51% 57% 48% 64% 55% 50%
AGE            
18-24 12% 9% 13% 6% 5% 9%
25-34 20% 17% 17% 15% 17% 20%
35-49 32% 37% 28% 37% 31% 33%
50-64 22% 23% 31% 28% 26% 23%
65+ 14% 13% 11% 14% 22% 15%
Avg. 18+ 45 46 45 47 49 45

Once weights have been applied to age and gender variations in the demographics not captured in the weighting protocol between the benchmark telephone and various online results are evident. However, there are no obvious patterns in the size or direction of the discrepancies [Table 5]. For example:

  • In Toronto, Panels A (18%) and B (20%) are close to the telephone benchmark for the incidence of persons in the managers/professionals occupation group (21%). Panel C is well below (15%) and Panel D is well above (32%) the benchmark.
  • Since Panel C has the lowest estimate of managers/professionals in Toronto, it might also be expected to have a low estimate of university graduates. It does not. In fact, Panel C’s university graduate proportion (42%) is higher than Panels A (33%) and B (34%) and closest of the four online panels to the telephone benchmark (39%).
  • Toronto is a major reception centre for new immigrants, explaining why only 6-in-10 adults name English as their mother tongue (telephone). At about 3-in-4, panel respondents are appreciably more likely to have first learned and still understand English than would be anticipated.
  • House dwellers represent 84% of the Halifax market according to the telephone survey, but only 75% of the market according to Panel D, 70% according to Panel B, and as few as 62% according to Panel A.
  • In Québec City, all panels produce an identical estimate of persons with “some post secondary education” (ranging from 46% to 49%). In each case, these estimates are substantively higher than the benchmark telephone study (36%).

TABLE 5

DEMOGRAPHIC SAMPLER – TORONTO, HALIFAX AND QUÉBEC CITY

TELEPHONE PANEL A PANEL B PANEL C PANEL D
TORONTO – WEIGHTED, PROJECTED        
EDUCATION        
Some Post Secondary 32% 44% 45% 40% 33%
University Graduate 39% 33% 34% 42% 56%
OCCUPATION        
Managers/Professionals 21% 18% 20% 15% 32%
Other White Collar 17% 17% 20% 18% 21%
Cler/Admin/Bus. Support 6% 9% 9% 9% 5%
Blue Collar 13% 9% 9% 5% 4%
DWELLING        
House 78% 67% 73% 70% 75%
Apartment 21% 31% 26% 29% 24%
HOUSEHOLD COMPOSITION        
Adults Only 60% 66% 66% 67% 69%
Adults w. Child 40% 34% 34% 32% 31%
MOTHER TONGUE ENGLISH        
61% 78% 77% 77% 74%
HALIFAX – WEIGHTED, PROJECTED        
EDUCATION        
Some Post Secondary 27% 47% 47% 46% 41%
University Graduate 36% 29% 29% 34% 50%
OCCUPATION        
Managers/Professionals 23% 16% 18% 12% 30%
Other White Collar 13% 16% 22% 15% 22%
Cler/Admin/Bus. Support 9% 14% 9% 10% 7%
Blue Collar 16% 10% 10% 9% 5%
DWELLING        
House 84% 62% 70% 65% 75%
Apartment 14% 34% 28% 32% 23%
HOUSEHOLD COMPOSITION        
Adults Only 62% 73% 73% 74% 75%
Adults w. Child 38% 27% 27% 26% 25%
QUÉBEC CITY – WEIGHTED, PROJECTED        
EDUCATION        
Some Post Secondary 36% 48% 49% 48% 46%
University Graduate 25% 21% 25% 29% 37%
OCCUPATION        
Managers/Professionals 12% 8% 10% 7% 13%
Other White Collar 18% 18% 18% 19% 18%
Cler/Admin/Bus. Support 7% 14% 14% 10% 11%
Blue Collar 16% 13% 14% 8% 8%
DWELLING        
House 69% 53% 56% 54% 61%
Apartment 30% 45% 41% 43% 37%
HOUSEHOLD COMPOSITION        
Adults Only 71% 77% 78% 75% 76%
Adults w. Child 29% 23% 22% 24% 24%
  1. General Media Habits

General media habits of Toronto’s adults (for example) are also highly variable across the online panels. Ranging from a high of about 82 minutes to about 57 minutes, the average number of minutes spent reading a printed version of any daily newspaper yesterday is substantively higher for online panel members than it is for telephone sample respondents, at about 50 minutes, on average [Table 6]. Television viewing time is also consistently higher for online panel respondents than is the case for telephone sample members. Radio listening time, however, hovers around the same level; irrespective of the data capture mode or the online supplier. Not surprisingly, average time spent on the internet yesterday among panel respondents is also higher than is the case among telephone sample respondents.

TABLE 6

GENERAL MEDIA BEHAVIOUR IN TORONTO

TELEPHONE PANEL A PANEL B PANEL C PANEL D
AVERAGE NUMBER MINUTES SPENT YESTERDAY….      
Reading Printed Newspapers 50 69 63 82 57
Reading Magazines 44 49 45 57 48
On the Internet 140 206 163 196 169
Listening to Radio 138 144 139 147 118
Watching TV 125 185 168 188 144
AVERAGE NUMBER OF PAST 5 WEEKDAY NEWSPAPER ISSUES READ      
2.8 2.7 2.7 2.5 2.5
  1. Newspaper Readership Read Any Paper Yesterday

As the demographics and general media behaviour vary substantively from panel to panel and between panels and the telephone benchmark, so do readership estimates (see Appendix for complete summary tables).

As one of the most salient media measures for daily newspapers, rea d yesterday estimates across the four panels and the three markets are worthy of scrutiny. Table 7 illustrates that there are no obvious patterns in direction or degree of variation.

Toronto

  • For example, compared to the telephone benchmark (22%), three online panels generate higher readership levels for the Toronto Star (B, 30%; C, 31%; D, 35%). At 24%, Panel A is close to the benchmark.
  • All panels generate estimates consistent with the telephone benchmark (11%) for Metro Toronto.
  • At 21%, Panel D’s read yesterday estimate for The Globe and Mail far surpasses the telephone benchmark (9%) and estimates from all other online panel suppliers.

Halifax

  • Did close to two-thirds of adult Halagonians read at least one daily newspaper yesterday (Panels B or D) or was it closer to one half of them (telephone benchmark; Panel A)?

Québec City

  • Panel D appears to generate read yesterday estimates for Toronto and Halifax dailies that are substantively higher than the telephone benchmark and, in some cases, than estimates by other online suppliers. Why, then, do we find that Panel D’s estimate for Le Journal de Québec (23%) is lower than every other online supplier’s and than the telephone benchmark (29%)?

Table 7

Read Yesterday Readership in Toronto, Halifax and Québec City

TELEPHONE PANEL A PANEL B PANEL C PANEL D
TORONTO          
Toronto Star 22% 24% 30% 31% 35%
The Toronto Sun 10% 14% 15% 12% 9%
The Globe and Mail 9% 8% 9% 12% 21%
National Post 5% 7% 7% 7% 12%
Metro 11% 12% 12% 11% 12%
24 hours 8% 12% 10% 9% 8%
Any 46% 48% 53% 53% 62%
HALIFAX          
The ChronicleHerald 40% 43% 54% 52% 54%
The Daily News 17% 19% 25% 20% 27%
The Globe and Mail 4% 6% 5% 6% 12%
National Post 1% 1% 1% 2% 1%
Any 48% 51% 63% 58% 65%
QUÉBEC CITY          
Le Journal de Québec 29% 31% 30% 26% 23%
Le Soleil 23% 23% 25% 25% 32%
Any 48% 47% 48% 45% 48%

Online Past Week Readership

It would be reasonable to assume that members of online panels experience fewer constraints in accessing newspapers online than does the public at large. This research confirms this assumption. Indeed, panel members exhibit appreciably higher levels of past week online readership than would be expected based on the telephone benchmark findings.

The “gaps” between online readership estimates from the telephone survey and the online panels are indicative of a fundamental difference between the data collection modes [Table 8]. In fact, these differences are sufficiently substantive across all online suppliers and the three test markets that they raise concerns about the ability of online panels to “match” the behaviour of all adult residents in Canada’s cities.

If all newspapers in all markets experienced the same level of disparity for online readership, one might conclude that the telephone survey is under-representing the online market for newspapers. Since the differences between the telephone estimates and each of the four online panels vary by paper/market, such a conclusion does not seem warranted:

    • estimates for past week The Globe and Mail online readership are similar for the telephone, Panels A, B, and C but very different for Panel D;
    • all online estimates are appreciably higher than the telephone estimate for online readership of both dailies in Halifax and for Le Soleil in Québec City.

TABLE 8

WEEKLY READERSHIP OF ONLINE EDITIONS

TELEPHONE PANEL A PANEL B PANEL C PANEL D
TORONTO          
Toronto Star 13% 19% 20% 22% 29%
The Toronto Sun 4% 9% 9% 10% 9%
The Globe and Mail 9% 9% 9% 12% 22%
National Post 3% 3% 4% 4% 8%
Metro 1% 4% 3% 4% 3%
24 hours 2% 4% 3% 4% 4%
Any 21% 28% 29% 34% 44%
HALIFAX          
The ChronicleHerald 14% 33% 37% 32% 37%
The Daily News 8% 15% 17% 14% 17%
The Globe and Mail 4% 6% 7% 8% 19%
National Post 1% 2% 3% 3% 5%
Any 17% 38% 42% 37% 47%
QUÉBEC CITY          
Le Soleil 7% 18% 14% 16% 21%

Different Panels, Same Readers?

For the following analyses, the focus is on Toronto as the base sizes for this city are appreciably larger than those in Halifax and Quebec City.

Even if the incidence of reading various newspapers in their print or online versions differed somewhat from panel to panel and between the panels and the telephone benchmark, it would be reasonable to anticipate that the profiles of readers of specific publications would be similar from study to study. In other words, a person who claims to have read yesterday’s Toronto Star or The Globe and Mail should have a similar demographic and behavioural profile, irrespective of the source of readership information. Is this the case? No, it is not.

Based on the examples displayed in Table 9, none of the online panels produces profiles of newspaper readers that are consistent with those generated by the telephone survey. Furthermore, the online panel profiles differ substantively, one to another.

    • What proportion of Toronto Star readers are university graduates? Is it about two-fifths as estimated in the telephone study (43%), about half as estimated by Panel C (49%) or almost three-fifths as estimated by Panel D (57%)?
    • What proportion of The Toronto Sun readers are over 65 years of age? Is it seven percent as estimated in the telephone survey or twice this proportion as estimated by Panels A (15%), B (14%) or C (16%)?

Similar variations are evident for other characteristics of Toronto’s newspaper readers (e.g., tenure, managerial/professional job status, etc.).

Readers of yesterday’s The ChronicleHerald in Halifax and the Le Journal de Québec also manifest quite different demographic characteristics, depending on whether they were interviewed in the telephone study or completed the survey online by one of the four suppliers.

TABLE 9

DEMOGRAPHIC SAMPLER FOR SELECTED NEWSPAPERS

TELEPHONE PANEL A PANEL B PANEL C PANEL D
TORONTO          
READ YESTERDAY’S ISSUE n=440 n=505 n=676 n=681 n=728
OF TORONTO STAR          
Age 65+ 28% 25% 25% 23% 20%
University Graduate + 43% 37% 42% 49% 57%
Home Language English 87% 92% 93% 94% 91%
Manager/Professional 20% 20% 19% 14% 31%
Own Home 79% 73% 81% 77% 84%
READ YESTERDAY’S ISSUE n=196 n=292 n=329 n=244 n=171
OF THE TORONTO SUN          
Age 65+ 7% 15% 14% 16% 10%
University Graduate + 19% 22% 27% 29% 44%
Home Language English 89% 94% 87% 93% 94%
Manager/Professional 22% 20% 20% 14% 31%
Own Home 72% 61% 69% 63% 83%
READ YESTERDAY’S ISSUE n=185 n=159 n=183 n=265 n=454
OF THE GLOBE AND MAIL          
Age 65+ 17% 27% 24% 24% 21%
University Graduate + 66% 55% 64% 67% 75%
Home Language English 92% 91% 87% 91% 93%
Manager/Professional 39% 27% 33% 25% 42%
Own Home 85% 73% 76% 76% 86%
HALIFAX          
READ YESTERDAY’S ISSUE n=110 n=185 n=276 n=261 n=268
OF THE CHRONICLEHERALD          
Age 65+ 24% 24% 20% 24% 20%
University Graduate + 41% 32% 32% 31% 51%
Home Language English 99% 99% 95% 98% 98%
Manager/Professional 25% 22% 17% 13% 30%
Own Home 80% 69% 68% 69% 83%
QUÉBEC CITY          
READ YESTERDAY’S ISSUE n=144 n=182 n=183 n=176 n=126
OF LE JOURNAL DE QUÉBEC          
Age 65+ 20% 18% 27% 21% 18%
University Graduate + 17% 15% 23% 19% 28%
Home Language French 99% 99% 100% 100% 100%
Manager/Professional 9% 9% 11% 6% 11%
Own Home 74% 57% 63% 57% 69%

Demographic and Reader Profiles

Comparisons of profile information within key demographic segments further support the hypothesis that online panel results differ from the benchmark telephone survey in ways that could influence estimates of newspaper readership and other media behaviour.

Blue-Collar Workers

Take, for example, blue-collar workers in Toronto [Table 10]. Depending on which online study is selected, fewer than 1-in-10 of Toronto’s blue collar workers would be between 18 and 24 years (9%, Panel C) or 1-in-3 would be (34%, Panel D). Based on the telephone survey, we would expect about 1-in-7 of these blue-collar workers to be young adults (15%, Telephone). As noted, all coding for occupation was performed by TNS Canadian Facts to ensure consistency of editing.

Of particular note is the propensity of online panellists to be individuals with English as their mother tongue and the language spoken most often in the home. Of Toronto’s blue collar workers identified in the benchmark telephone survey, we would

expect about 6-in-10 to name English as their mother tongue (59%, Telephone). In contrast, each of the test’s online panels produces a profile in which at least 7-in-10 (73%, Panel D) and as many as 9-in-10 blue-collar workers first learned English (89%, Panel A; 80%, Panel B; 87%, Panel C).

Toronto is Canada’s largest settlement region for new immigrants. For many of these immigrants, English is not their mother tongue. Are measurement tools that appear to be so heavily skewed toward the English-speaking/reading market a viable tool for estimating media behaviour for the entire adult population in a city such as Toronto?

TABLE 10

SELECTED DEMOGRAPHICS FOR BLUE COLLAR WORKERS IN TORONTO

TELEPHONE PANEL A PANEL B PANEL C PANEL D
GENDER          
Male 76% 70% 78% 64% 68%
Female 24% 30% 22% 36% 32%
AGE GROUP          
18-24 15% 11% 16% 9% 34%
MOTHER TONGUE          
English 59% 89% 80% 87% 73%

Online blue-collar workers also differ from those in the benchmark telephone survey in which daily newspapers they claim to have read yesterday, in their “content” readership and in the amount of time they spend with various media [Table 11].

Yesterday’s readership of the Toronto Star is higher among blue collar workers captured in the online panels, and especially Panel D (30%) relative to the telephone survey benchmark (15%). Similarly, The Toronto Sun, a tabloid with historically strong appeal among blue-collar Torontonians (21%, Telephone) ranges from a “read yesterday” readership level of 11% for Panel D to a high of 36% for Panel B. In summary, there is considerable and inexplicable “bounce” in the estimates of readership by blue- collar workers among the online panel studies and between the panel studies and the benchmark telephone survey.

Not surprisingly, the online cohort spends much more time on the Internet but also spends more time watching television but less time listening to the radio than do blue-collar workers in the telephone sample. Using the time spent on the Internet yesterday as the example, average time spent estimates varies dramatically between the telephone survey (93 minutes) and online estimates and among the online estimates, from a low of 123 minutes for Panel B to a high of 190 minutes for Panel C.

TABLE 11

MEDIA BEHAVIOUR OF BLUE COLLAR WORKERS IN TORONTO

TELEPHONE PANEL A PANEL B PANEL C PANEL D
READ YESTERDAY’S ISSUE:          
Toronto Star 15% 24% 21% 19% 30%
The Toronto Sun 21% 24% 36% 24% 11%
The Globe and Mail 3% 3% 4% 6% 8%
National Post 2% 3% 3% 1% 8%
Metro 11% 14% 14% 16% 21%
24 hours 12% 21% 18% 11% 19%
USUALLY READ:          
Sports pages 27% 31% 30% 33% 28%
Finance/Business pages 14% 15% 12% 21% 20%
AVG. # MINUTES SPENT          
On Internet 93 182 123 190 167
Listening to radio 196 159 137 162 166
Watching TV 120 183 145 215 149

Managers & Professionals

As evident in Tables 12 and 13, managers and professionals in Toronto experience levels of variation on key demographic and media behaviours similar to those experienced by their blue-collar counterparts.

TABLE 12

SELECTED DEMOGRAPHICS FOR MANAGERS/PROFESSIONALS IN TORONTO

TELEPHONE PANEL A PANEL B PANEL C PANEL D
GENDER          
Male 59% 56% 59% 62% 61%
Female 41% 44% 41% 38% 39%
EDUCATION          
University Graduate 57% 53% 51% 62% 74%
MOTHER TONGUE          
English 64% 71% 75% 77% 72%
HOUSEHOLD TENURE          
Own 82% 71% 73% 71% 86%
HOUSEHOLD COMPOSITION          
Adults Only 58% 65% 64% 61% 63%
Adults & Teens/Children 42% 35% 36% 39% 37%

Did one-fifth (20% Telephone) or one-third (34%, Panel D) of Toronto’s managers/professionals read yesterday’s issue of the Toronto Star? Do about one-third of them “usually” read the financial/business pages of daily newspapers (Telephone and Panels A, B, C) or do one-half of them (50%, Panel D)?

Their “other media” behaviour is also subject to considerable variation, depending on which online panel is compared to the benchmark telephone study.

TABLE 13

MEDIA BEHAVIOUR FOR MANAGERS/PROFESSIONALS IN TORONTO

TELEPHONE PANEL A PANEL B PANEL C PANEL D
READ YESTERDAY’S ISSUE:          
Toronto Star 20% 27% 29% 30% 34%
The Toronto Sun 10% 16% 16% 11% 8%
The Globe and Mail 17% 13% 15% 20% 27%
National Post 7% 9% 10% 8% 15%
Metro 12% 12% 11% 11% 12%
24 hours 7% 12% 6% 10% 7%
USUALLY READ:          
Sports Pages 30% 25% 25% 23% 31%
Finance/Business Pages 36% 30% 32% 32% 50%
AVG. # MINUTES SPENT          
On Internet 151 196 161 186 170
Listening to Radio 124 128 144 136 109
Watching TV 108 140 144 159 125

Young People (18 – 24 Years)

Appreciably more online panel youths are “students” than is the case among 18 to 24 year olds contacted in NADbank’s telephone survey. Those included in the online panels, like blue-collar and professional/managerial workers, are also especially likely to have English as their mother tongue. In this regard, however, there is not only variation between telephone and online panel youth, but also among the online panels (72%, A; 82%, B; 74%, C; 64%, D) [Table 14].

TABLE 14

DEMOGRAPHICS AND MEDIA BEHAVIOUR OF 18-24 YEAR OLDS IN TORONTO

TELEPHONE PANEL A PANEL B PANEL C PANEL D
EMPLOYMENT STATUS          
Student 39% 57% 49% 57% 46%
EDUCATION          
Some Post Secondary 48% 53% 57% 55% 56%
University Graduate 15% 14% 13% 16% 20%
MOTHER TONGUE          
English 57% 72% 82% 74% 64%
READ YESTERDAY’S ISSUE:          
Toronto Star 18% 19% 17% 27% 24%
The Toronto Sun 7% 8% 13% 14% 10%
The Globe and Mail 4% 8% 3% 10% 11%
National Post 2% 4% 3% 6% 6%
Metro 13% 15% 16% 17% 19%
24 hours 8% 11% 10% 11% 14%
USUALLY READ:          
Sports Pages 29% 15% 19% 22% 16%
Finance/Business Pages 8% 11% 8% 11% 16%
AVG. # MINUTES SPENT          
On Internet 187 275 218 296 236
Listening to Radio 129 120 136 120 135
Watching TV 143 160 171 195 148

Conclusions

  • There was as much consistency between 2006 and 2007 TNS Canadian Facts panels as there was between the two telephone samples for the top-line demographic and media habits data. Based on specific newspaper readership results, however, this consistency would appear to be a misleading metric for evaluating the validity of the online protocol.
  • There is considerable “bounce” among online suppliers’ study findings for key demographic and behavioural subgroups:
    • Critical currency data differ from one online panel supplier to another;
    • The characteristics of a newspaper’s audience differ substantively depending on which online panel is used as the source;
    • Demographic subgroups within the population (e.g., blue-collar workers; young people) differ substantively from one online panel to another in terms of their other demographic characteristics and in terms of their readership/other media behaviour.
  • Newspaper readership estimates (read yesterday) vary more extensively between the telephone and online studies in smaller centres (Halifax, Québec City) where sample sizes are smaller and recruitment of sufficient panel members is more challenging than in larger centres such as Toronto.
  • Among the online panel suppliers, findings vary in amplitude and direction from measure to measure. In this environment of flux, it would be difficult to develop a set of criteria on which to select an online panel supplier.
  • There is no firm basis on which to develop a conversion factor or weight that could bridge telephone and online findings because the directional and amplitude changes from panel to panel and city to city are too extensive.
  • Variation from one online panel to another appears to be a harbinger of the variation that might be experienced if a single online panel supplier were taking measurements over multiple points in time (year-to-year changes that are results of turnover in panel membership). However, only two data points were available, using the TNS Canadian Facts panel, and it is likely premature to judge the long term results from using a single web-based panel.
  • The extent to which differences between telephone and online results are a function of moving from an administered (telephone) to a self-completion questionnaire (online) cannot be estimated.

Discussion

The nature of the research that NADbank conducts requires a representative sample in order to accurately determine audience sizes and profiles for the daily newspapers being surveyed. The modified RDD telephone protocol has served NADbank well. The long-term expectation is that it will become increasing difficult to collect quality samples by this methodology. This research is part of an ongoing investigation into potential strategies for maintaining the quality of the NADbank Study.

The findings of this investigation confirmed the results of the first test that a web-based panel does not provide a representative sample, and secondly that different panels produce different results. While top level consistency was found in the TNS Canadian Facts panel (2006 to 2007), it is likely a red herring and does not provide a strong case for the suitability of the online panel protocol for media measurement studies. It would not appear to be the time for NADbank to move from its current telephone survey platform to the type of online panel used by commercial market research firms for multiple clients. These panels, regardless of the supplier, produce results that are sufficiently variable and fall short of face validity criteria (e.g., mother tongue in Toronto) as to render them unsatisfactory for estimating newspapers’ market share.

Given that a full shift to a new platform is not warranted it will be critical to continue to explore alternatives and potential transition opportunities to ensure that the platform provides valid, reliable results. It is essential that NADbank maintain data quality and trust in currency estimates. Next steps will be to explore alternative mechanisms for recruiting online panel members so that they more closely resemble a random sample of the population at large. In other words, ruling out existing commercially available “off-the-shelf” online panels for a newspaper audience study is not equivalent to ruling out a transition to any form of online data collection in the future.

NADbank will monitor developments such as the hybrid launched in 2008 for a large-scale financial services study in the U.K. (Cooke 2007) to determine whether a multi-modal approach could resolve some of the coverage and response rate issues associated with the current telephone platform.

Another option will be to consider building a panel from a random and representative sample of households (area frame) in major urban markets that could be accessed according to the preferences of the respondent (mail, online, telephone, face-to- face). Since the cost to develop a panel frame of sufficient size to meet NADbank’s needs and the costs of keeping the panel “refreshed” would be high, consideration might be given to working with a research supplier and/or other media organizations so that multiple users share the development and maintenance costs.

The inability to reach consumers with traditional research tools is growing but technologies are evolving to provide new avenues to reach those consumers. It is still ‘early days’ and a great deal of exploratory work will need to be done to assure ourselves that the new research protocols provide businesses with the data they need to make sound decisions. This test was another step along the way.

Appendices

Additional Summary Tables

TABLE A-1 GENDER AND AGE

TORONTO

WEIGHTED ACTUAL – unweighted

TELEPHONE TELEPHONE PANEL A PANEL B PANEL C PANEL D

COMPLETED SURVEYS 2063 2001 2214 2112 2029

GENDER

Male 49% 43% 52% 36% 45% 50%

Female 51% 57% 48% 64% 55% 50%

AGE

18-24 12% 9% 13% 6% 5% 9%

25-34 20% 17% 17% 15% 17% 20%

35-49 32% 37% 28% 37% 31% 33%

50-64 22% 23% 31% 28% 26% 23%

65+ 14% 13% 11% 14% 22% 15%

AVERAGE 18+ 45 46 45 47 49 45

HALIFAX*

COMPLETED SURVEYS 257 403 476 466 465

GENDER

Male 48% 41% 55% 36% 44% 48%

Female 52% 59% 45% 64% 56% 52%

AGE

18-24 12% 6% 12% 4% 5% 7%

25-34 19% 11% 18% 12% 16% 15%

35-49 30% 31% 30% 32% 29% 30%

50-64 24% 34% 22% 35% 29% 34%

65+ 14% 18% 19% 17% 21% 13%

AVERAGE 18+ 45 50 46 50 49 47

QUÉBEC CITY

COMPLETED SURVEYS 507 578 625 629 526

GENDER

Male 48% 42% 49% 41% 47% 47%

Female 52% 58% 51% 59% 53% 53%

AGE

18-24 11% 9% 15% 8% 5% 5%

25-34 18% 15% 19% 25% 17% 21%

35-49 27% 29% 23% 32% 28% 32%

50-64 26% 29% 31% 27% 30% 33%

65+ 17% 18% 11% 9% 18% 9%

AVERAGE 18+ 46 48 44 44 48 46

*Note that the base size for telephone interviews in Halifax is smaller than the online test because in the traditional telephone survey, two waves are generated and combined to generate annual estimates. The total annual sample for the telephone survey is 400.

TABLE A-2

DAILY NEWSPAPER READERSHIP IN TORONTO – WEIGHTED, PROJECTED

TELEPHONE PANEL A PANEL B PANEL C PANEL D
TORONTO STAR          
Read yesterday 22% 24% 30% 31% 35%
5 day cume 38% 39% 45% 47% 51%
Saturday 28% 32% 39% 38% 44%
Sunday 18% 21% 26% 26% 31%
7 day cume 45% 46% 53% 54% 59%
Online past week 13% 19% 20% 22% 29%
Total weekly 50% 53% 58% 61% 66%
THE TORONTO SUN          
Read yesterday 10% 14% 15% 12% 9%
5 day cume 22% 27% 28% 25% 19%
Saturday 8% 13% 15% 12% 9%
Sunday 12% 18% 19% 16% 11%
7 day cume 26% 32% 33% 31% 23%
Online past week 4% 9% 9% 10% 9%
Total weekly 27% 35% 35% 34% 27%
THE GLOBE AND MAIL          
Read yesterday 9% 8% 9% 12% 21%
5 day cume 19% 15% 17% 20% 34%
Saturday 10% 8% 9% 10% 19%
6 day cume 20% 16% 18% 22% 36%
Online past week 9% 9% 9% 12% 22%
Total weekly 24% 20% 22% 27% 43%
NATIONAL POST          
Read yesterday 5% 7% 7% 7% 12%
5 day cume 11% 11% 12% 12% 20%
Saturday 5% 5% 6% 6% 11%
6 day cume 12% 11% 13% 13% 21%
Online past week 3% 3% 4% 4% 8%
Total weekly 14% 13% 14% 15% 24%
METRO TORONTO          
Read yesterday 11% 12% 12% 11% 12%
5 day cume 22% 23% 22% 23% 23%
Online past week 1% 4% 3% 4% 3%
Total weekly 22% 24% 23% 24% 24%
24 hours          
Read yesterday 8% 12% 10% 9% 8%
5 day cume 16% 22% 19% 20% 16%
Online past week 2% 4% 3% 4% 4%
Total weekly 17% 23% 23% 21% 18%
ANY PAPER          
Read yesterday 46% 48% 53% 53% 62%
5 day cume 68% 67% 69% 71% 77%
Saturday 40% 44% 51% 50% 61%
Sunday 27% 33% 39% 37% 37%
Weekly print cume 73% 73% 76% 77% 82%
Online past week 21% 28% 29% 34% 44%
Total weekly 77% 77% 80% 82% 87%

TABLE A-3

DAILY NEWSPAPER READERSHIP IN HALIFAX – WEIGHTED, PROJECTED

TELEPHONE PANEL A PANEL B PANEL C PANEL D
THE CHRONICLEHERALD  
Read yesterday 40% 43% 54% 52% 54%
5 day cume 63% 63% 71% 71% 73%
Saturday 42% 43% 52% 48% 55%
Sunday 31% 34% 42% 39% 44%
7 day cume 69% 65% 74% 74% 76%
Online past week 14% 33% 37% 32% 37%
Total weekly 72% 75% 83% 80% 83%
THE DAILY NEWS  
Read yesterday 17% 19% 25% 20% 27%
5 day cume 32% 38% 43% 41% 44%
Saturday 15% 17% 20% 15% 24%
Sunday 15% 18% 20% 16% 21%
7 day cume 34% 39% 45% 44% 47%
Online past week 8% 15% 17% 14% 17%
Total weekly 37% 44% 50% 49% 52%
ANY PAPER*  
Read yesterday 48% 51% 63% 58% 65%
5 day cume 73% 73% 81% 77% 84%
Saturday 48% 51% 63% 53% 66%
Sunday 39% 43% 52% 45% 53%
Weekly print cume 77% 74% 83% 79% 86%
Online past week 17% 38% 42% 37% 47%
Total weekly 81% 81% 89% 86% 92%
*Note Any Paper includes the national newspapers which are not displayed here.  

TABLE A-4

DAILY NEWSPAPER READERSHIP IN QUÉBEC CITY – WEIGHTED, PROJECTED

TELEPHONE PANEL A PANEL B PANEL C PANEL D
LE JOURNAL DE QUÉBEC          
Read yesterday 29% 31% 30% 26% 23%
5 day cume 48% 46% 43% 41% 45%
Saturday 34% 37% 34% 37% 30%
Sunday 21% 27% 25% 30% 22%
7 day cume 54% 51% 51% 50% 50%
LE SOLEIL          
Read yesterday 23% 23% 25% 25% 32%
5 day cume 35% 35% 33% 31% 46%
Saturday 32% 32% 33% 40% 34%
Sunday 23% 23% 26% 30% 0%
7 day cume 41% 40% 42% 45% 51%
Online past week 7% 18% 14% 16% 21%
Total weekly 43% 45% 47% 51% 56%
ANY PAPER          
Read yesterday 48% 47% 48% 45% 48%
5 day cume 68% 63% 59% 58% 67%
Saturday 58% 58% 56% 62% 56%
Sunday 41% 43% 45% 50% 22%
Weekly print cume 77% 71% 69% 72% 75%
Online past week 7% 18% 14% 16% 21%
Total weekly 78% 74% 72% 77% 78%

References

Chakrapani, C. (2007) On the validity of online panels. Canadian Journal of Marketing Research, volume 24, issue 1, p.7-15.

Crassweller, A., Williams, D., Thompson, I. (2007). Online data collection – A solution or a band-aid?, Paper resented at the 2007 WRRS in Vienna.

Cooke, M., Watkins, N., and Moy, C. (2007). Market measurement: the next frontier for panel research. Panel Research 2007, ESOMAR World Research Conference, Orlando.

Rogers, J., Crassweller, A. and Williams, D. Between random Samples and Online Panels: Where is the Next Lily Pad? Panel Research 2008, ESOMAR World Research Conference, Dublin.

The authors

Anne Crassweller is President of the Newspaper Audience Databank Inc. (NADbank) in Toronto, Canada. Judy Rogers is the President of Research Resolutions & Consulting Ltd. in Toronto, Canada.

Donald Williams is Vice-President, Research Director at the Newspaper Audience Databank Inc. (NADbank) in Toronto, Canada.