Surveying the field: A chat with Nik Nanos

  • December 07, 2017

Katya Hodge, Manager of Media and Public Relations, spoke with Nik Nanos about the polling his firm is conducting for the CBA.

The CBA has partnered with Nanos Research to do quarterly polls of the membership to find out what members want from the national association, in order to better focus its work.

So far Nanos Research has sent two polls into the field. In the interview Nanos discusses a bit of the methodology of the polls and offers up some very preliminary results.

A transcript follows

Transcript

Well, in the survey that Nanos has done on behalf of the Canadian Bar Association, we measured satisfaction and value. And what we've seen in the first two waves is that both of those measures, on a scale of one to ten, are just a little below seven out of ten. So one thing that we've learned is that there's an opportunity for growth in terms of people's perceptions related to the satisfaction that they have with the CBA and the value that they receive for their CBA membership.

So the way the survey is done it is an online survey, and what we've done is created two random samples of the membership and invited every member, who will have a chance to share their views over the course of the year, to participate in a survey. So people are randomly selected to be in a sample across all of the branches and parts that make up the Canadian Bar Association.

The reason why we do this in an online survey is, first of all, because the cost is very efficient, second of all we can cover off more content in an online survey, and third we have to remember lawyers are busy. To try to deploy this as a telephone survey would be cost prohibitive and actually wouldn't probably really work too well because one of the key tenants to good research is to engage respondents and participants on their terms and not on your terms. So the best methodology is the online methodology that we've proposed for the CBA.

Quite well. The response rate that we've had for the first two waves of survey is around 10%. You might think that's low, but the reality is that response rates are not indicative of the quality of the research, they're just indicative of what the response rate is and that you know only so many people are available at that time. What we look at is who is responding and how that matches up to the membership list, and what we've seen is that there's a natural pattern in terms of the people that are responding to the to the survey that aligns with the natural pattern in the membership list, so we have a very high level of confidence in the reliability of the results. So for the survey that we did, for example the first survey which was sent out to half of the membership the accuracy for that is plus or minus 2.6 percent 19 times out of 20, and the second survey that we did we actually sent it out to everyone who is a member. So that's the good news. There'll be a larger data set in order to do a more robust analysis, but because everyone was invited there's no randomness in the selection of the participants.

Actually, you have to think of it not just in terms of accuracy but in terms of the trend right what we need to do is to look at how people's perception of their satisfaction with the Canadian Bar Association changes, or their perceptions of the value that they receive for membership over time. So people may be fixated over the specific number, but what I always look at is the direction. And I think this is where the power of the tracking over time, because the tracking is done every 90 days, it’s going to be very critical to the senior leadership at the Canadian Bar Association and also to the branches right across because now we'll have data in terms of not just the big national Picture, but how new lawyers think about the CBA, how people in different branches kind of value their membership in the Association and all those kind of micro trends, so that's really where the power is going to be.