Over the past three or four years that I’ve worked in the US cultural sector, I’ve noticed that arts organizations are experimenting with the way that we count. Or, perhaps it is more accurate to say that we are developing more holistic ways of measuring what really counts. The recent Counting New Beans study challenged theaters to go beyond economic impact or numbers of “butts in seats” to measure the intrinsic value of their work. Similarly, many organizations are striving to measure intangibles such as attitudinal change or increased empathy in customers. My colleague Gail Crider and I spent the summer looking at different ways that cultural organizations are measuring the impacts and outcomes of their programs as well as the frameworks they used. We presented our findings at a meeting of the minds in Baton Rouge called Social Theory, Politics and the Arts.
One of my favorite things we discovered were the collectives of organizations that came together to create a large-scale study of impact. Some of these collaborations, like Dallas’ Thriving Minds initiative, included a range of social service and cultural organizations concentrated in a particular community and others were diverse in terms of geography, but included organizations with very similar goals. The Association of Zoos and Aquariums (AZA) is a great example of the latter category, and their story provides a useful example of a practice that any type of organization could employ.
It all started with an assumption about impact. The zoos and aquariums represented in the AZA have an important aspect in common: they are all accredited institutions that are bound by ethical standards in terms of animal care, public presentation and recreation. The leaders of these institutions always felt strongly that zoos and aquariums contributed to large-scale conservation efforts because exhibitions taught visitors about animals and their habitats, and an assumption was made that this knowledge would eventually lead to shifts in attitudes and behaviors. It was a good theory, but it had never been proven. As scientists themselves, the leaders weren’t completely satisfied speaking from an educated “hunch,” and wanted to test their theory to ensure that, as a field, they were having an impact on conservation.
Leaders in all organizations make decisions based on both analyzing data and strong gut feelings, and I do believe that it’s important to find a balance between the two. However, just as it’s important to question the validity and quality of data, it’s also important to question your hunches. There are aspects of our business as cultural organizations that we assume to be true. As Andrew Taylor of American University pointed out in a presentation, the assumption that we still need a physical box office space persists despite the proliferation and popularization of online ticketing platforms. Some of these assumptions – including physical spaces, relationship with customers, and business models – are deeply seeded in the DNA of our organization and it takes a lot of courage to call them into question.
For the zoos and aquariums, the prevailing assumption had to do with impact on conservation. Member organizations from across the country came together to design and implement a study on the impact of their work on conservation efforts called “Why Zoos and Aquariums Matter: Assessing the Impact of a Visit to a Zoo or Aquarium” with support from the National Science Foundation and in partnership with Institute for Learning Innovation and Monterey Bay Aquarium. This was an elegant, multi-layered study that used (among other methods) immediate pre- and post- visit surveys to measure any change in visitor knowledge and attitudes about conservation. Later on, follow up interviews were also conducted with a portion of these visitors to see whether any of their actions had changed since visiting the zoo or aquarium. Sure, visitors now knew that turning off the water contributed to habitat preservation, but did they actually do it?
The participating organizations were pleased with results of the study, which indicated that zoos and aquariums do contribute to conservation efforts. Gathering this data was useful in countering harmful claims from detractors – such as animal rights activists – who claimed that the zoos and aquariums did more harm than good to animals and their habitats. It also helped organizations communicate their value to the community, visitors and donors. The results pointed leaders towards the types of activities they should do more of in order to maximize impact. What’s more, the multi-institutional study resulted in an evaluation tool that was useful for the AZA’s individual member organizations.
Ohio’s Akron Zoo is a great example. At Akron Zoo, Executive Director Pat Simmons and her team are planning a new exhibit called “Grizzly Ridge,” which seeks to raise awareness of the human interventions that threaten grizzlies and their habitats. I had a chance to catch up with Simmons earlier this fall and discuss how the AZA’s study has provided a helpful, ready-to-use tool for her organization. She and her team have designed the exhibition with visitor education in mind, drawing on the lessons learned about what works and what doesn’t from the study’s findings. To evaluate their effectiveness in changing people’s attitudes and actions, they plan to use the survey methods from “Why Zoos and Aquariums Matter.” Not only will they use the survey results as evidence for funders and other external audiences, they will also rely on evaluation data to inform how they can continuously improve the exhibit’s education materials.
Overall, Simmons noted that the model presented in the AZA study has been a useful tool for her own organization as well as other cohort institutions. Too often, great studies get set aside because the lessons coming out of them aren’t digestible. AZA, on the other hand, published its research not to be filed away, but to be put into action by smart leaders like Simmons.
Digging deeper into the meaning of this study, I also considered how the simple act of questioning assumptions can strengthen an organization. The leaders in this story put a deeply-held belief under the microscope – a very difficult thing to do – and were able to affirm that their theory of change was correct. Furthermore, the leaders were better able to direct their efforts once they understood the why and how behind their success in mission achievement. What do we know to be true about our work, and when do we operate off of a hunch? What’s at stake if we question our assumptions? These are important questions and while you will need to muster up a great deal of courage, it’s not necessary to have a service organization or a professional evaluator on board to ask them.
