Arts organizations are asked to create dozens of reports every year evaluating their programs to justify the funding they receive so you probably aren’t inclined to embark on an additional self-evaluation of their own.
However, the group, IETM – International network for contemporary performing arts, recently issued a self-evaluation tool kit specifically designed for arts organizations with the goal of empowering groups to take control of the evaluation narrative. IETM say that many funders employ criteria that suits their own agenda, or perhaps to justify their existence, and may either misrepresent or “hinder artistic processes and quality.”
If you decide to be proactive and to construct and conduct your own evaluation, you will step into a stronger position to define what is of value and what can be considered an impact in connection with your work.
You will be able to choose the paradigm, the approach, the logic which fits your needs best. You will be empowered to assert your own agenda.
The toolkit they provide is the most comprehensive evaluation guide I have ever seen. (I mean, I had never heard of autoethnography before.) The guide discusses the pros and cons of performing an evaluation in-house versus engaging an outside consultant versus participating in part of the data collection effort of a larger entity (i.e. national or regional arts council survey).
The toolkit is incredibly detailed with guidelines for the entire process: deciding to do the evaluation; deciding on the goals; the strengths and weakness of quantitative and qualitative measures; designing the process and measurement tools; tips how to ask questions; how to ensure you are surveying a valid sample; motivating people to participate; different equipment checklists for data collection methods; and suggestions about presenting the findings.
The toolkit clarifies the difference between seeming synonymous terms like outputs, outcomes, and impacts. They also offer suggestions about clarifying the meanings of terms you use. When you say artistic vibrancy, what does that mean? Okay, now ask your friends what they think that means.
Where I feel the guide is most helpful is evaluating different data collection methods so you aren’t left wondering if you should do a survey, roundtable, focus group, etc. They address each method describing what it is, discussing the suitable application of information collected by that method, how to prepare to use that method, how to administer the survey/act during the discussion, how to process the information gathered and then how to use it.
By the way, the references to terms like autoethnography notwithstanding, the toolkit is very easy and enjoyable to read. There are comics from the site xkcd throughout. Their advice on getting feedback on terms like “artistic vibrancy” was provided thusly:
Ask friends and acquaintances how they understand ‘artistic vibrancy’ for example. They probably think you are weird already, so your questions will not surprise them. Test your definitions with them. These other views and versions will be useful when you start to formulate questions for the assessment respondents.
In the same vein, they provide suggestions about presenting your findings in a manner characteristic of an arts organization. Good graphic design, infographics, storytelling, sure,
But why stop here?
You could organise a conference, an exhibition, a workshop, a production, a parade, a flash mob. You could make it into art.
