A proposal to establish indicators for the evaluation of municipal councils’ socio-cultural policies

How can municipal sociocultural policies promote community empowerment, citizenship or social cohesion? What objectives should be proposed at the municipal level to promote processes


article propose des indicateurs pour l'évaluation des politiques socioculturelles qui permet non seulement d'obtenir une information utile, mais aussi de contribuer à produire des changements et à cerner des stratégies possibles d'intervention. L'article décrit un système flexible et rigoureux d'indicateurs à l'usage des techniciens et des élus municipaux intéressés à évaluer leurs actions dans le domaine socioculturel. Nous présentons le résultat d'un processus de recherche qui a inclus un travail de validation de la part d'experts et de techniciens familiers avec l'animation socioculturelle et l'application de ce système à partir d'une étude de cas dans une municipalité espagnole. À partir des indicateurs proposés, on perçoit clairement une forme d'appui aux créateurs locaux et la promotion de la citoyenneté, ainsi que le recours aux activités socioculturelles comme stratégie pour favoriser la diversité culturelle et citoyenne. Mots-clés : Politiques socioculturelles ; évaluation ; indicateurs ; municipal.
How can municipal sociocultural policies promote community empowerment, citizenship or social cohesion?What objectives should be proposed at the municipal level to promote processes of sociocultural community development?How can we identify local needs in order to stimulate community development processes?In response to these questions, this paper proposes indicators to evaluate sociocultural policies that will enable us to obtain useful information and, at the same time, contribute to produce changes and possible outlines for strategic interventions.The proposal describes a system of flexible and rigourous indicators intended for municipal technicians and politicians interested in evaluating their sociocultural actions and strategies.We present the result of a research process which included validation by experts and technicians in or related to sociocultural community development, and the application of this system based on a case study of a Spanish municipality.Among the proposed indicators specific consideration is given to the support of local creators and the promotion of civic creativity, as well as the use of sociocultural activities as a strategie to contribute to cultural and civic diversity.Keywords: Sociocultural policies; evaluation; indicators; municipal.¿Como pueden las políticas socioculturales municipales impulsar el empoderamiento comunitario, la ciudadanía o la cohesión social?¿Que objetivos debería plantearse un municipio para fomentar procesos de animación sociocultural?¿Como podemos identificar las necesidades de un territorio para estimular procesos de desarrollo comunitario?Para responder a estas preguntas, el artículo propone indicadores para la evaluación de las políticas socioculturales que nos permita no solo obtener información útile, sino también que contribuya a producir cambios y a apuntar posibles estrategias de intervención.El artículo describe un sistema de indicadores flexible y riguroso dirigido a técnicos y políticos municipales interesados en evaluar sus acciones en el ámbito sociocultural.Presentamos el resultado de un proceso de investigación que ha incluido un trabajo de validación por parte de expertos y técnicos afines a la animación sociocultural y la aplicación de este sistema a partir de un estudio de caso en un municipio español.Entre los indicadores propuestos se contempla de forma explícita el apoyo a los creadores locales y el fomento de la ciudadanía, así como el recurso a las actividades socioculturales como estrategia para favorecer la diversidad cultural y ciudadana.Palabras clave : Políticas socioculturales; evaluación; indicadores; municipal.
How can municipal councils' socio-cultural policies both aid and encourage the empowerment of the local community and favour local citizens' levels of creative activity and social cohesion?What objectives should a municipal council set for itself in order to foster socio-cultural activities?How can we identify our area's requirements in terms of the stimulation of processes of community development?In answer to such questions, this paper contains a creative proposal for the establishment of indicators for the evaluation of socio-cultural policies.It is a proposal which will enable us not only to obtain information about the true nature of the current situation, but will also help to bring about changes and identify possible strategies for future action by local authorities.
This paper is the result of work carried out as part of a doctoral thesis presented at the University of Girona in 2009.It highlights the lack of instruments of evaluation in the field of public policy, especially with regard to policies in the socio-cultural domain, and proposes and outlines a system of indicators intended for use by municipal officials and politicians interested in evaluating the activities and strategies undertaken in the socio-cultural context.At the same time this instrument should also make it possible to identify requirements and orientate future lines of action with a view to improving such policies, thus encouraging the development of municipal policies in this domain.
In the first section of this text we outline the evaluation criteria that we have applied when designing indicators for municipal councils' socio-cultural policies.In the second section, we describe the methodological process we have followed and finally we give our observations on the structure of the system of indicators we have devised.

An outline summary of the objectives of socio-cultural policies
A system of indicators would have no validity without a conceptual context to give it meaning and points of reference by which to evaluate it.We shall now give a brief outline summary of the theoretical framework of socio-cultural policies on which we have based our system of indicators.
Socio-cultural policy encompasses any services, programmes, projects or activities included within various different policy areas which have as a common objective the fostering and development of the most social dimension of culture, or more precisely all those which operate in the fields of participation, social inclusion and creation of identity, and in the processes of formation, dynamization, creation and promotion of the culture of each community.In terms of the holistic perspective on which we base our analysis we have found no frame of reference that specifically analyses socio-cultural policies.For this reason we have taken as our starting point the models and paradigms of cultural policies, and more precisely the paradigm of empowerment 1 .We have applied, then, an ideological model which focuses on enabling citizens and giving autonomy to groups and communities, which encourages policies giving access to culture and which stimulates lifelong creative and educational processes.We have therefore conceived socio-cultural policies as policies which should seek synergies between the various different actors operating in an area (local branches of the public administration, the private sector, the third sector and local residents) on a basis of cooperation, the fostering of participative mechanisms and a model of relational leadership.
Table 1: Evaluation objectives and basic contents for municipal councils' socio-cultural policies 1.To analyse the scope and characteristics of the facilities and interacting municipal services available Facilities and infrastructures are essential tools for community development in that they are communal spaces which permit both the provision of the socio-cultural activities on offer in an area and the subsequent contact and interaction with them.For this reason it is important to take into account the following basic aspects : the diversity of the existing facilities and services (both multi-functional and specialized); the means available for providing activities in terms of both technical aspects and infrastructures; the presence of specialized technical and professional staff; and the degree of coordination between services.
2. To examine the socio-cultural activities on offer and the resources enabling local entities to foster and promote these activities Socio-cultural activities contribute both to personal and community development and to the creation of a sense of identity through the participation and independent management of the groups involved.In this respect socio-cultural policies need to take into account the following aspects : diversity in terms of both those organizing activities and of the variety of and different ways of organizing the socio-cultural activities concerned, the financial resources available, the quality of the activities on offer and the coordination and networking required to make the efforts involved coherent and profitable.
3. To analyse the assistance provided and the policies giving access to the municipal socio-cultural activities on offer Socio-cultural policies should not be exclusive, but rather must cater to and provide opportunities for the full range of local residents' cultural requirements and interests.In order to ensure this councils may employ strategies such as the following : diversifying the activities on offer; promoting initiatives involving local social networks; introducing cost-reduction policies benefiting a wide range of collectives; decentralizing and ensuring fair distribution of facilities; monitoring local residents' opinions with regard to their requirements and interests.4. To identify whether programmes are proposed from a subjective viewpoint and not simply in terms of the objective requirements of the cultural sector concerned Transversal contacts and networking can encourage the proposal of programmes, projects and activities from a more universal viewpoint that is more enriching for local citizens.To this end councils may foster transversal contacts and networking in the following ways : by encouraging political and specialist leadership; by fostering systematic and methodical approaches; by employing clear evaluative procedures; and by taking into consideration the cultural and educational aspect.5. To analyse the support given to local creative talent and the municipal backing provided to foster local citizens' creativity Creative expression contributes to innovation, to social and cultural change and also to the forging of a community spirit.The following approaches are therefore required : to encourage local creative talent (both professional and amateur); to offer aid for creative activity; to make creative forms of expression accessible to local citizens; and to take innovation into account as a criterion for assistance to cultural projects.6.To examine municipal support for the diversity of cultures and origins present in the municipal area in terms of the use, creation and expression of socio-cultural activities In a multi-cultural society, development and change in any municipal unit are dependent on a willingness to listen and to engage in dialogue between the various cultural groups living in the area.With this in mind, socio-cultural policies will be subject to the following requirements : to foster programmes of educational adaptation and cultural promotion; to diversify the socio-cultural activities on offer; to develop multicultural programs; to encourage citizens' initiatives originating from collectives representing cultural minorities; and to introduce access programmes for groups threatened by poverty or social exclusion.

To analyse municipal support for promoting the creation of associations and citizen participation
Participation is an indispensable pre-requisite of a capacity for autonomous organization and individual and collective involvement in community projects.Socio-cultural policies should encourage it in the following ways: by fostering the creation of associations; by providing resources to citizens to promote their participation (facilities, financial resources, technical support, etc.); and by encouraging processes of institutional participation.
Within this framework and working on the basis of bibliographical research 2 , we set ourselves seven specific objectives to cover the full range of activities and aspects that should be included in any municipal socio-cultural policy, and which we outline in Table 1, together with the most important contents and aspects that should be included in each of the evaluation objectives proposed.

Towards the construction of a system of evaluation indicators
The definition of the conceptual framework which we have proposed in the preceding section constituted the first phase in the construction of a system of indicators for the evaluation of municipal councils' socio-cultural policies.We thus defined the objective of our study and the theoretical and political context in which our system of indicators is to be positioned.Once this framework had been established, we conducted our research in three phases: the initial proposal of indicators and the design of instruments for their construction and implementation; the validation of the proposal of indicators on the basis of validation by experts and their empirical application; and in conclusion the final drafting of the proposal.Let us now examine these phases at greater length.
Adopting a more detailed approach after defining the political and theoretical context, we established parameters for defining the evaluation of each objective, which thus led us to establish the relevant indicators.In this way we designed instruments so as to define and construct each indicator.These instruments enable us not only to define the criteria for the reading and interpretation of each indicator, but also to understand and interpret each indicator within the system proposed.Two types of instrument are designed for the construction of the indicators: one to provide a quantitative estimate, and the other to deal with more qualitative aspects.
In relation to the quantitative estimate we designed 15 fields to define the aspects that were most relevant for the application, understanding and interpretation of each indicator, as indicated in Table 2.
With regard to the qualitative analysis, we defined the parameters of analysis for each evaluation objective.These parameters are as follows: framework for the objective, indicators and level of success, qualification, and additional information.Each of these parameters is described in Table 3.
Once the tools for the construction of indicators have been designed and the first proposal has been defined, we enter into the third phase of the research process: that of validation.Validation is conducted on the basis of two methodologies: validation by experts; and the application of the proposal to a case study.
Validation by experts is intended to give the proposal validity and reliability in the same way that is achieved through triangulation systems for researchers by which, for Cohen and Manion (2002, p. 337), reliable and valid data can be obtained in research by the use of two or more independent observers or participants.The validation by experts that we conducted was carried out in two stages.The first, which was considered as a pilot test, was vital for the designing of the validation instrument.It had a double objective: on the one hand, to check understanding of the material prepared for validation; and on the other, the validation itself of the indicators 2. See among others López de Aguileta (2000), Pose (2006), Puig (1994Puig ( ), Úcar (1992)).
presented.Those participating consisted of five validators, who were experts and professional workers connected to the field of socio-cultural activities and evaluation.Their contributions were used to prepare the material and to fine-tune the indicators that were to be sent in the second stage.Those participating in this second stage were 14 validators from different parts of Spain (out of the 18 that we selected and for whom documentation was sent), and in the case of four of them we conducted prior interviews.Of the 14 referred to, 50% were academics linked to the study of socio-cultural subjects, followed by 28.6% who were academics linked to the field of public policy work.The others were academics from the field of evaluation and professional people working in the socio-cultural domain.The validators were requested to perform the following tasks: The validation process led to the conceptual clarification of certain indicators and of the limitations that should be applied to some of them, especially with regard to the collection of data.The need was also noted to simplify and break down some qualitative indicators to facilitate

Formula
Definition of the statistical formula on the basis of which the indicator will be defined.

Technical information
Definition of the variables, of the concepts involved and the technical observations required for the collection of data from the indicator.

Structure
Analysis of any interesting sub-component of the indicators.

Unit
Unit of measurement of the indicator (percentage, index, etc.).

Regularity
Regularity with which data must be obtained in order to measure the indicator (monthly, twice a year, annually, etc.).

Source of information
Where the data for constructing the indicator are to be collected.

Level
Level of success in attaining or adapting to the indicator.

Interpretation
Comparative information used to give meaning to the indicator.

Limitations
Description of the limitations of the indicator when measuring its objective.
Description of what the indicator does not measure.
a better application of the instrument concerned, and the elimination of some quantitative indicators.The process also made it possible to evaluate the importance and significance of each indicator in relation to the attainment of its objective (which made it possible to differentiate between basic and secondary indicators).
The changes suggested by the process of validation by experts are included in a new proposal which is subjected to empirical testing through its application in a particular municipal area.The methodological process employed is a case study.The following quotation from Stake (1995) is useful for situating this methodology in its context.
The real business of case study is particularization, not generalization.We take a particular case and come to know it well, not primarily as to how it is different to others but what it is, what it does.There is emphasis on uniqueness, and that implies knowledge of others that the case is different from, but the first emphasis is on understanding the case itself.(Stake, 1995, p. 20) Within the diversity of types of case study 3 , and bearing in mind the classification made by Stake (1998), this point in the process constitutes an instrumental case study.Its objective is to attain general understanding of a subject or of a theoretical aspect on the basis of a deliberately 3. See Rodríguez, G.; Gil, J; García, E. (1996, p. 92-98), Stake, R.E (1995, p. 16-17), Vázquez, R.;Angulo, F. (2003, p. 16-17), Bisquerra, R. (ed.) (2004, p. 314-316).

Properties Description
Framework This point contains a description of the criteria and indicators proposed.It is in effect the equivalent of a justification and explanation of the criteria and indicators selected.

Indicators & level of attainment
The instrument is based on the model proposed by the Arts Council England (2003).
A table is presented with various indicators to evaluate the objective, each of which is broken down into four levels in accordance with its success in meeting the criteria.
The four levels are: • Advanced : The desired situation has been attained with regard to the indicator or objective that we proposed.
• Established : Action is being taken to attain the situation of the indicator and objective.Some aspects corresponding to the indicator and objective have been consolidated.
• Emerging : Certain elements corresponding to the indicator and objective have begun to be attained, but the process is only in its initial stages.
• Absent : The indicator or objective has not been attained.
Some indicators can be used to analyse various different objectives.The establishment of the level of attainment of the policy is decided by the officials carrying out the self-evaluation process.The mark and rating given to each objective is used as an orientative tool, so the person conducting the evaluation must indicate the level of attainment that he considers most appropriate to the situation in his municipal area.

Rating
Scoring system which makes it possible to reflect the level of success in attaining each of the objectives, evaluated on the basis of the scoring for each indicator in accordance with the following scale: Advanced (3 points), Established (2 points), Emerging (1 point), Absent (0 points).

Additional information
Any information which is not collected in either the quantitative indicators or in the proposed qualitative indicators, but which is considered interesting to collect.
selected specific case.The application that we conducted was used both to confirm, change or increase our knowledge of the evaluation of municipal councils' socio-cultural policies and to carry out an exploratory analysis for its application in other municipal areas.The municipal area selected for the case study was Celrà, a village of nearly 4 000 inhabitants located in the Gironès district of Catalonia in Spain.Its high level of social and cultural dynamism and its large number of socio-cultural services in proportion to its demographic characteristics, together with the ease of access to relevant data for researchers, were key elements for this choice4 .
The results obtained from the empirical application led to a new proposal of indicators, which we present in the following section.

A flexible and creative system of indicators for the evaluation of municipal socio-cultural policies
The final proposal which is made for an evaluation system consists of a total of 78 indicators divided into two dimensions: the socio-cultural field, organized around seven evaluation objectives with a total of 58 indicators; and the contextual field, arranged around five evaluation objectives with 20 indicators.While the socio-cultural dimension makes it possible to identify and analyse the socio-cultural policy conducted by the local council, the contextual aspect enables us to provide a framework for these policies.
In each field we include basic indicators and secondary indicators, and in the case of the sociocultural dimension we also distinguish between quantitative and qualitative indicators.Basic indicators are those considered necessary for the evaluation of the objective to which they refer.For their part, secondary indicators make it possible to conduct a more detailed evaluation of the objective and to contribute additional information which may be of interest for the analysis of socio-cultural policies.
The result, then, is a proposal that is both selective and open, which can be enlarged or modified with a diversity of indicators and which can be applied either fully or partially, so that objectives can be evaluated separately.In this way a municipal council may apply only the indicators of those objectives which it is interested in evaluating, although full application is recommended in view of the relational character that links the various different indicators together.
The instrument is intended for use by municipal officials and politicians, since it is they who promote and implement municipal policies, who have an in-depth knowledge of the municipal area and who have all the necessary information at their fingertips.Although the responsibility for its application is assumed by officials and politicians, the implementation process may be undertaken in a variety of ways and may involve differing levels of participation on the part of the public, such as, for example, its application on the basis of more or less informal mechanisms for participation by local citizens.

Contributions to the field of socio-cultural community development
Finally, we should like to highlight some of the contributions of the system of evaluation that we have designed and indicate a certain number of proposals for the future in the field of the evaluation of socio-cultural policies.This is a tool that helps to provide a disciplined structure in which to place the implementation of socio-cultural policies, complementing and accompanying other tools and methods of evaluation used in this field.It thus represents a further step forward in the evaluation of sociocultural policies, and an element that contributes to the growth of evaluative culture.In the same way, to the extent that it is an instrument of self-evaluation for officials and politicians, it also constitutes an instrument for learning in that it can also help to reflect on professional practice in the field.
In a domain with such a limited tradition of documentation, it may also help to establish more systematic provision of information, acting as a guide both for the collection of data and for the standardization of criteria for all municipal councils that apply it.In this way it may help to minimize problems such as dispersion, the difficulty of making comparisons in temporal and spatial terms, the duplication of efforts, and a general lack of rigour in the collection of data in the socio-cultural field.
Finally, it can also become an excellent instrument for orientating both policy and the technical application of socio-cultural policies, very often developed by individual activists.
With regard to proposals for the future of evaluation in the socio-cultural field, we should like to insist on the need to apply the proposal in numerous municipalities so as to fine-tune and improve the indicators proposed; to design and standardize further methods of evaluation; to generate instruments for the collection of data; to expand the system of indicators for the evaluation of the entire socio-cultural dynamic in the area (and not just that promoted by municipal councils' public policies); and to introduce a system of evaluation to detect the impact of socio-cultural policies on other public policies.We are convinced that these features would help to contribute a greater degree of rigour, credibility and solidity to the field of socio-cultural community development and policies related to it.
To evaluate the quantity of indicators proposed in relation to the evaluation objective, proposing the elimination or inclusion of indicators, if necessary.• To evaluate the attributes of each indicator in terms of comprehensibility, relevance, coherence, comparability and measurability in the case of quantitative indicators, and in terms of comprehensibility, relevance and coherence in the case of qualitative indicators.• To contribute comments and observations in relation to each indicator, if they considered it appropriate to do so.

Table 2 :
construction of quantitative indicators