Transcript Slide 1
Antonio Mocci External evaluation Ede - the Netherlands, 11 December 2009 Evaluation criteria Society Economy Environment Project Evaluation Impacts Needs problems issues Objectives Relevance Outcomes/ Results Inputs Outputs Efficiency Effectiveness Utility Sustainability Objectives for evaluating INTENT Consider the process of result production, of output realisation Analyse products in comparison to planned results Consider the impact of project outcomes on partner organisations and on VET systems. Process Working process was characterised by: Balanced responsibilities Respect for individual expertise and interests Use of ICT (Web and virtual workplace). Process A two level organisation to achieve two different objectives Results and outputs production Confrontation with stakeholders Products Three main products: The unit profile on International Entrepreneurship Training programme in five sub modules Assessment criteria and portfolio of International Entrepreneurship A robust dissemination programme ; Stakeholder involvement; Network interrelation Products Some characteristics of products The Unit profile is articulated according to the structure and the requirements of ECVET: this means the declination of a set of knowledge, skills and competence required to perform in an international context effectively and appropriately when interacting with others. Applicable to different geographical contexts and to different sectors. Products Some characteristics of products The Training programme is articulated in five submodules with specifications for the KSC structure of each sub module and for its articulation in didactic units. Pre requisite are indicated as well, in terms of personal motivation, abilities and previous intercultural experiences. Materials for participants and teachers are listed according to three levels: beginners, intermediate and advanced. Some characteristics of products Products The Assessment criteria set standards for the assessment of trainees in the programme; they also set guidelines for certification of the training programme. The Portfolio International Entrepreneurship Competence provides employers and employees, trainers and trainees a record of progress in key attributes of International Entrepreneurship Competence. It has three parts: A Passport of International Entrepreneurship Competence which records all formally assessed competences, A Biography of International Entrepreneurship Competence in which assessees may record personal experiences and encounters that may have contributed to their international and intercultural development, A Dossier of Evidence of International Entrepreneurship Competence in which assessees may keep any documentary or recorded evidence of their progress and actual competence to date. Project Impact Internal impact External impact On partner organisations On VET systems Project Internal Impact Internal impact seems to be stronger and more visible in relation to four areas: first, the integration of the results (or part of them) into qualification systems. Second, the integration of part of results into teachers’ and trainers’ “toolbox”. Third, the extension from the trade sector to other sectors and to different professional roles (workers, managers, entrepreneurs). Fourth, the extension to both IVET and CVET, although continuous training seems to be a more promising field of development. Project Internal Impact Points of attention: Existing results (UP, TP, AC) need to be translated in national languages. Adaptation needs to consider national legal frameworks and standards. The assessment criteria have to be adapted to the target group (students in the formal education system or non-formal system). Strong points: Flexibility of the solution adopted to design and deliver the training programme. Easy transferability of the unit profile and of training programme to other sectors and to different professional levels. The process of producing results involved several target groups (institutions, companies, VET providers), and this added value to results. Relevance of intercultural issues and competences in the process of building the European Union (Active citizenship). The outputs’ structure make possible the implementation of EQF principles (levels 3, 4, 5). Teachers, trainers and professionals have the possibility to use and customize training materials and assessment criteria. Training materials can be addressed to a variety of target groups, including entrepreneurs. Project External Impact Intent products - the unit profile, the training programme and the assessment criteria - provide for concrete and tangible tools to be utilised for using and applying ECVET. The concrete application of the ECVET tools and principles, the establishment of a credible network to be considered as a reference for system development, the capacity to transfer the INTENT approach into EQF application through NQF upgrading, seem to be the most outstanding impact effects this partnership produced on VET systems and represent the concrete added value INTENT gave to European lifelong learning. Conclusions Relevance: To what extent are the project objectives justified in relation to needs? Can their raison d’être still be proved? Do they correspond to local, national and European priorities ? In relation objectives formulated in the application form and articulated by the partnership in the inception phase, it is reasonable to say that INTENT was in line with the European policies finalised at improving mobility of students and workers. Consistency is particularly relevant with respect to the ECVET and EQF implementation policies. National priorities and frameworks have been considered. Conclusions Effectiveness: To what extent have the objectives been achieved? Have the interventions and instruments used produced the expected effects? Could more effects be obtained by using different instruments? Objectives have been achieved and planned results produced. Though time schedules had to be modified sometimes (for example, the research activity took longer than expected), outputs have been produced and planned effects were obtained. Results exploitation process was intense in the final part of the project, but it did not finish, according to what project partners declared. Beside internal effectiveness (the capacity to achieve objectives) it seems that the project had also external effectiveness (the capacity to produce results able to satisfy beneficiaries’ expectations). Conclusions Utility: Are the expected or unexpected effects globally satisfactory from the point of view of direct or indirect beneficiaries ? Though external evaluation does not deal explicitly with beneficiaries’ satisfaction, the analyses carried out by partners on training participation testify a visible interest and approval from participants. As for indirect beneficiaries (partner organisations and key actors) collected opinions indicate remarkable satisfaction for work methodology and results usability. Some recurrent indications (such as the need for considering national frameworks and the need for translation in national language) may improve results’ utility. Conclusions Sustainability: Are the results and impacts including institutional changes durable over time? Will the impacts continue when public funding is over ? The impact effects realised by the partnership seem to have activated a process of change and further development of outcomes which is going on in many partner countries. Durability over time seem guaranteed by partners’ commitment and by their roles as VET policies planners and implementers. Thank you for your attention !!!