Flipped Classroom and Aronson’s Puzzle: a combined evaluation in the master’s degree in pre universitary teaching

Aula invertida y puzle de Aronson: una evaluación combinada en el Máster del profesorado

Carlos Lázaro-Carrascosa^a, Isidoro Hernán-Losada^b, Daniel Palacios-Alonso^c, Ángel Velázquez-Iturbide^d

^aEscuela Técnica Superior de Ingeniería Informática, Universidad Rey Juan Carlos, Móstoles, España

^bEscuela Técnica Superior de Ingeniería Informática, Universidad Rey Juan Carlos, Móstoles, España

^cEscuela Técnica Superior de Ingeniería Informática, Universidad Rey Juan Carlos, Móstoles, España

^dEscuela Técnica Superior de Ingeniería Informática, Universidad Rey Juan Carlos, Móstoles, España

This article presents a double evaluation carried out in the subject Didactics of Computer Science and Technology, corresponding to the Master's degree teacher training in secondary education, baccalaureate, vocational training and languages taught by Universidad Rey Juan Carlos. Students of the subject had to learn how to prepare simple web pages, using HTML, CSS and JavaScript programming languages. To this end, flipped classroom technique was used to present the necessary contents, combined with an adaptation of Aronson's cooperative learning puzzle technique, used to carry out a group practice that reflected the knowledge acquired. It is worth mentioning, as a complement to the two techniques used, the use of an adapted assessment rubric, which was provided to the students at the beginning of the teaching block. The evaluation was carried out during two consecutive academic years, 2018/2019 and 2019/2020. There were important differences between the two studies: in the first study, students' previous self-assigned level was much higher (2.8 points as opposed to 1.4 points on a scale of 1 to 5). The other difference, even more relevant, was that in the second year all teaching was done at home, in a non-attendance format, on a mandatory basis, due to the period of confinement decreed by the state of alarm at that moment, because of the pandemic caused by the SARS-CoV-2 virus, popularly known as coronavirus. It is remarkable that, despite these differences, the students expressed their satisfaction with the learning acquired and with the tasks performed in both cases. The techniques used were well-appreciated, in first year more than in the second, and especially flipped classroom. Scores obtained were, in addition, always very relevant.

Keywords:
Flipped Classroom
Aronson’s Jigsaw
Cooperative learning
Teacher training
Assessment
Face-to-face teaching
On-line learning.

Este artículo presenta una doble evaluación realizada en la asignatura Didácticas de la Informática y la Tecnología, correspondiente al Máster universitario en formación del profesorado de educación secundaria, bachillerato, FP e idiomas que imparte la Universidad Rey Juan Carlos. Los alumnos de la asignatura debían aprender a preparar páginas web sencillas, utilizando los lenguajes HTML, CSS y JavaScript. Con tal fin se utilizó la técnica flipped classroom o aula invertida para dar a conocer los contenidos necesarios, combinada con la adaptación de la técnica de aprendizaje cooperativo puzzle de Aronson, utilizada de cara a la realización de un trabajo grupal que reflejase los conocimientos adquiridos. Cabe destacar, como complemento a las dos técnicas utilizadas, el uso de una rúbrica de evaluación adaptada, que fue facilitada al alumnado en el inicio del bloque didáctico. La evaluación fue realizada durante dos cursos académicos consecutivos, 2018/2019 y 2019/2020. Se produjeron diferencias importantes entre ambos estudios: en el primero de ellos, el auto asignado nivel previo de los estudiantes fue mucho mayor (2,8 puntos frente a 1,4 puntos en una escala de 1 a 5). La otra diferencia, aún más relevante, consistió en que en el segundo curso toda la docencia se desarrolló en casa, en formato no presencial, de forma obligatoria, debido al período de confinamiento decretado por el estado de alarma en vigor a la sazón, a causa de la pandemia provocada por el virus SARS-CoV-2, popularmente conocido como coronavirus. Es destacable que, a pesar de estas diferencias, los alumnos manifestaron su satisfacción respecto al aprendizaje adquirido y al trabajo realizado, en ambos casos. Las técnicas empleadas fueron bien valoradas, especialmente flipped classroom y principalmente en el primero de los dos cursos académicos. Las calificaciones de los trabajos fueron, además, siempre altas.

Palabras clave:
Aula invertida
Puzle de Aronson
Aprendizaje cooperativo
Formación del profesorado
Evaluación
Docencia presencial
Docencia a distancia.

The Master's Degree in Teacher Training for Secondary Education and Baccalaureate, Vocational Training and Language Teaching (Boletín Oficial del Estado, 2008) is taught in a large number of public and private universities all over Spain. This qualification is mandatory to access to the national teaching Spanish system. For this reason, this master is in demand. The master is composed of 60 ECTS credits, which are divided into common subjects, specific subjects of each specialty (mathematics, language, social sciences, English, etc.), practicum and a final master's work. Specifically, the experimentation carried out has been taken place within the specialty of computer science and technology offered by Rey Juan Carlos University.

Most of students enrolled in this specialty are engineers from different branches: industrial and computer engineers are usually the most common, although there are also electrical, biomedical, mining, civil, agronomist, telecommunications or topographical engineers, and even some architects. Part of the content subject to experimentation is a section of a training complement that consists of instructing in basic web programming, which includes an explanation of the fundamentals of HTML markup language, CSS presentation language and JavaScript language (Gauchat, 2012), used to provide web pages with simple interaction.

Student profile has two main characteristics: on the one hand, all students have a technical background. On the other hand, knowledge of computer programming is dissimilar: although it is not a branch of knowledge completely alien to practically any student, some of them have extensive experience, while for others the opposite is true. There are also some people who claim that their current programming level is almost null. It should also be considered that the content to be taught has a high practical component: the required result is the development of a complete project, and students, people who usually study, work and often have family duties, suggest that face-to-face classes can serve to advance the project. All these considerations led us to use the technique of inverted class or flipped classroom, suitable for students with different abilities and oriented to use the time in classroom for practical issues (Lage, Platt, & Treglia, 2000).

Flipped classroom also clearly contemplates the possibility of including cooperative learning in classroom (Lage, Platt, & Treglia, 2000) (Fortanet, Díaz, Pastor, & Ramón, 2013), and Aronson's puzzle (Aronson, Blaney, Sikes, Stephan, & Snapp, 1975) represents a variant of this type of learning. It is a particularly adequate alternative, which divides global task into different parts, each of which is entrusted to one of the members of the group. This fits in well with the fact that several languages are involved in web pages, and, above all, with the fact that participants have different backgrounds (Perkins & Tagler, 2011), which is why Aronson's puzzle was adapted and combined with inverted classroom.

Another important element in the study was evaluation headings, or rubrics. Rubrics are methodological tools that allow evaluating the knowledge and performance of students. Rubrics help students to better understanding the criterion that teacher has followed to grade a job and what is lacking in order to reach the completion of their learning objectives (Reddy & Andrade, 2010).

Pedagogical techniques used in this research work will be described in section 2. Section 3 is addressed to reviewing some classroom experiences that have used these or similar methodologies. Section 4 describes in detail the process conducted, results are presented in section 5 and, finally, conclusions are summarized in section 6.

The technique known as flipped classroom consists of changing the modus operandi that has been considered traditional when it comes to teaching: time in the classroom is dedicated to explaining contents, generally through master classes, and time at home is dedicated to proposing practical tasks related to contents learned. Inverted classroom proposes, in general, to use technological means (videos, web pages, summary documents in slide format, diagrams, articles, etc.) to learn contents at home and to use classroom time to solve doubts or concerns and, in a generally collaborative way, to solve issues associated with contents (Bishop & Verleger, 2013). One of the original motivations for applying this technique lies in the search for coincidences between teaching styles of teachers and learning styles of students, thus improving overall process (Lage, Platt, & Treglia, 2000).

To highlight the impact of inverted classroom, the Flipped Learning Global Initiative (Flipped Learning Worldwide, 2019) is based on a group of teachers and researchers from 49 countries who work on everything related to this technique. On their website, we can find what they call basic principles:

- Thinking: the technique requires a profound redefinition of the roles of students, teachers, and of the times in which both interact.

- Time: the correct application of the technique requires large amounts of time, so it is essential to have explicit support from school management.

- Keep it simple: success is based on achieving more profitable relationships between students and teachers, keeping the process as simple as possible.

- Professional development: it is necessary, from a pedagogical point of view, to know perfectly which the best practices within the inverted classroom are.

- Technology: it is essential to apply the right technology, having a great mastery over it.

- Take it global: sharing experiences is important in order to cover correctly the above points.

Aronson's puzzle (Aronson, Blaney, Sikes, Stephan, & Snapp, 1975) is a cooperative technique that, like inverted classroom, is based on a change of roles in the sense that teacher ceases to be the only expert in the classroom and begins to play a mediating and consulting role, leaving the prominence to students. It is appropriate for any work that can be divided into phases or sub-tasks and works best if students have some prior experience with the assigned task. (Perkins & Tagler, 2011)

1. Explanation of the technique: in activities that promote the autonomy of students, it is important to clarify the aims of the activities, as well as to agree on certain rules to be fulfilled within the group work: commitment, punctuality, etc.

2. Mother group configuration: the work to be carried out is divided into different tasks. The teacher usually makes this division. Each task will correspond to a role, which will be assumed by each of the members of the groups into which the class will be divided, in such a way that in each group there appears a representative of each role, called expert. It is frequent that the teacher carries out the composition of the groups, because this task may require a previous knowledge of the students.

3. Start-up of the group of experts: experts in each task form a group, in which all together try to deepen the resolution of the same, through different processes: search for documentation, partial presentations, pooling, and preparation of a final report may be some of them.

4. Meeting in the mother group: each expert returns to his original group, and contributes what he has learned for the achievement of the group work.

Rubrics are methodological instruments that allow us to standardize the assessment based on specific criteria. In other words, a rubric is a “document that articulates the expectations for an assignment by listing the criteria or what counts and describing levels of quality from excellent to poor” (Reddy & Andrade, 2010). Therefore, a rubric must have, at least: the evaluation criteria (the specific qualities observable in the product or in the process carried out by the students) and the descriptors (offer a detailed explanation of what a student has to do for demonstrating a proficiency or skill) with rating scales.

Use of rubric determines its objective. If only the teacher to assign the grade uses it, rubric is not for learning. If rubric is used for the student to self-evaluate, to evaluate other students or to receive feedback from the teacher, then rubric is for learning (Andrade, 2005).

Flipped classroom is being used with certain frequency in many different subjects, including computer science, both in secondary and higher education studies, and in these experiences, it can be observed that, in general, acceptance results tend to be good, although there is not a large number of studies that allow relevant conclusions to be drawn regarding the improvement of learning. On the other hand, studies do not usually cover long periods of time, focusing on specific lessons or didactic units (Bishop & Verleger, 2013; Aguilera, Manzano, Martínez, del Carmen, & Yanicelli, 2017; Urquiza, 2015).

As for Aronson's puzzle, we found that its use implied a statistically significant improvement in reading comprehension in secondary school students (Hapsari, Suparman, & Putrawan, 2019), as well as an improvement in attitude and ability in linguistic matters (Susanti, 2018). In the field of mathematics, we again find a significant statistical improvement in students who used the technique for teaching irrational roots and logarithms in a secondary school (Mbacho & MWebi, 2018). In addition, the technique has been linked to an increase in certain social skills (Aronson, 2002).

Focusing on the context of this article, in the work of (Rodríguez & Campión, 2016), we find an evaluation of the application of Flipped Learning in a subject of the Faculty Master’s Degree (Boletín Oficial del Estado, 2008), where it offers good results of acceptance of the technique that include verbalized opinions of the students. In addition, same article stands out that methodologies of this nature are not used more frequently by teachers due to the lack of time to learn them, among other causes. Therefore, using them in the period of teacher training may be particularly appropriate, in order to encouraging their future use.

The effectiveness of rubrics as a pedagogical instrument has been studied. Reddy and Andrade publish a review of rubric use in higher education (Reddy & Andrade, 2010). In their study, they assert that the perception of students is positive because they see what they needed to do, what they had achieved, and what they needed to improve. Atkinson and Lim agree with the previous study (Atkinson & Lim, 2013). They show that students are pleased with the rubrics and request their use in the following courses. Although rubrics have also been criticized, there are no well-founded studies showing that rubrics are not suitable for learning (Panadero & Jonsson, 2020).

Finally, we will highlight an experience in which, as in our proposal, cooperative learning is explicitly combined with inverted classroom (Fortanet, Díaz, Pastor, & Ramón, 2013). This article offers good results in terms of acceptance of the technique and classroom dynamics, and mentions the difficulty encountered in applying group techniques due to the conventional distribution of spaces.

Introduction to web programming is a subset of the subject we teach. It is based on the explanation of three languages: HTML (markup language used to format the page, including elements such as titles, lists, images, tables, etc.), CSS (style language used to decorate HTML elements, handling concepts such as color, margins, background, etc.) and JavaScript (programming language used for the user to interact with the page, offering output values to a given calculation or process) (Gauchat, 2012). The goal required to pass the subject is to program a simple web page that must present a basic content related to the teaching of some subject related to technology, as a tutorial or didactic unit. Also, the web page has to offer a self-evaluation form that includes a simple corrector. Programming of the page requires the combined use of the three languages, but it should be noted that parts of the process, although enjoying a certain autonomy, are not independent, so the individual study of each of them is harmonized with the need to work cooperatively to reach the result.

It is noteworthy that the students who study this master have to reconcile their personal and professional life with these studies, whose classes begin at 5 p.m. and end at 9 p.m. For this reason, they prefer that classes have a practical component that makes them more attractive, and they prefer to seize them to advance practical work that they must deliver to pass the subject. We will now explain the processes carried out in the two experiments.

The experiment was replicated in the following academic course, in order to increase the sample, try to confirm the results and reinforce the conclusions. The number of students who participated in this course was 28. Among them, we find computer engineers of diverse specialties (computers, management, systems, business computing), industrial engineers, also of different branches (mechanics, industrial technician, industrial technology, electrical) and other profiles of the scientific-technological area (engineers in audiovisual systems, topographical, and biomedical). In a survey performed at the beginning of the academic year, 21 of them expressed knowledge of various computer-programming languages, and 14 out of these cases, languages were related to web programming. In a Likert scale from 1 to 5, where 1 corresponds to null experience in the specific topic surveyed, and 5 corresponds to full experience, average mark was 2.8.

We had five working sessions of approximately a hundred minutes each. Sessions were separated by a week. In the first of them, the methodology used was explained, which was based on an inverted classroom during the whole process: material to be used would be provided previously, and time in classroom would be dedicated to carrying out the group project. From second session to fifth session - dedicated to evaluation - doubts of all kinds would be solved, and Aronson's puzzle methodology, also introduced in the first session, would help them. Next, procedure followed in each of the sessions will be detailed.

First working session was therefore devoted to explaining the methodology which it was developed. On the one hand, always through a learning platform based on (Moodle Project, 2019), bibliography would be provided and, above all, links to websites specialized in offering dynamic tutorials related to the languages. Summarized notes would also be provided in the form of slides, including practical examples already programmed. This material was briefly exposed in class. On the other hand, the working groups were configured in a balanced way, in such way that in each one of them at least one person with certain experience in the matter would be included. In addition, it would be necessary to establish the roles, which were four in this case: HTML expert, CSS expert, JavaScript expert and head of group. This last role, reserved for the most experienced person, was to maintain an overview of the project, deciding on the overall structure of the page, and organizing the necessary interactions between them. In general terms, roles had to be chosen by descending order of experience or training in the subject: headfirst, then JavaScript, then CSS and finally HTML. In addition, it was also explained what subject matter the website should have to prepare, and, above all, evaluation rubric that was used to award the rating was presented. As we will see, this element is of special interest, it is highly valued by students, as it includes in detail and from the beginning what is expected of their work.

Second session, with groups already configured and some of the material reviewed at home, began with a brief round of general doubt resolution, to give way to a meeting of experts: in this meeting, new groups were temporarily configured, formed by those people who shared a role. Some action guidelines were provided for the meeting, in order to make it more dynamic:

- HTML role: Work the general structure of the page, with paper prototypes. Discuss what elements are necessary to achieve that structure.

- CSS Role: Raise the difficulties encountered with the general scheme of decoration of elements. Study the possibility of decorating common elements, as well as individual elements. Contribute with general ideas to decorate the most frequent elements.

- JavaScript role: Confirm understanding of the general scheme for including language in web pages. Review the code examples provided. Try to choose the appropriate types of questions for each of form elements.

- Coordinating role: Study the necessary interactions between the three roles, establishing possible dependencies and joint decisions. Review the rubric provided, commenting on the points.

Third and fourth sessions were dedicated to group work within the mother group. The teacher solved all the doubts that arose, both those related to methodology and evaluation as well as those referred to purely technical aspects. Although, a second formal meeting of experts was not held, specific interactions between the different groups were encouraged and produced in a collaborative manner.

Fifth session was reserved for public presentation of the work. In each group, intervention of all the members was mandatory. Presentations lasted about ten minutes, with the teacher offering a brief feedback at the end. In addition, students filled out a questionnaire of acceptance of the methodology followed in this last session.

Questionnaire was divided into four blocks: firstly, previous level of the participants; secondly, assessment of aspects related to the flipped classroom technique; thirdly, assessment of aspects related to Aronson's puzzle technique; and fourthly, degree of conformity with statements related to general aspects related to perceived satisfaction and benefit; (see Table 1). In all cases, a five-level Likert scale was used to collect the answers. In addition, second and third blocks also included open questions on the appropriateness of the content to the techniques, and in the case of Aronson's puzzle (third block), also on the role carried out.

The number of students who participated in this course was 32. We also found engineers of different specialties, as it is usual in this master’s degree branch: computer, industrial, electrical, agronomist, telecom, etc. However, relation with the computer languages was not very strong this time. This aspect was rated in the question on previous experience in the matter: it gave a result of 1.4 points (minimum 1 point, maximum 5 points) exactly half of the previous course.

The original intention for this course was to replicate the previous experiment carried out the last year, incorporating some lessons learned, in order to obtain conclusions that more relevant. However, this was not entirely possible, because the way of teaching changed suddenly, to a strictly online format, due to the obligation to stay at home required by the state of alarm caused by the COVID-19 pandemic (Arango, 2020; García-Peñalvo, Corell, Abella-García, & Grande-de-Prado, 2020).

In these circumstances, we needed to adapt the methodology. First step did not suffer any variation respect of the previous year: a brief online explanation of the methodology. Secondly, we would offer at the beginning all the material necessary to carry out the project: objectives, links, evaluation rubric, examples of codes and, improving from last year, different videos that explained all the concepts. Thirdly, we would attend the groups by e-mail, using specific forums, and with video conferencing if it was necessary or requested. Finally, we had to lower the level of demand related to Aronson's puzzle: we explained the entire methodology, but the division into roles was not mandatory. Despite this, the groups were fairly balanced in terms of programming experience, and the students had at their disposal standard technological means to collaborate with each other (mail, chat, videoconferencing).

The questionnaire (see Table 1) was also adapted. One question was added in the flipped classroom block, referring the videos included in the material. In addition, the questions on the Aronson’s puzzle were replaced by some open questions, considering the application of the technique was no longer mandatory. Furthermore, several questions were incorporated to the questionnaire to relate the application of the techniques to the emergency experienced.

Finally, we can see the rubric used in the two studies in Figure 2. It is divided into several parts: structure of the web, different HTML elements and their decoration with CSS and some questions related to the evaluation form, including those of correctness and validation.

First of all, Table 2 offers the scores obtained in the global work by the eigth groups in the 18/19 academic year and by the eleven groups in the 19/20 academic year, using a range from zero to ten. Table 2 also offers the average of both courses.

Average mark obtained was 8.4 points in the course 18/19. All groups exceeded the minimum mark (5 points), with seven of the eight groups above the minimum notable (7 points), three out of eight reached the outstanding (at least 9 points) and two groups obtained the maximum score.

Average dropped to 7.8 points in the course 19/20. The number of groups was higher, because more flexibility was allowed in the composition of the groups. It was also important that there was maximum balance possible in the groups, with at least one person with certain background in programming. Nevertheless, size of the groups was more variable, even allowing to some people to work on their own. In fact, the lowest mark was given to a single person group, but all groups except the aforementioned individual exceeded the minimum mark, nine groups reached at least 70%, five groups got a at least 90% and we had two maximum marks.

Group	Course 18/19	Course 19/20
1	6.7	3.7
2	7.3	5.3
3	7.5	5.5
4	8.1	7
5	8.5	7.7
6	9.2	8.4
7	10	9
8	10	9.4
9		9.7
10		10
11		10

Average	8.4	7.8

Secondly, Figure 3 presents the outcomes of the question regarding previous experience in the field, as declared by the participants.

The particular question was the following: What experience did you have in the design/programming of web pages? (From 1 –null experience- to 5 –full experience-). The average of the students in the 18/19 course (blue in the picture) reached a mark of 2.8 points, while the average of the students in the 19/20 course (red in the picture) were of 1.4 points.

Figure 4 shows the results of the questionnaire referring to the flipped classroom methodology in the course 18/19.

Question regarding the links provided reached an average of 4.3 points; that related to code provided, 3.9; the statement, 4.3; the evaluation heading, 4.4; the work in the classroom, 4.0; and finally, the resolution of doubts, 4.3. Average of all questions related to flipped classroom was 4.2.

Figure 5 shows the results of the same questions, and the one related to the videos provided, in the course 19/20.

In this occasion, question regarding the links reached an average of 3.5 points; the code, 3.3; the statement, 3.5; the evaluation heading, 3.4; the work at home, 4.0; the resolution of doubts, 3.8 and the videos, 3.7. Average of all questions related to flipped classroom was this time 3.6.

Open answers were quite similar in both years: they stressed the convenience of the technique, although there were some cases who indicated that it would be good to have more previous training. In the second year, some others indicated that it would be better for the technique if the interaction with the teacher had could be face to face.

Figure 6 provides the results of the questionnaire related to Aronson's cooperative puzzle methodology during the course 18/19.

Question regarding the division into roles obtained an average of 4.2 points; the one related to the meeting of experts, 3.6; the one related to the possibility of interacting with other groups, 4.0. Average of all the questions related to the Aronson puzzle was 3.9.

Additionally, answers related to Aronson's puzzle showed their general agreement, but in some cases, students demanded some more sessions in order to improve their status as experts.

About the course 19/20, Aronson’s puzzle was the technique that suffered more changes. As we mentioned before, mandatory online classes were combined with the difficulty of the situation, including important changes in work, also in childcare and perhaps in close relationships with sick people. All these elements conducted us to give more flexibility to the way of working, with it is not compatible with the establishment of rigid roles. However, technique was explained, and the use of roles was suggested. At the end of the experience, students answered some open questions about the technique, and, in most cases, work was divided more or less as suggested. However, it was also common that the most important workload fell on the more experienced ones. On the other hand, interaction between the groups was almost non-existent, being replaced by direct contact with the teacher. Finally, students generally agreed with the convenience of the technique in a work like this, although they said that maybe it was not the best moment to apply it.

Figure 7 shows the results of the questionnaire relating to the general satisfaction perceived during the course 18/19.

Question about the satisfaction with the learning acquired reached an average of 3.7 points; that related to the satisfaction with the personal work carried out, 3.8; that related to the division of work, 4.2; the general rating of the flipped classroom methodology reached 4.0 and the general rating of Aronson's puzzle methodology, 3.9. Average of all questions related to general satisfaction was 3.9.

Moreover, Figure 8 shows the results of the questionnaire relating with the general satisfaction perceived during course 19/20.

Satisfaction with the learning acquired reached 3.9 points; satisfaction with the work performed, 4.2; division of work, 4.0; general rating of the flipped classroom methodology, 3.4 and general rating of Aronson's puzzle methodology, 2.9. Average of all questions related to general satisfaction was 3.7 this time.

Finally, it should be noted that overall average score of the complete questionnaire was 4.3 the first year and 3.6 the second year. Table 3 provides these data, with details for each role played during the first course.

We have applied the inverted classroom methodology to a group of master's students who were expected to be able to construct web pages. This technique is well complemented by cooperative learning, so we have used for this purpose the Aronson’s puzzle methodology. This last methodology also takes advantage, on the one hand, of the possibility of dividing the overall work into smaller parts, and on the other hand, allows forming groups in which there is a difference of previous level to use an adequate schedule by all members of the team. Rubrics address to guide the work towards required quality criteria.

The experiment has been carried out during two consecutive academic courses, although, in the second one, the conditions were different, and this affected, especially, to the application of the Aronson’s puzzle technique, as explained above.

Referring to Aronson's puzzle it should be noted that in our proposal we have given special importance to the Aronson mother groups as opposed to expert groups. That was precisely because of the intention to encourage interaction between people of different levels (so that the more advanced students take on coordination tasks and focus their learning on high-level aspects. Those less knowledgeable students can be guided, as opposite). In addition, the parts of the project are not independent, what means that the work with the mother group should be larger.

The results of the evaluation of the projects carried out were good or excellent. They all met a large part of the requirements under the heading. The lower marks were generally due to the delivery of a less complete work regarding the interaction part with the page, that it is the hardest part of the project. Consequently, this leads us to conclude that it may be interesting to reinforce the material related to the role associated with the JavaScript language.

We must highlight two facts related to these results: firstly, all members of all groups actively participated both in the process and in the presentation of the works in the first year. This seems relevant to us, because in groups where the difference in the previous level of the components is high, the imbalance in workload and exposure time is usually greater. More experienced people tend to take on a much greater load, which translates into less overall learning. We had the impression that the students got a good overview of the work, despite the division of roles. But it is true that this can be better checked, perhaps through an overall test. In relation to this, we have observed that, although the groups were as balanced as possible, the best scores were generally for the more experienced groups. However, the effort and achievements made by other groups with less previous experience is also noteworthy: we observed that these latter groups increased their knowledge significantly. Secondly, the group of the second year had less background than in the previous year, and they had to work in complicated conditions, but the results were also successful, including many excellent projects.

Speaking about the first year, in general, the questionnaire gave good results: firstly, the methodology to be applied was well understood. The few cases in which it was not so good we were able to correct it later with the group work, both on a daily level and in the meeting of experts or in the interactions with other groups. Secondly, the flipped classroom methodology was well received. In this section, the inclusion of the evaluation rubric, which was especially well appreciated, should be highlighted. The code provided was the only element in which the statistical mode does not reach the maximum score, which leads us to think that perhaps it should be improved in its documentation or complemented with some demonstration in class, or with some video, that explains it in more detail. Thirdly, Aronson's puzzle methodology received good reviews, although it was more difficult to understand exactly what was expected of each role. To this end, several explanatory documents were prepared, in addition to clarifying doubts in person. In any case, the dependence that exists between the different roles hinders the application of the technique, although it possibly also enriches it. On the other hand, the meeting of experts, which in general was received positively, also obtained some criticisms: it generated certain doubts at the start, although the students had a support document in which, for each role, what was expected from the meeting. In any case, it is possible that the meeting should have been held later or repeated in some additional session -also demanded by some students. Fourthly, the general satisfaction of the students coincides with the breakdown made in the previous points. The average scores are high, but it is advisable to review the low perception of the learning acquired by some students. In some cases, it could correspond to coordination roles, which do not associate the fact of directing the project with classical learning. Also, people with very little previous experience who have not contributed enough to group work, although we have the perception that this phenomenon, if any, has been small. On the other hand, it is noteworthy that the correct division of labor is one of the most highly valued aspects. In any case, to confirm these data it is necessary, in the future, to ask more directly about the workload of each student.

Finally, we will highlight the fact that the average score of the complete questionnaire reached a remarkable value (4.3), and that this is repeated for all roles, although the most experienced role gives an even higher score (4.6), while with the least experienced the opposite happens (4.1). In general terms, we still have an average student satisfied in the latter case, but it may be advisable to reinforce the elements that encourage participation and interest in this profile.

Results of the second year were also good, but the marks decreased. Students commented that the work became more difficult, and this could have influenced their responses. Regarding the flipped classroom, the general opinion gathered in the open answers was quite good, but some of the students missed more personal attention, although they did not claim it during the course. But it is true that, although all the doubts were quickly addressed, we prioritized their own time management, with few undemanding interventions. About the general satisfaction, it is remarkable that the marks referred to learning acquired and with the individual work carried out reached remarkable scores, even higher than the last year. They are very significant questions, which makes us optimistic about the appropriateness of applying these techniques, despite the circumstances. In addition, the question related to the Aronson’s puzzle technique obtained a quite low mark. This is logical, having into account that the technique was applied in very loose terms. With a view to possible similar situations in the future, or simply for distance learning, we believe it could be useful to incorporate into the methodology the systematic use of collaborative tools to build code or share documents.

As general conclusions, the proposed methodology has been satisfactory, and particularly appropriate in contexts where, on the one hand, there is some prior knowledge of the subject matter to be worked on, and on the other hand, there is some difference of prior level between the participants. The technique can be also applied in an online context, although it is necessary to increase personalized attention to ensure effective learning. Anyway, the foundations have been laid for experimentation with a control group to reinforce the results in a more appropriate context (larger number of groups and students).

On the other hand, in addition to the well-known advantages that the proposed techniques provide, it is of special interest to involve future teachers directly in this type of methodology, due to the multiplicative effect that the fact that they know them can have.

Role played	Average score 18/19	Average score 19/20
HTML	4.1	N/A
CSS	4.4	N/A
JavaScript	4.3	N/A
Coordinator	4.6	N/A
GLOBAL	4.3	3.6

This work has been financed with the research projects iProg (TIN2015-66731-C2-1-R) of the Spanish Ministry of Economy and Competitiveness and e-Madrid-CM (S2018/TCS-4307) of the Autonomous Region of Madrid. The e-Madrid-CM project is also financed by the ESF and ERDF structural funds.

Aronson, E., & Patnoe, S. (1997). The jigsaw classroom: Building cooperation in the classroom (2nd ed.). Addison Wesley Longman.

Atkinson, D., & Lim, S. L. (2013). Improving assessment processes in higher education: Student and teacher perceptions of the effectiveness of a rubric embedded in a LMS. Australasian Journal of Educational Technology, 29(5), 651-666. https://doi.org/10.14742/ajet.526

Fortanet, C., Díaz, C. G., Pastor, E. M., & Ramón, J. L. (2013). Aprendizaje cooperativo y flipped classroom. Ensayos y resultados de la metodología docente. In XI Jornadas de Redes de Investigación en Docencia Universitaria: Retos de futuro en la enseñanza superior: Docencia e investigación para alcanzar la excelencia académica (1151-1162). Instituto de Ciencias de la Educación. http://rua.ua.es/dspace/handle/10045/43329

Martínez, J., & Gómez, F. (2010). La técnica puzzle de Aronson: descripción y desarrollo. 25 Años de Integración Escolar en España: Tecnología e Inclusión en el ámbito educativo, laboral y comunitario. In P. Arnaiz, M. D. Hurtado, & F. J. Soto (Eds.), Consejería de Educación, Formación y Empleo de la Comunidad de Murcia.

Mbacho, N. W., & MWebi D.R. (2018). Jigsaw cooperative learning strategy and students’ self-concept in mathematics school of education, Laikipia University, Kenya. International Journal of Social Science and Economic Research I, 1(9).

Sein-Echaluce, M. L., Fidalgo-Blanco, Á., & García-Peñalvo, F. J. (2015). Metodología de enseñanza inversa apoyada en b-learning y gestión del conocimiento. In Á. Fidalgo Blanco, M. L. Sein-Echaluce, & F. J. García-Peñalvo (Eds.), La Sociedad del Aprendizaje. Actas del III Congreso Internacional sobre Aprendizaje, Innovación y Competitividad. CINAIC 2015 (14-16 de Octubre de 2015, Madrid, España) (pp. 464-468). Fundación General de la Universidad Politécnica de Madrid.

Urquiza, J. (2015). Estudio preliminar del uso de la clase invertida en la asignatura de Procesadores de Lenguajes. In M. Rubio Sánchez, J. Urquiza Fuentes, & C. Cáceres Taladriz (Eds.), Actas de las VI Jornadas en Innovación y TIC Educativas – JITICE 2015 (pp. 37-40). http://hdl.handle.net/10115/13700