Project Overview

Dragon Architect is an educational game, written in HTML, CSS, and JavaScript, created at the University of Washington. It was used to investigate key open problems in the design of games that teach computational thinking skills (strategies that are used in computer science). To teach these concepts, Dragon Architect uses block based programming which allows students to drag and drop blocks to create a program to move a dragon around a virtual world and build things.

Games like Dragon Architect can actually increase representation in computer science. Research has shown that many students, especially non-male identifying, lack confidence that they have the ability to learn computer science, which can prevent them from ever trying. By playing educational games, students are able to build confidence in their own skills(Hoegh & Moskal, 2009). Playing a game can also change a student’s perspective on the field of computer science in general. For example, computer science is perceived as a field more appropriate for men, discouraging non-male identifying students from participating, but a game that is specifically created to appeal to all genders can help change that misconception. Finally, a game like Dragon Architect can increase overall interest in computer science. One hope for a game like Dragon Architect is to provide students an opportunity to discover their interest in computer science.

The original game was created for middle school students. Our project was to redesign the game for college students, specifically students who have never learned computer science or who are just starting out.

We transformed the front end into a modern and engaging user experience. We also added a Python display, representing the block code, to better connect Dragon Architect with the Carleton Intro Computer Science course. The original game taught students about loops and functions; our final major addition to the game was adding four new levels to teach variables.

User Study Results

To evaluate whether the changes we made to the game accomplished our goals of altering the game to make it effective for use with a college student population, we conducted a formal user study. Our user study had a single-group pretest-posttest design, which means that each participant was tested twice, once before and once after a treatment. In our case, this took the form of participants completing a “before” survey, playing the game for 30 minutes, then completing an identical “after” survey.

We recruited 30 Carleton students from class years 2024 through 2020 who had never taken a computer science class and had little to no experience with programming to participate in our study. The surveys that were completed by participants before and after playing were identical, and each participant was only compared to their past self. They had two sections.

The first contained questions that assessed their attitudes towards computer science. This section contained 11 Likert-Scale questions that ask participants how much they agree with a given statement about computer science, computing, or computational thinking. Participants are given four possible responses that range from strongly disagree to strongly agree. These questions were drawn from two validated surveys developed by Computer Science Education Researchers. [Hoegh & Moskal, 2009; Cetin & Ozden, 2015]

The second section of our survey contained four questions that assessed the participants’ understanding of a few computational thinking skills, including loops, variables, and procedures. These questions present a snippet of python code and ask the participant to choose the image that displays what the cubes and dragon would look like after the code is run.

To analyze any change in the participants’ attitudes towards CS, we calculated an average attitude score for each student from before and after playing the game. We did so by turning their responses into integer values, flipping the scale for statements that were negative towards computer science, then averaging across the 11 questions. For example, a response of “strongly disagree” to a positive statement about computer science was coded as 0 while a “strongly agree” response to a positive statement about computer science was coded as 3. Therefore, possible attitude scores ranged from 0 to 3, with 3 indicating positive attitudes towards computer science.

Before playing the game, participants had an average attitude score of 1.4. After playing the game, participants had an average attitude score of 1.8. A t-test of dependent means revealed that this is a significant difference in attitude score from before to after playing. The effect size related to this change is 1.3, so we can conclude that this effect is also large.



To analyze any change in the participants’ computational thinking skills, we calculated the percent of skills questions that each student answered correctly before and after playing the game. Before playing the game, participants answered an average of 14% questions correctly, and about half of the participants answered none of the questions correctly before playing the game.

After playing the game, participants answered 38% of the questions correctly, on average. A t-test of dependent means revealed that this is a significant difference from before to after playing. The effect size related to this change is 0.86. For reference, the cutoff for a small effect size is 0.2, and effect sizes larger than 0.8 are seen as large.



Overall, these are exciting preliminary results that suggest that our updated version of the game effectively introduces Carleton students to computational thinking and improves Carleton students’ attitudes towards Computer Science.

Resources

alternative text
alternative text
alternative text
alternative text
alternative text

ABOUT US

Catalina
Alvarez-Ruiz

Catalina (she/her) is a computer science major and cognitive science minor from Massachusetts. She enjoys creative writing, spending time outdoors, and playing soccer. After Carleton, she plans to work as a software engineer at Optum in Boston.

Kate
Grossman

Kate (she/her) is a computer science major, math minor, and women's and gender studies minor from Evanston, IL. At Carleton, she is a captain of the varsity swim team. Next year, she plans to move to California as a part of SAP's Silicon Valley Next Talent program as a software developer.

Rie
Kurita

Rie (she/her) is a computer science and economics major, and music minor from Tokyo, Japan. She enjoys acting in/watching musical theaters and traveling aborad. After Carleton, she plans to work as a software engineer for Kohl's in San Jose, California.

Ellie
Mamantov

Ellie (she/her) is a computer science and psychology major from Knoxville, TN. In her free time, she enjoys playing ultimate frisbee and cooking. After Carleton, she plans to attend graduate school to pursue her research interests in human-robot interaction.

Starr
Wang

Starr (she/her) is a computer science major from Guangdong, China. In her free time, she loves outdoor activities, traveling and photography. You can check her photography work here. After Carleton, she plans to move to Seattle and work as a software engineer at Microsoft.