This article documents my journey as a User Experience Designer at Nanyang Technological University for one and a half years.
To comply with my non-disclosure agreement, I have omitted and obfuscated confidential information in this case study. All information in this case study is my own and does not necessarily reflect the views of the University.
THE BACKGROUND
Nanyang Technological University has a reputation for being a tough school. It demands more of its students than a lot of other institutions do.
Increasingly, the university have been receiving feedback that students aren’t receiving enough formative feedback from their professors. Overtime, students are not able to discern and question their arguments.
In the defence of the professors, their semestral time-table are usually packed with many lectures to conduct, leaving very little room to offer help to students. Their priority, however, is to write research papers with the intention of driving growth for the university.
TODAY’S SOLUTION
The university recognise this issue and set up peer tutoring centers across faculties where peer tutors are paid to help students with their studies
Problems identified through quantitative research:
- Lengthy on-boarding Process
In order for a first-time user to book a peer-tutoring session, the student has to perform 17 actions on their website. By using Hotjar, we have found out that 16 out of 40 users dropped out while attempting the on-boarding process. A time-study was also conducted, and it took an average user 7.83minutes to complete the process. - Students are not aware about peer-tutoring service.
NTU’s main channel of promoting peer-tutoring service is via E-mails. It achieved an open rate of 18%. A random sample survey within a faculty found that 70% of 30 students are not aware of this service. - Unable to meet the growing demand for peer tutoring services
Research data have shown that there’s a 35% onward increase in demand for peer tutoring services semester after semester.
For confidentiality reasons, I have omitted the actual values for these metrics.
NTU’S VISION
The vision, as we articulate it today for NTU is to be technologically-enabled ready(TEL) by 2020. That means to say; it is determined to digitalise 50% of all courses across all faculty by the year 2020. To empower learners to be critical thinkers, all classroom learning in future will be more dialogue driven. One that facilitate the asking of difficult questions and to challenge the cognitive arguments raised by learners.
MENTOR research project
Singapore Millennium Foundation(SMF) Grant and the Cradle@NTU have funded Dr Kenneth Ong, the assistant chair of School of Humanities and Social Science and his co principle investigators(PI) over $700,000 to introduce a digital solution to the problems faced in existing peer tutoring centers, and be a part of facilitating the University vision of being TEL-ready by 2020.
INITIAL PROJECT GOALS
Analyse the current processes of existing peer tutor programs from a web application and migrate them into a new digital platform; a mobile application called MENTOR.
Some of the tech solutions proposed by the stakeholders:
- Proximity-based peer tutoring service.
- Web-based remote tutoring service with screen sharing capability
- Personalized matchmaking of tutors based on tutees’ preference
MY ROLE
Between September 2016 to March 2018, I led the research and collaborated with two other UX designers and an external agency we’ve hired to develop the mobile application.
THE KICKOFF
At the outset of the project, we weren’t given a clear direction on how we should approach the project. This empower us to diverge and understand the problem better. The obvious starting point for us was to understand the existing state of all peer tutoring centers in NTU.
Most popular peer tutoring programs(PTP) identified
We dug out all the past quantitative data collected from research reports and analyse them.
Findings:
- The Language and Communication Cube(LCC) is the most popular peer tutoring program(PTP) in NTU and it accounts for 55% of all peer-tutoring services.
- Over 77% of the demand for peer tutoring program comes from the College of Engineering(COE).
For confidentiality reasons, I have omitted the actual values for the data collected.
Probe further…
The engineering faculty have the largest student population in the whole of NTU and the two disciplines that engages most in PTP are students from the College of Mechatronics and Aerospace(MAE) and Electrical and Electronics Engineering(EEE)
Being very clear of who we are solving for, we centred our qualitative research around students from the EEE/MAE discipline in the college of engineering.
Qualitative Research
Some of the questions we’ve put out to our interviewees
1. Can you walk me through how do you use the roster management system on a daily basis?
2. What is the peer tutoring program trying to do?
3. How do you measure the success of the Peer tutoring program?
“Questions give data meaning” — Mark Ong
Essences Gathered
#Insight 1 — Inefficient roster management system
During our interview with Roger, tutor manager of LCC and a professor, we learned that he spends an average of two hours daily maintaining excel sheets and the roster management system.
#Insight 2 — 47% of engineering students are foreigners
For confidentiality reasons, I have omitted the actual values of the student demographics.
College of Engineering: 53% are locals while 47% are foreign students predominantly from China and India.
Students from the College of Engineering are graded on their written and oral communication in English. Hence the high demand for peer tutoring in Language and Communicate Cube (LCC).
On the other hand, the local engineering students have their own set of challenges.
Challenge #1: 2nd year direct admission students who have completed their national service struggles with the advanced engineering modules taught. The assumption is made that these students should already understand the basic fundamentals. As such, many students were not able to keep up with the contents delivered.
Challenge #2: With so much contents packed in one module, coupled with a tight timeline, professors have to rush through them, compromising on the quality of the lectures.
“Duration of each tutorial lesson = 1 hour. Lecture = 2hours. It’s unrealistic for professors to cover each topic & question in-depth” — Sally (MAE Tutee)
Mechatronics and Aerospace(MAE) and Electrical and Electronics Engineering(EEE) peer tutoring programs are deployed to fill this gap.
#Insight 3 — Conceptual understanding takes a second seat
By seating in for a few of 1 to 1 peer-tutoring sessions, we made an observation that all of them came in with an expectation of getting a model answer without internalising the logic. This seems obvious at first, but we probe further, we discovered that there’s an underlying reason that no one talks about.
Honest students who answer questions in their own words often get penalised because keywords were not used. On the other hand, those who memorised and answer word for word gets full mark.
This creates an insidious culture of students who chase the marking rubrics without understanding the application of it.
“Students who are honest starts to alter their study habits and behaviour to chase grades” — Zhen Ming Hao, tutor from EEE.
#Insight 4 — Flawed system creates opportunity for fraud
Peer tutoring services in NTU are fully-funded. Students do not have to pay for them. Because you are allowed to pick your tutors, a few tutors were caught colluding with their friends to make bookings with them just so they can earn extra allowance.
#Insight 5 — Peer tutors are the cornerstone
Because of how quickly tutors get booked on a monthly basis, tutor managers who run the centers make the assumption that there is a lack of peer tutors in the center. But after peeling into the data, we’ve discovered that there is a segment of tutors that are usually booked weeks ahead. Investigating further allow us to understand that the problem doesn’t lie with the lack of tutors, but the QUALITY tutors that can breakdown abstract concepts in the lowest denominator so that everybody can feel understood.
Design Studio with our Stakeholders
Just before diving straight into designing and prototyping a solution, we decided to rope our stakeholders in a design studio. The objective here was to share early insights found in the field, recap the viability of the solutions proposed by our stakeholders and involve them in a design thinking process aimed at translating insights found from our qualitative research into potential solutions
A RECAP OF THE SOLUTION PROPOSED BY THE STAKEHOLDERS
In engaging the stakeholders in a design studio, we were able to encourage a collective effort in identifying the limitation of their proposed solutions.
Unearthing the real problem..
The goal of deploying MENTOR app as a mean to facilitate peer tutoring services only scratches the surface of the problem. A deeper problem, however, is the way how the university tries to strike a balance between designing a world class elaborate curriculum and publishing good research papers done by professors.
- Designing a world class elaborate curriculum:
In an effort to teach so much in a short span of time, professors have to bullet through their presentation and prioritise the contents that are the most important. Because they could not explore any topics in great depths, students lack conceptual understanding to form the basis of a strong argument. Students who have not developed an inquisitive nature to do independent research will also find themselves in the ghetto.
2. Publishing good research papers:
An important metric that determines a university ranking in an international arena is highly dependent on the number of citations that points back at the research papers done by university authors.
In order for professors to stay longevity on their jobs and be considered for promotions, they have to write research papers on a scheduled deadline. This means that any free time outside the professors teaching schedule have to be devoted to the art of their research.
When the priority is shifted from the need to nurture a student’s quality of learning to the school’s reputation, the student suffers.
Overtime, students are more used to swallowing information without even attempting to understand it and critically evaluate it over the years of their studies in Nanyang Technological University.
At the end of the day, we must never forget that the very existence of a university was built not for the bureaucracy, but at the heart of the students. Universities have the greatest responsibility to influence a student’s future as they march down their final rite of passage in their student life.
MISSION
It was very important to us that we understood this as this has form the basis of our underlying mission. To be the bridge between the professors and students. More efforts will be put in place to train quality tutors so as to pull these weaker students back on course so that they have the confidence to keep up with their professors and peers. Thus encouraging their cognitive ability to think critically.
A CULTURE OF EXCELLENCE
Tutees may not all be able to be autonomous at the beginning. The goal of the tutoring is to gradually minimize the instruction and help tutees to become independent learners. Tutors are there to help tutees understand their depth of knowledge, recognize and challenge their thought processes, and find their own method of learning. In order for tutees to gain such learner autonomy, tutors need to listen actively and keep the communication tutee-centered. Tutors confirm by asking open-ended questions. Tutors encourage tutees to work independently using what they learned during the tutoring session.
We are establishing this culture as a means to strengthen NTU’s vision of being TEL-ready by 2020. As the university is moving into a new phase of remote learner centric, the aim of the peer tutoring programme is to empower weaker students with a renewed confidence on finding their most suitable pace of learning.
INTRODUCING THE MENTOR APP
Booking a session with a tutor has never been more straightforward. Getting help from a subject expert is just 4 steps.
- Select preferred time slot for consultations with peer tutors
- State purpose of booking and subject title
- Authenticate with your student email before receiving a booking confirmation
- Head down to tutor centre and check-in with receptionist on the day of your consultation
INTRODUCING IMPROVED ROSTER MANAGEMENT SYSTEM.
Tutor manager can now create, update, delete and manage schedules for tutor.
Peer tutors inputs availability.
Peer tutors would be notified by the tutor manager to lock down their availability for the week and are given specific time frame to fill up their availability.
Gone were the days where popular tutors clocks in more hours of tutoring sessions upon tutees’ request, leaving out the unpopular ones.
Every tutor will be given equal opportunity to have sessions with tutees in a week set by the tutor manager.
Tutor manager has the final say.
All the tutor manager have to do is to make the final confirmation and publish the roster. Notification will be sent to peer tutors and the availability of peer tutors would be reflected on the front-end of MENTORs’ mobile app.
OUTCOMES
With the new roster management system, say goodbye to long hours of administrative work. Tutor manager can place more focus on understanding the learning needs of tutees and work closely to improve the quality of teaching by peer tutors.
UX PROCESSES THAT DIRECT OUR DESIGNS.
1. Roster Management System
We’ve put together a fictional character that represents the needs and goals of our tutor manager to help guide our design direction.
Introducing Roger Winder, the tutor manager
Journey or Roger
We further our analysis by analysing the different touch-point of Roger Winder through the use of a journey map
By mapping out Roger’s journey; we were able to understand the context of his frustrations experienced at different touch points.
We’ve expanded our research by introducing a visual representation of the different systems he interact with on a daily basis when trying to roster and matchmake tutor-tutee.
RECOMMENDED SERVICE BLUEPRINT
2. MENTOR BOOKING APP.
During our ideation and prototyping phase, we referred intensively to the persona and storyboard we’ve put together.
Introducing Thomas, our tutee from School of Engineering
Thomas’ story
A storyboard was pieced together to help us visualise the problems better so that we can identify design opportunities
Rapid Prototyping
We translate our tutees’ needs and goals to user interface design. Here’s a user flow of MENTOR Booking app.
User flow of our High-fidelity mock ups.
Usability test
We’ve involved 5 usability test participants in our study. Here’s an example of a scenario we put out to them.
Scenario
“Just imagine you’re going through lecture notes and have trouble understanding the concept of a specific topic in Thermodynamic and require a tutor to teach you. You recalled a friend of yours shared with you previously about this MENTOR app and decided to check it out”
Task
1. Look at this app for 5 seconds. Then look away and answer this question (w/o peeking): How would you decide if you wanted to try it?
2. I want you to book a consultation with a tutor at 9am on 5th June 2017.
On the side…
My personal favourite question i love to put out to Usability test participants is this:
“How would you describe this product to a friend”
Key finding from our Usability test
1. 6 out of 10 participants prefer not to message the tutor to arrange for a consultation but rather have a feature that allow them to make bookings.
2. 8 out of 10 participants took longer than expected to finish their task. We’ve timed them to measure effectiveness and efficiency of booking a consultation.
After performing all the usability testing, we reiterated on our designs THE
PIVOT — Final Prototype
OUTCOMES
After reiterating on our designs, we are proud that we’ve managed to streamline 25 UI screens to 9 UI screens in the final prototype.
- Significantly reduce the cost of implementation (App development cost)
- Usability test participants took half the time required to accomplish their task during our user testing.
To top this off, our team have put together a heuristics of being a great tutor after synthesising the qualitative research data found from our interviews with students. Tutor manager could use this to run trainings or have 1–1 check-in sessions with peer tutors
OTHER RECOMMENDATIONS
TO SUM THIS UP
Modern society is becoming more complex, information is becoming available and changing more rapidly prompting students to constantly rethink, switch directions, and change problem-solving strategies. Thus, it is increasingly important to prompt reflective thinking during learning to help learners develop strategies to apply new knowledge to the complex situations in their day-to-day activities.
