Judging in FIRST Tech Challenge

[EddieD]

JoAnn,

While I think "staged" presentations are a great start and probably low hanging fruit, I would encourage FRIST to continue to improve in this area.

In my opinion there is no substitute for real feedback or at a minimum being able to see what other teams are doing. As it stands right now our students are putting in hundreds of hours to make sure their ideas and vision are conveyed to the judges. At the end of the day teams could be knocked out of contention for a WIDE variety of reasons. How is it reasonable to expect mentors or students to analyze how successful they were without some sort of feedback?

I am not suggesting this is an easy issue to solve. The judges are not always trained/willing to provide good feedback, there is very limited time to review and accomplish the tasks the judges are already tasked with, etc. However, the latest change seems to go in the wrong direction. At least last year we had something to work with. Now there is just a feeling that the Judges awards are more like an Art or Music competition where the judges choose their preference.

One other suggestion I have that might be low hanging fruit would be to start recording and capturing the winning presentations/notebooks at worlds. If at the end of each season you could point to that material and say that was a great example of excellent engineering, outreach, etc. I think it would go a long way towards teams feeling they know what is expected.

Regards,

Eddie

[korimako]

JoAnn,

Would FIRST (FTC) consider a simple (public) survey of Mentors/Coaches (and or students) about the feedback/self evaluation feelings? The representation here in the forums is a very small (likely biased) selection of coaches/mentors. Even if FIRST has no wishes to address/change the issue, the survey would be a great baseline to see if the self evaluation idea has gained/lost ground in the future. It would also be a great opportunity to solicit input on coach involvement ideas at the same time for the same reasons. Those seem to be the most mentioned issues about improvement.

[FTC8811]

As a rookie coach I was frustrated by the lack of information and transparency across all aspects of FIRST FTC. It does seem as if those "in the know" do not fully appreciate how much there is to know and how hard it is to find out the things we should all know in order to be successful coaches and mentors.

Having now coached FTC teams and been a judge at events, I have a better understanding of how the process works and how to prepare my teams for their competitions. I have become one of those that are "in the know" and much of what happens makes sense to me now and I would not change much about the competition process or the qualifying process. I do think FIRST needs to release more of their Judge Advisor and other event training materials to head coaches.

I do like how the advancement criteria are set up. The balance is important. Teams who cannot afford the latest and greatest new sensors and customized parts, those who cannot afford machine tools... they can compete and win with their minds, their hearts, and their outreach efforts. This is vitally important to urban and low resource environments. There are potential issues with alliance partners and selection, and there are areas to improve with judging feedback.

A lot of the mystery went away for me when I volunteered to be a judge at a tournament. What I learned is that there are internal processes and the whole judging thing is not as arbitrary as it might appear to the outside. FIRST is caught between asking more of the judges and not putting so much of a burden on the judges that more are required, training is required to be a volunteer judge, or more time is needed in an already long day. I see the problems from both sides. I think a feedback form could be helpful but it could become yet another checklist of "do more of this, less of that" I'm not sure the teams don't already know if they are honest with themselves.

As a judge I see teams at competitions that are totally obsessed with their robot and "winning". I doubt they have even heard the words Gracious Professionalism. Sadly, I also see robots being built and tuned by adults (even in the pits at Super Regionals!) while the kids are standing around waiting to play with their "video game" on wheels. In my mind, they are not benefiting from the entire FIRST experience. Changing this culture requires leadership from the adults. I have often heard these same adults complaining (loudly) about how something was unfair or needed to change when their teams did not advance. Judges circulate during the competition phase. We see things and make note. Its not all quantifiable "feedback" and I'm not sure quite honestly how I would provide feedback to some of these teams (and their adults) had I been required to do so.

[FTC3491]

Teams need help to improve

Most teams aren't looking for detailed feedback on how to win awards. Teams, mentors and coaches work hard to compete in FTC. They would like to know how to be a better team.

With the robot game, it's easy for teams to identify what's working or what should be fixed. But it is very challenging for most teams to identify other issues they should be working on.

The self reflection form is a place to start, but it's often hard for a team to identify their own weaknesses without some direction on where to look. Could the judges provide feedback that identified three areas the team could work on to improve?

It could be hand written on sticky notes that are put inside the front cover of the Engineering Notebook. The notes could be written at the end of the interview and not be part of the final judging decision or process. The notes don't need to be long or detailed.

Or what about using the Self Reflection form and just highlighting a few of the items the team should think about?

Teams could use that information either to identify areas where they could improve, or if they feel they are already strong in that area, they could work on how to present that information in the interview.

[FTC10683]

This just happened to my team and they are devastated. We are a rookie team this year and scored the most RP and were first or second in our league for the last 2 league matches and then all day at the League championship. We had a great alliance partner (the #1 team) for the alliance matches and there was a communication problem during the final match and 3/4 robots disconnected and couldn't reconnect - both us and our alliance partner and one of the other teams. We lost that match and were out and received no awards and therefore our whole season was for nothing. We are advancing into the Regional, but my students are discouraged from what happened at the championship. As a rookie team looking at the set up this year, the focus was on league play and scoring well at the league level in order to give your team a chance to advance, but that didn't seem to be the case. Additionally we have NO feedback on our notebook or judging performance so we don't know how to improve in those areas so that we are able to do better at the regional competition. Any advice?

[JoAnn]

This just happened to my team and they are devastated. We are a rookie team this year and scored the most RP and were first or second in our league for the last 2 league matches and then all day at the League championship. We had a great alliance partner (the #1 team) for the alliance matches and there was a communication problem during the final match and 3/4 robots disconnected and couldn't reconnect - both us and our alliance partner and one of the other teams. We lost that match and were out and received no awards and therefore our whole season was for nothing. We are advancing into the Regional, but my students are discouraged from what happened at the championship. As a rookie team looking at the set up this year, the focus was on league play and scoring well at the league level in order to give your team a chance to advance, but that didn't seem to be the case. Additionally we have NO feedback on our notebook or judging performance so we don't know how to improve in those areas so that we are able to do better at the regional competition. Any advice?

Hello FTC10683,

It sounds to me as though your team had a very successful season. For a rookie team to do so well in the competition is quite an accomplishment, and I hope that you and the students on your team are feeling very proud of how much you learned and achieved this season. With more than 4000 teams participating in the program, it's helpful to remember that most teams don't win an award, and most teams don't win at every competition they attend. There's a lot to be learned from not winning, and sometimes those lessons can be life changing. There's an awesome Tedx Talk by Jia Jiang: Lessons Learned from Rejection that might be worth watching.

The FTC Mentor Guide offers some good advice about how to be more successful in Judging. Making sure that your team has done their very best to include every Judged Award requirement is the best place to start. It's also helpful for the team to practice their own Elevator Pitch, practice their presentation (for each other, for other adults, for other teams, for other teachers). Presentation counts - a lot.

I hope that you and your students take some time to celebrate your fantastic accomplishments!

Joann

[JoAnn]

Most teams aren't looking for detailed feedback on how to win awards. Teams, mentors and coaches work hard to compete in FTC. They would like to know how to be a better team.

With the robot game, it's easy for teams to identify what's working or what should be fixed. But it is very challenging for most teams to identify other issues they should be working on.

The self reflection form is a place to start, but it's often hard for a team to identify their own weaknesses without some direction on where to look. Could the judges provide feedback that identified three areas the team could work on to improve?

It could be hand written on sticky notes that are put inside the front cover of the Engineering Notebook. The notes could be written at the end of the interview and not be part of the final judging decision or process. The notes don't need to be long or detailed.

Or what about using the Self Reflection form and just highlighting a few of the items the team should think about?

Teams could use that information either to identify areas where they could improve, or if they feel they are already strong in that area, they could work on how to present that information in the interview.

Hello FTC3491,

Would your team consider working with other adults to rehearse their presentation there? By the time a team gets to an event, it's often too late for them to make any dramatic changes in their presentation, or in the content or quality of their Engineering Notebook. Practicing ahead of time would give the students the opportunity to work out the bugs in their presentation, in time to make a difference. The students could even practice with one another, taking turns having one student fill out the self reflection sheet while the other students rehearse their presentation.

With different judges in different regions all seeing different groups of students, it would be really difficult for every team to have their feedback expectations met in any sort of consistent way - and we really are aiming for consistency, with everything in the program.

JoAnn

[FTC4318]

It seems to me and others in the community that this whole "self reflection" stance is based on not wanting to put judges in the position of explaining their decisions rather than it being in the best interest of the students. What educational model has "self reflection" as the singular mechanism for improvement? What job? What really are we preparing them for with this approach? FLL provides feedback based on rubrics to teams. Florida FTC does use scored rubrics for judging and gives report cards to their teams on their Engineering Notebooks: http://ftc.flfirst.org/Docs/2015-201...wards_V3.0.pdf

Why not adopt their model for all of FTC? They have done a terrific job of standardizing what is by its nature a subjective process. This is a STEM program - let's use the math to make it as fair and as transparent as we can.

A way to avoid questioning/confrontation about why one team got this award over another is to simply to stand by your judges and say FTC does not take awards away from teams once they are given. That would not be GP. Do invite feedback on how to make the judging process better but make it clear that there is no "replay" in judging. End of story.

[FTC6389]

Hi JoAnn,

Our team wanted to add something to this discussion. It seems to us that the limericks the judges are required to make up are in lieu of telling the audience why a team was selected to win an award. Telling the audience why a team was selected would "provide feedback" only to teams that receive awards. We feel that what FIRST is missing is that the award ceremony is the perfect opportunity, time, and place to provide feedback. By telling everyone exactly why the judges chose a specific team to win the Inspire award will help other teams understand what it takes to bring their own team to that level. Do you know that many teams leave before the awards ceremony, because they don't feel like they are going to win anything, and therefore they feel there is no value in them staying? By not telling your audience why a team was selected by the judges, you are doing the exact opposite of what you claim you want to do. Not telling the world why a team was selected creates wonder and doubt about the entire judging process. We understand that there will always be a subjective aspect to judging, just as their is in diving or gymnastics or any other event in which judges score competitors. The complete lack of judging transparency FIRST has chosen is not in the best interest of the students involved in our opinion, and as you have said many times, that's what FIRST is all about, right? How are teams supposed to understand exactly what team X did during their season to receive an award? Many teams don't have social media on which they share all their designs, outreach activities, engineering notebook, etc. If FIRST is truly interested in helping every team get better, it seems to our team that the path forward is obvious.

Respectfully,

- The Lazybotts

[LukeTheCoder64]

I'd like to re-open this discussion after the publishment of Game Manual Pt 1, which states....

FIRST Tech Challenge does not permit feedback provided to Teams during or after their Interview has taken
place at official Tournaments. FIRST Tech Challenge judging is a subjective process; the goal is to prepare
student Team members with real life Interview skills, and to continue to build upon those skills from Event to
Event.
FIRST encourages Teams to utilize the Self-Reflection Sheet to evaluate themselves through the Interview.
This sheet is accessible online. Teams should not ask the Judges for feedback after the interview is complete.
An essential aspect of FIRST Tech Challenge Judging is the subjectivity, and that FIRST Tech Challenge
encourages students to learn how to self-evaluate. Although it may be that Teams are discouraged by this,
learning this process is an invaluable life skill.

So the argument presented is: "No, we can't provide feedback, because you need to learn to 'self-reflect'."

Luke

[Alec]

Judges are volunteers and volunteers are not all created equal. There would be liability issues (among other ramifications) if FTC judges were allowed to give feedback.

Judges do give a lot of indirect feedback. It is a skill for students to learn to pick up on indirect feedback.

Moreover, mentors can glean valuable feedback by judging other teams -- but please don't volunteer to judge primarily for this reason, or risk being less than equal to good volunteers.

[FTC Ringer]

Judges are volunteers and volunteers are not all created equal. There would be liability issues (among other ramifications) if FTC judges were allowed to give feedback.

Judges do give a lot of indirect feedback. It is a skill for students to learn to pick up on indirect feedback.

What liability issues are those? Liable for what?
And how do judges give indirect feedback? Whether they give you an award or not?

[Alec]

What liability issues are those? Liable for what?
And how do judges give indirect feedback? Whether they give you an award or not?

Well, as I said, volunteers are not all created equal. Even great volunteers can have a bad day. Not permitting feedback from judges protects volunteers and FIRST against wrong or inappropriate feedback to minors.

[LukeTheCoder64]

Well, as I said, volunteers are not all created equal. Even great volunteers can have a bad day. Not permitting feedback from judges protects volunteers and FIRST against wrong or inappropriate feedback to minors.

If a single judge gives inappropriate feedback to a participant, we should be questioning why that judge was selected, not why feedback from judges is wrong.

[Alec]

Another ramification of volunteer judges is that you are bound to get incomplete and/or inconsistent feedback. Sometimes feedback can do more harm than good. Plus the quality and accuracy of the feedback will vary from region to region, giving teams in some regions a competitive advantage over teams in other regoins. All things considered, it seems necessary not to permit feedback from FTC judges.