Hey folks! I’m Jonah Kellman, and am the current testing manager for the IJP Level 5. I want to take some time to explain the decisions behind the advancement process and maintenance. Before we go too far, I want to mention that this process is always evolving. If you’re in the L5 track or working with an L5 candidate, and feel that there are improvements that can be made, please don’t hesitate to reach out to L5TestingManager@InternationalJudgeProgram.org
Description
Level Five Judges are regional and international leaders. An L5 is an expert in leadership, policy philosophy, investigations, event logistics, and community building. They are prepared to head-judge tournaments with a few hundred players, and several teams of judges, potentially with complex expectations from the tournament organizer, coverage, or other specialized tournament logistics. L5s guide tournament policy and define and innovate how tasks are done at large events. L5s are highly involved in their regional and the international community, mentoring judges for L3, L4, and L5 and participating in projects. Alone, an L5 is capable of identifying a new L1 with promise, mentoring them through L2, L3, and L4.
Level Five is the pinnacle of the judge program, and as such, expectations are high. We expect L5s to not only be capable of handling pretty much anything that comes at them during an event but to also be able to innovate and find improvements, particularly in their field of expertise.
Any L5 should be able to HJ a reasonably large tournament and handle complex rules and policy situations, but we also expect the ability to help refine policy, develop new standards for tournament operations, mentor the next generation of leaders or similar exceptional abilities.
Advancement
To be promoted to Level Five, a candidate must complete the following requirements. Unlike the processes for Levels 1-3, these requirements occur in a prescribed order.
Step 1: Experience as an L4
These requirements may be completed in any order.
Work at least six multi-day events, acting in any of the following roles at each of them (there is no time constraint on these events):
- Team lead five or more judges
- Shadowing, to provide feedback and evaluation, an L4 candidate in the L4 certification process
- Serve as head judge for a medium-sized competitive event
- Serve as a support judge, a head judge with a support judge, or one of multiple head judges for a single tournament.
One of the key roles of an L5 judge is that of leadership, and so we want you to demonstrate your ability to lead. While there isn’t a requirement to take on different types of roles, serving in only one of the listed roles for all six events may raise questions later during the advancement process.
Getting this practical experience leading allows you to learn from your mistakes, determine areas of strength and help refine your leadership style.
2. Choose three. You may choose the same mode more than once.
- Write a review as an L4, in the previous year
- Must be from events that the candidate and the other judge both worked
- Each review must contain detailed and actionable feedback.
- Participate as a panel member or practical evaluator for a Level Four process, in the previous year
- Perform a Level Three advancement interview, in the previous year. Regardless of the result, a review documenting the interview is required
- Actively participate in an IJP core project
- Create quality educational content (subject to approval)
The IJP is built on community participation – if high-level judges aren’t invested and involved in the continuation and development of other judges, it will falter.
Both the event and community requirements have significant overlap with the L4 maintenance requirements. A judge who has been level four for a few years should naturally meet these expectations relatively easily.
Step 2: Self Review
The candidate must submit a self-review indicating their preparedness for L5 according to the following requirements. The review will be assessed by an approved L5 panel lead, who may advance them to the L5 panel process, or return the application with feedback for the candidate.
The self-evaluation should cover all of the quality categories and include:
- Strengths and weaknesses
- Examples that support the candidate’s evaluation
- A demonstration of understanding the quality
- Rating from 1 (exceeds expectations) to 4 (deficient) compared to baseline L5
I could probably write a several page article on the importance of self-evaluation in general and the self-review in the advancement process in particular, but I’m going to try to keep it brief.
As mentioned in the L4 details, as you advance in the program, you will receive less feedback, and it becomes more important to be able to curate your own development. Beyond that, the self-evaluation sets an important baseline for judges who are evaluating you as part of the advancement process, including your recommenders, your panel and the testing manager.
The requirements for the self-review are designed to make sure that the self-review is functional for that purpose. The first three present a comprehensive argument for the candidate’s advancement, while the last makes sure that everyone is comparing the candidate to the same set of benchmarks and is clear in their evaluation.
Step 3: Recommendation Review
Candidates must also submit a recommendation from an L5 judge who was not their recommender for L4. This evaluation is a significant commitment of time and energy – to observe the candidate and then to provide them and their future panel with the feedback necessary for their growth and success. However, it’s an important part of the advancement process for a candidate to be able to find a mentor who is willing to put in the effort to help them succeed and who believes in them.
The recommendation should cover all of the quality categories and include:
- Strengths and weaknesses
- Examples that support the recommender’s evaluation
- Rating from 1(exceeds expectations) to 4 (deficient) compared to baseline L5
For qualities where the recommender gives a rating of 2 (meets expectations) or better, gives the same rating as the candidate, and feels like they do not have anything substantial to add to the candidate’s self-review, they do not need to include a written segment.
The recommendation is similarly critical to the advancement process. One of the most important things about the recommendation is not the written recommendation itself, but the development of a longer-term mentorship connection. Working with somebody who has been through a similar advancement process means that they can guide you around obstacles that the candidate doesn’t know exist, and having an ally at level five means that there is somebody who can argue for the candidate, and champion their cause. This can come in the form of helping the candidate find leadership opportunities, relaying critical feedback from other judges who don’t know them as well, or even prodding the testing manager to respond to an email that was sent weeks ago.
The recommendation itself shares the same requirements as the self-review, so that the candidate and future panelists can directly compare the two documents. To help mitigate the time spent on the written recommendation, the recommender does not need to thoroughly evaluate areas where they would just echo what the candidate already said. Including the rating is still important, and the recommender can still include further detail, especially if they feel it will help the candidate’s growth.
The recommendation is not only a tool in the advancement process, but it also serves as feedback and encouragement for the candidate.
The application, consisting of the checklist, self-review, and recommendation will be assessed by the L5 Testing Manager, who may advance them to the L5 panel process, or return the application to the candidate with feedback.
The L5 Testing Manager may, at their discretion, refer the decision to advance a candidate to a committee of 3 randomly chosen L5s volunteers to assist with the decision. In the case of a returned application that was not referred to a committee, the applicant may request their application be reviewed by such committee.
This step is important, so as to make sure that the candidate has the best chance of success. Having a low-quality self-review may hinder the candidate’s ability to demonstrate their strengths later in the evaluation process, while having low-quality self-review skills will surely hinder the future development of the candidate. Allowing a candidate to advance who is not ready, not only wastes the time of the panel but also of the candidates themselves.
Step 4: Exam and Panel Selection
After being approved for the L5 panel process:
- Once they are approved for the panel process, they may take the exam. They must complete the exam with a score of 80% or higher before a panel can occur. Unlike other IJP exams, this test is closed book. This exam will be proctored and timed.
- After being approved for the L5 panel process, the L5 Testing Manager will choose an L5 approved to lead L5 panels as the panel lead. The panel lead and coordinator then choose one additional panelist and up to one observer
The exam is a potential blocker. If the candidate fails their exam, they are not able to proceed in the process until they’ve demonstrated that they’ve studied and improved their knowledge, a cooldown has elapsed, and they’ve retaken and passed the exam.
The panel selection process is straightforward. Once a candidate has made it to this stage, they are asked about their availability at upcoming large events, and a solicitation is sent to L5 judges to see who is available at those events. Digital evaluation interviews are possible but are limited to when a candidate does not have any events for a significant period of time where a panel could assemble.
Step 5: Pre-Panel Interview and Practical
All candidates will be asked several basic questions that can help create a more developed picture of the candidate and allow for focusing the panel interview on areas of concern. The panel lead may choose to ask more questions during this pre-panel interview. The panel lead may lead the interview themselves, request that a panelist do so, or request that the L5 Testing Manager appoint a pre-panel interviewer.
The pre-panel interview is a long-form conversation about the candidate’s skills, knowledge, and opinions. Generally, the interview lead provides multiple open-ended questions for the candidate to respond to, and subsequently, the interviewer may ask for additional responses, pose additional questions, or ask for clarification. This is usually done via written responses to the interview questions, but the option to complete the interview via in-person chat or online interview is also available.
The contents of the pre-panel interview, notes from the interviewer, and an evaluation from the interviewer are provided to the panel lead and panelists.
The first batch of questions cover all aspects of judging, including diplomacy, self-evaluation, leadership styles, mentorship, policy philosophy, logistics and more. The candidate is given time to ponder these questions – they don’t necessarily have right or wrong answers, and so it’s up to the candidate to figure out how they want to approach them.
Follow-up questions can focus on areas that the panel lead deems important. It could be that they just have a few questions about a candidates strengths and then feel that they won’t need to evaluate it further because it is a clear strength, or it could be asking questions about areas of improvement, to develop a stronger line of inquiry for the final panel.
The assigned panel lead then requests a practical evaluation, focusing on areas of improvement highlighted in the candidate’s application and pre-panel interview. The assessment happens at a multi-day event where an evaluator and the candidate are both on staff. The evaluator observes the candidate’s proficiencies, leadership, and success at the event and writes a review assessing the candidate. The evaluator provides the review to the panel lead.
The review should cover a broad range of skills expected from a Level Five Judge, including, but not limited to:
- Pre-event communication
- Preparation for team or head judge tasks
- Communication with other judges, leads, and head judges
- Success at team tasks
- Mentorship
Evaluators are strongly encouraged to include any assessment or feedback relevant to the L5 skillset.
Using the insight gained from the self-evaluation, recommendation, and pre-interview, the candidate undergoes a practical evaluation. This differs from the L4 practical evaluation, in that it is focused on what the panelists want more information on. It may ask for information about the candidate’s investigations, or see how they deal with complex logistical tasks. This is intended to serve as a supplemental resource to the written documents.
It is possible that during the Pre-Panel Interview or Practical, a significant deficiency is detected. In the event that occurs, and the panel lead believes that the candidate will fail the panel, the candidate is given the appropriate feedback and guidance, with the opportunity to resubmit their application once they have had time to address the concerns.
While one good or bad day shouldn’t change the outcome of a candidate’s advancement process, a candidate can demonstrate significant deficiencies at any point throughout the process. A candidate who does so will be given specific feedback on the point of failure, and ideally given guidance on how to improve it or to whom they can speak to develop their skills. A candidate’s process will be held in stasis, and not expire for an extended period of time, while they’re actively working on improvement.
Step 6: Panel Interview
The L5 Testing Manager, candidate, panel lead, and panelists will coordinate an appropriate time and venue for the panel. In-person panels are highly recommended, but online panels may be considered in circumstances where an in-person panel would be extremely difficult to schedule.
The panel is a group interview where the panel asks questions of the candidate to assess them in each quality. The questions will vary, depending on the candidate’s known strengths and weaknesses.
The panel lead will assess the candidate in each of the 5 quality categories, including each subcategory.
Each category will be evaluated and assessed on this scale.
- Exceeds Expectations
- Meets Expectations
- Area for Improvement
- Deficient
Candidates who are evaluated by the panel lead during the panel to meet expectations in a majority of categories, with no deficiencies, are promoted to Level Five at the panel.
Unsurprisingly, the panel will evaluate the candidate on the same qualities and using the same scale as the rest of the process. Requiring no deficiencies is a change from old processes. However, old advancement procedures had between seven and eleven qualities that were being evaluated, whereas this process only has five, with several sub-categories. It is possible for a candidate to receive a deficiency in a sub-category, but have the whole category be an area for improvement, and consequently pass.
Candidates who fail their panel will be given feedback on where they need to improve, and will have the opportunity for an expedited process to return to the panel stage.
Maintenance
To maintain the Level Five certification, a judge must complete the following items each year
- Choose one —
- Pass three out of four advanced set update quizzes throughout the year
- Pass an Advanced Rules Practice and Advanced Policy Practice test
- Choose three, at least one A and one B must be chosen. —
- Lead a total of 6 teams as a Team Lead at events (A)
- Head-judge an event with at least twelve judges (A)
- Work at least six multi-day events (A)
- Serve on one or more advancement panels (B)
- Actively participate in a core project (B)
- Create educational content (ex: article, conference presentation, video, etc.), subject to approval (B)
- Write a self-review
- Maintain membership in the IJP by being up-to-date on their membership dues
As with all other levels, there is an expectation of activity. For L5, that includes working large events in leadership roles and continuing to help support and grow the community.
Self-evaluation is included, as it is always important to make sure you’re at the top of your performance.
Level 5 can only be maintained three times before an evaluation maintenance is required. This maintenance has three requirements:
- Passing the Advanced Rules and Advanced Policy tests
- A recommendation from an L5 judge who has not recommended you previously
- A panel evaluation (similar in structure and format to L5 Advancement panels) by two L5 judges who have not previously served on a panel for that individual
The primary goal of evaluation maintenance is to make sure that L5s remain sharp, as well as to help ensure that high level judges are receiving in-depth feedback.
As we’ve mentioned in several places, senior judges get less feedback than others, so requiring a more thorough evaluation is intended to keep them from plateauing, or worse, losing skill over time.
Frequently, these recommendations and evaluations will be perfunctory for candidates who are active and engaged in the community and judging. However, for judges who are spending less time judging, this may serve as either an opportunity for them to step down to level four, or as a reminder to sharpen their skills and continue to pursue excellence.
