Reflecting on Pilot Workshops
Last updated on 2025-10-14 | Edit this page
The page serves as a facilitation guide for a debrief session designed to help lesson developers share and reflect on their experience of teaching a lesson still under development. We refer to these events as “pilot workshops” but participants in debriefs may report on testing only a part of their new lesson, rather than a complete draft. The content of this page is adapted from an episode of Collaborative Lesson Development Training, modified for standalone debrief sessions when the training programme was restructured in 2025.
Begin by reminding participants of the following expectations as part of training.
- The Carpentries Code of Conduct
- How to get in the queue to participate - hand-raising, typing in chat, etc.
- Microphones muted when you aren’t speaking (if virtual)
- Any other expectations for interaction
Your main task is to facilitate a discussion among participants about piloting a new lesson and iterating on the it to improve the design, content, and delivery afterwards. Your objectives are to help participants:
- Reflect on their experience piloting lesson content and any feedback they received from learners, helpers, co-instructors, and other observers present for the event.
- Identify improvements they could make to the design, content, and delivery of the lesson.
- Engage in constructive discussion with other participants.
Your session may include Lesson Developer trainees joining to complete their certification after Collaborative Lesson Development Training as well as other participants from the Carpentries lesson development community reporting on pilot workshops they have been involved in. Please give priority to Lesson Developer trainees, allowing them sufficient time to report back on piloting their lesson and ask any questions they brought to the session. Active participation in a Pilot Workshop Debrief session is a required checkout step for them to complete their Lesson Developer certification.
In this session, we will reflect on and discuss the experience of teaching new lesson material for the first time. We will also explore ways that you could improve your lessons and workshops based on your observations and the feedback you received during and after the event.
This exercise can take a long time when you have a lot of lesson teams in your training, but it is important to give every group the chance to reflect on their pilot. You may need to facilitate it differently if time becomes an issue: for example, you could paste the prompts into the Etherpad and ask all groups to write their answers, before summarising responses, identifying trends, and calling on particular trainees to share more about their individual responses.
Lesson developers will typically join these sessions with specific questions for the facilitator and other participants. In addition to answering these questions, try to identify points in their debriefs that could be turned into actionable feedback or modifications to the lesson they taught. Some points you might consider raising with them:
- Not every feedback comment requires action.
- Is the material too dense? Does it need extra explanations?
- Are there points in the lesson where adding a diagram would help with explanation?
- Is there too much content in any of the episodes? Could it be split into smaller teaching units?
- Does any of the content need to be re-organised to improve the flow and narrative?
- Are there enough exercises and practical work?
- Were the learning objectives met?
Discussion: lesson pilots (45 minutes)
Report out to the facilitator and other session participants your lesson pilot. Participants should choose what to focus on, e.g. based on their own reflections and feedback received during/after the event. The following prompts may help to guide your report:
- What worked well both in terms of content and delivery?
- What did not work as well?
- How close were your time estimates to the actual time needed to teach the material and finish exercises?
- How did the audience perceive the difficulty level of the material?
- What will you do differently next time?
- What will you change in the material you taught?
- What will you change in the way you collect feedback in future pilots?
- Did you identify anything that could be added as an Instructor Note to guide you and others next time the lesson is taught?
If multiple participants are joining to debrief from the same lesson pilot event, only one of them needs to report out. Others may choose to add their own reflections and ask additional questions.
We hope that the experience of teaching a new lesson was useful, even if it is still in the very early stages of development. We recommend that you continue to take this iterative approach to lesson development: testing new content early and often, and not being afraid to make significant changes to design, content and delivery based on your experience.
Do not aim for perfection before teaching your lesson
Perfect always takes so long because it don’t exist.
- Jeff Rosenstock
Next Steps
Piloting somebody else’s lesson?
Participants joining this session to report on a beta pilot workshop, i.e. an event where they taught a new lesson developed by somebody else, may not be able to take all of the actions listed below. The facilitator should encourage them to focus on the constructive feedback they could provide to the developers of the lesson they taught and to consider what role they want to take in the ongoing development and maintenance of the lesson afterwards.
Moving forward from your lesson pilot, the next steps to consider are:
- Record all the identified changes and improvements that can be made to the lesson material and assign a person from the team responsible for each task. We recommend opening a GitHub issue for each separate action item.
- Schedule follow-up co-working sessions with your team to carry on working on fixing issues and adding new content to maintain the momentum.
- Add/Update your lessons’ Instructor Notes based on what you learned and to help other instructors who will teach your lesson in the future.
- Think about the timeline for your next pilot(s), even provisionally, to help you set milestones and targets to work towards.
- Is the lesson ready to move to the next stage of the life cycle, e.g. pre-alpha -> alpha or alpha -> beta?
- Consider writing up your pilot experience in a mini blog post that you can share with the community.
Designing and developing quality lessons is hard - there are many pieces of a puzzle that have to come into place: both pedagogical and organisational. Do not be disheartened with the amount and type of feedback you may have received. Even Carpentries lessons that have been around for 10+ years receive improvement suggestions and fixes almost daily. From our experience, bigger lessons that are delivered over a few days require several full pilots before they can even be considered for a beta release. Planning smaller lesson trials (where you test only a portion of a lesson) and doing them more often with a friendly audience from your local groups and close colleagues is more manageable and will help you make steady progress.
- “Perfect is the enemy of good” - your lesson does not need to be perfect before you pilot or release it for community review. Early feedback from the target audience will help you avoid straying off your lesson plan.
- Identify changes and improvements you want to make as a result of trialling your lesson and schedule co-working sessions to work on these tasks.