We engaged MOOC learners (mentors; n=120) as collaborators in the design and improvement of an online course through an activity called the Mentor Academy; an online instructional program where mentors create original problem sets for future iterations of a course. Following a design research methodology, we trace the development of the program’s design and report on the features of a learning design that effectively support mentors in the problem creation process. Our analysis shows that mentors were offered three kinds of support: community building, logistical, and pedagogical. The following kinds of changes and additions were made to the design: expanded project requirements, improved technical infrastructure, and increased opportunities for interaction in discussion forums and live peer chat sessions. We present an analysis of problem sets for quality, diversity of subject area, and usability.