Participatory design has emerged as a popular approach to foreground ethical considerations in social robots by incorporating anticipated users and stakeholders as designers. Here we draw attention to the ethics of participatory design as a method, distinct from the ethical considerations of the social robot being co-designed. More specifically, we consider the ethical concerns posed by the act of stakeholder participation - the morals and values that should be explicitly considered when we, as researchers or practitioners, devise protocols for participatory design of social robots ("how" stakeholders participate). We use the case of robot-assisted sexual violence mitigation to exemplify ethical considerations of participatory design protocols such as risk of harm, exploitation, and reduction of stakeholder agency. To incorporate these and other ethical considerations in the creation of social robot participatory design protocols, we advocate letting stakeholders design their own form of participation by including them in the creation of participatory design sessions, structures, and processes.