In an ideal world, deployed machine learning models will enhance our society. We hope that those models will provide unbiased and ethical decisions that will benefit everyone. However, this is not always the case; issues arise from the data curation process to the models' deployment. The continued use of biased datasets and processes will adversely damage communities and increase the cost to fix the problem. In this work, we walk through the decision process that a researcher will need to make before, during, and after their project to consider the broader impacts of research and the community. Throughout this paper, we observe the critical decisions that are often overlooked when deploying AI, argue for the use of fairness forensics to discover bias and fairness issues in systems, assert the need for a responsible human-over-the-loop to bring accountability into the deployed system, and finally, reflect on the need to explore research agendas that have harmful societal impacts. We examine visual privacy research and draw lessons that can apply broadly to Artificial Intelligence. Our goal is to provide a systematic analysis of the machine learning pipeline for visual privacy and bias issues. With this pipeline, we hope to raise stakeholder (e.g., researchers, modelers, corporations) awareness as these issues propagate in the various machine learning phases.