To design tools and data pipelines for explainable B2B machine learning (ML) systems, we need to recognize not only the immediate audience of such tools and data, but also (1) their organizational context and (2) secondary audiences. Our learnings are based on building custom ML-based chatbots for recruitment. We believe that in the B2B context, "explainable" ML means not only a system that can "explain itself" through tools and data pipelines, but also enables its domain-expert users to explain it to other stakeholders.