<AI Smry>: This work introduces a new human-and-model-in-the-loop framework for evaluating the toxicity of generative models, and proposes two novel methods for safe conversational agents by either training on data from the framework in a two-stage system, or ”baking-in” safety to the generative model itself.