<AI Smry>: An evaluation model that learns to predict human-like scores to input responses, using a new dataset of human response scores is presented and it is shown that the ADEM model’s predictions correlate significantly, and at a level much higher than word-overlap metrics such as BLEU, with human judgements at both the utterance and system-level.