As people increasingly use social media as a source for news consumption, its unmoderated nature enables the diffusion of hoaxes, which in turn jeopardises the credibility of information gathered from social media platforms. To mitigate this problem, we study the development of a hoax detection system that can distinguish true and false reports early on. We introduce a semi-automated approach that leverages the Wikidata knowledge base to build large-scale datasets for veracity classification, which enables us to create a dataset with 4,007 reports including over 13 million tweets, 15% of which are fake. We describe a method for learning class-specific word representations using word embeddings, which we call multiw2v. Our approach achieves competitive results with F1 scores over 72% within 10 minutes of the first tweet being posted, outperforming other baselines. Our dataset represents a realistic scenario with a real distribution of true and false stories, which we release for further use as a benchmark in future research.