The recent surge in the adoption of machine learning techniques for materials design, discovery, and characterization has resulted in an increased interest and application of Image Driven Machine Learning (IDML) approaches. In this work, we review the application of IDML to the field of materials characterization. A hierarchy of six action steps is defined which compartmentalizes a problem statement into well-defined modules. The studies reviewed in this work are analyzed through the decisions adopted by them at each of these steps. Such a review permits a granular assessment of the field, for example the impact of IDML on materials characterization at the nanoscale, the number of images in a typical dataset required to train a semantic segmentation model on electron microscopy images, the prevalence of transfer learning in the domain, etc. Finally, we discuss the importance of interpretability and explainability, and provide an overview of two emerging techniques in the field: semantic segmentation and generative adversarial networks.