This paper introduces a supervised deep-learning network that jointly optimizes the physical setup of an optical microscope to infer fluorescence image information. Specifically, we design a bright-field microscope's illumination module to maximize the performance for inference of fluorescent cellular features from bright-field imagery. We take advantage of the wide degree of flexibility available in illuminating a sample to optimize for programmable patterns of light from a customized LED array, which produce better task-specific performance than standard lighting techniques. We achieve illumination pattern optimization by including a physical model of image formation within the initial layers of a deep convolutional network. Our optimized illumination patterns result in up to a 45% performance improvement as compared to standard imaging methods, and we additionally explore how the optimized patterns vary as a function of inference task. This work demonstrates the importance of optimizing the process of image capture via programmable optical elements to improve automated analysis, and offers new physical insights into expected performance gains of recent fluorescence image inference work.