Drug membrane interaction is a very significant bioprocess to consider in drug discovery. Here, we propose a novel deep learning framework coined DMInet to study drug-membrane interactions that leverages large-scale Martini coarse-grained molecular simulations of permeation of drug-like molecules across six different lipid membranes. The network of DMInet receives three inputs, viz, the drug-like molecule, membrane type and spatial distance across membrane thickness, and predicts the potential of mean force with structural resolution across the lipid membrane and membrane selectivity. Inheriting from coarse-grained Martini representation of organic molecules and combined with deep learning, DMInet has the potential for more accelerated high throughput screening in drug discovery across a much larger chemical space than that can be explored by physics-based simulations alone. Moreover, DMInet is highly flexible in its nature and holds the possibilities for other properties prediction without significant change of the architecture. Last but not least, the architecture of DMInet is general and can be applied to other membrane problems involving permeation and selection.