Incorporating human expertise and domain knowledge is particularly important for medical image processing applications, marked with small datasets, and objects of interests in the form of organs or lesions not typically seen in traditional datasets. However, the incorporation of prior knowledge for breast tumor detection is challenging, since shape, boundary, curvature, intensity, or other common medical priors vary significantly across patients and cannot be employed. This work proposes an approach for integrating visual saliency into a deep learning model for breast tumor segmentation in ultrasound images. Visual saliency emphasizes regions that are more likely to attract radiologists' visual attention and stand out from its surrounding. Our approach is based on a U-Net model and employs attention blocks to introduce visual saliency. Such model forces learning feature representations that prioritize spatial regions with high levels of saliency. The approach is validated using a dataset of 510 breast ultrasound images.