Compared with data with high precision, one-bit (binary) data are preferable in many applications because of the efficiency in signal storage, processing, transmission, and enhancement of privacy. In this paper, we study three fundamental statistical estimation problems, i.e., sparse covariance matrix estimation, sparse linear regression, and low-rank matrix completion via binary data arising from an easy-to-implement one-bit quantization process that contains truncation, dithering and quantization as typical steps. Under both sub-Gaussian and heavy-tailed regimes, new estimators that handle high-dimensional scaling are proposed. In sub-Gaussian case, we show that our estimators achieve minimax rates up to logarithmic factors, hence the quantization nearly costs nothing from the perspective of statistical learning rate. In heavy-tailed case, we truncate the data before dithering to achieve a bias-variance trade-off, which results in estimators embracing convergence rates that are the square root of the corresponding minimax rates. Experimental results on synthetic data are reported to support and demonstrate the statistical properties of our estimators under one-bit quantization.