This work presents a novel data-driven multi-layered planning and control framework for the safe navigation of a class of unmanned ground vehicles (UGVs) in the presence of unknown stationary obstacles and additive modeling uncertainties. The foundation of this framework is a novel robust model predictive planner, designed to generate optimal collision-free trajectories given an occupancy grid map, and a paired ancillary controller, augmented to provide robustness against model uncertainties extracted from learning data. To tackle modeling discrepancies, we identify both matched (input discrepancies) and unmatched model residuals between the true and the nominal reduced-order models using closed-loop tracking errors as training data. Utilizing conformal prediction, we extract probabilistic upper bounds for the unknown model residuals, which serve to construct a robustifying ancillary controller. Further, we also determine maximum tracking discrepancies, also known as the robust control invariance tube, under the augmented policy, formulating them as collision buffers. Employing a LiDAR-based occupancy map to characterize the environment, we construct a discrepancy-aware cost map that incorporates these collision buffers. This map is then integrated into a sampling-based model predictive path planner that generates optimal and safe trajectories that can be robustly tracked by the augmented ancillary controller in the presence of model mismatches. The effectiveness of the framework is experimentally validated for autonomous high-speed trajectory tracking in a cluttered environment with four different vehicle-terrain configurations. We also showcase the framework's versatility by reformulating it as a driver-assist program, providing collision avoidance corrections based on user joystick commands.