Standard methods in supervised learning separate training and prediction: the model is fit independently of any test points it may encounter. However, can knowledge of the next test point $\mathbf{x}_{\star}$ be exploited to improve prediction accuracy? We address this question in the context of linear prediction, showing how debiasing techniques can be used transductively to combat regularization bias. We first lower bound the $\mathbf{x}_{\star}$ prediction error of ridge regression and the Lasso, showing that they must incur significant bias in certain test directions. Then, building on techniques from semi-parametric inference, we provide non-asymptotic upper bounds on the $\mathbf{x}_{\star}$ prediction error of two transductive, debiased prediction rules. We conclude by showing the efficacy of our methods on both synthetic and real data, highlighting the improvements test-point-tailored debiasing can provide in settings with distribution shift.