Heart failure (HF) poses a significant public health challenge due to its rising global mortality rate. Addressing this issue through early diagnosis and prevention could significantly reduce the disease's impact. This work introduces a methodology for HF risk prediction using clinically acquired 12-lead electrocardiograms (ECGs). We present a novel, lightweight dual-attention ECG network designed to capture complex ECG features essential for early HF prediction, despite the notable imbalance between low and high-risk groups. The network features a cross-lead attention module and twelve lead-specific temporal attention modules to capture cross-lead interactions and local temporal dynamics within each lead. To prevent model overfitting from limited training data, we leverage a large language model (LLM) with a public ECG-Report dataset for pretraining on an ECG-report alignment task. The network is then fine-tuned for HF risk prediction using two specific cohorts from the UK Biobank study, focusing on patients with hypertension (UKB-HYP) and those who have had a myocardial infarction (UKB-MI). Our findings show that LLM-informed pretraining significantly improves the network's HF risk prediction capability in these cohorts. Moreover, the dual-attention mechanism enhances interpretability and predictive performance, ensuring a transparent and reliable prediction process. The method outperforms existing models, achieving average C-index scores of 0.6349 and 0.5805 on the UKB-HYP and UKB-MI test sets, respectively. This performance demonstrates our approach's effectiveness in managing complex clinical ECG data and its potential to improve HF risk assessment across various populations.