Intelligent reflecting surface (IRS) has recently appeared as a potential technology for 6G, and received much attention from academia and industry. However, most of existing works on IRS focus on how to compute the phase shift for performance enhancement, and the problem on how to obtain the computed phase shift at the IRS side is generally neglected. In this paper, we consider compressing the computed phase shift at the receiver side to the IRS through a bandwidth-limited feedback channel. In particular, we propose and investigate a novel attention mechanism named as global attention by exploiting the attention map over both spatial and channel dimensions. This allows us to to push the limit of phase shift feedback compression by utilizing the two-dimensional information, which is in sharp contrast to exiting works that only consider either the spatial or channel dimension. Besides, to cope with the problem of mismatched distribution of the phase shift, we introduce the generalized divisive normalization (GDN) layer and inverse generalized divisive normalization (IGDN) layer to the proposed global attention phase shift compression network (GAPSCN). Furthermore, due to practical constraints on the IRS, it is desirable to consider a simplified GAPSCN (S-GAPSCN), where a lightweight multi-scale simplified global attention module (MSSGAM) is proposed in the decoder located at the IRS side to compensate for the performance degradation due to the simplified structure. Simulation results show that the proposed GAPSCN is able to achieve a reconstruction accuracy close to 1 and performs much better than existing algorithms. The performance of the proposed S-GAPSCN can approach that of the GAPSCN but with a much lower computational load.