Federated Deep Learning (FDL) is helping to realize distributed machine learning in the Internet of Vehicles (IoV). However, FDL's global model needs multiple clients to upload learning model parameters, thus still existing unavoidable communication overhead and data privacy risks. The recently proposed Swarm Learning (SL) provides a decentralized machine-learning approach uniting edge computing and blockchain-based coordination without the need for a central coordinator. This paper proposes a Swarm-Federated Deep Learning framework in the IoV system (IoV-SFDL) that integrates SL into the FDL framework. The IoV-SFDL organizes vehicles to generate local SL models with adjacent vehicles based on the blockchain empowered SL, then aggregates the global FDL model among different SL groups with a proposed credibility weights prediction algorithm. Extensive experimental results demonstrate that compared with the baseline frameworks, the proposed IoV-SFDL framework achieves a 16.72% reduction in edge-to-global communication overhead while improving about 5.02% in model performance with the same training iterations.