Semantic communication has emerged as a new paradigm to facilitate the performance of integrated sensing and communication systems in 6G. However, most of the existing works mainly focus on sensing data compression to reduce the subsequent communication overheads, without considering the integrated transmission framework for both the SemCom and sensing tasks. This paper proposes an adaptive source-channel coding and beamforming design framework for integrated sensing and SemCom systems by jointly optimizing the coding rate for SemCom task and the transmit beamforming for both the SemCom and sensing tasks. Specifically, an end-to-end semantic distortion function is approximated by deriving an upper bound composing of source and channel coding induced components, and then a hybrid Cramér-Rao bound (HCRB) is also derived for target position under imperfect time synchronization. To facilitate the joint optimization, a distortion minimization problem is formulated by considering the HCRB threshold, channel uses, and power budget. Subsequently, an alternative optimization algorithm composed of successive convex approximation and fractional programming is proposed to address this problem by decoupling it into two subproblems for coding rate and beamforming designs, respectively. Simulation results demonstrate that our proposed scheme outperforms the conventional deep joint source-channel coding -water filling-zero forcing benchmark.