With the increase of pages and buttons in real-world applications, industrial-scale recommender systems face multi-domain and multi-task challenges. On the one hand, users and items in multiple domains suffer inconsistent distributions. On the other hand, multiple tasks have distinctive sparsity and interdependence. Personalization modeling is the core of recommender systems. Accurate personalization estimation helps to capture the degree of user preference for items in different situations, especially in the case of multiple domains and multiple tasks. In multi-task and multi-domain recommendation, how to introduce personalized priors into the model in the right place and in the right way is crucial. In this paper, we propose a plug-and-play Parameter and Embedding Personalized Network (PEPNet) for multi-task recommendation in the multi-domain setting. PEPNet takes features with strong bias as input and dynamically acts on the bottom-layer embeddings or the top-layer DNN hidden units in the model through the gate mechanism. By mapping significant priors to scaling weights ranging from 0 to 2, PEPNet introduces both parameter personalization and embedding personalization. Embedding Personalized Network (EPNet) selects and aligns embeddings with different semantics under multiple domains. Parameter Personalized Network (PPNet) influences DNN parameters to balance interdependent targets in multiple tasks. To further adapt to the characteristics of the model, we have made corresponding engineering optimizations on the Embedding and DNN parameter update strategies. We have deployed the model in Kuaishou and Kuaishou Express apps, serving over 300 million daily users. Both online and offline experiments have demonstrated substantial improvements in multiple metrics. In particular, we have seen a more than 1\% online increase in three major domains.