Large Language Models (LLMs) (e.g., ChatGPT) have shown impressive performance in code generation. A large-scale study released that writing programs requires programming thinking, i.e., analyzing and implementing requirements in programming logic (e.g., sequence, branch, loop). Existing studies use LLMs to generate programs from requirements directly and do not explicitly introduce the programming thinking. This paper explores how to unlock the programming thinking of LLMs in code generation and proposes an approach named TiP. Our idea is to decompose code generation into two steps and progressively lead LLMs to analyze&implement requirements in programming logic. Specifically, TiP first generates a code sketch, which provides a high-level solving process using programming logic but omits implementation details (e.g., APIs). Then, TiP implements the sketch into a program using specific programming languages. We conduct extensive experiments on three public benchmarks (i.e., HumanEval, MBPP, and MBCPP). (1) TiP outperforms the state-of-the-art baseline - ChatGPT by up to 17.5% in Pass@1, 11.02% in Pass@3, and 9.84% in Pass@5. (2) Human evaluation shows that TiP outperforms ChatGPT in three aspects (i.e., correctness, code quality, and maintainability). (3) TiP is effective for different LLMs. (4) We explore multiple choices (e.g., chain-of-thought) for the code sketch and validate the superiority of our design. (5) We discuss the complementarity between TiP and post-processing approaches (e.g., CodeT).