Ethylene leakage detection has become one of the most important research directions in the field of target detection due to the fact that ethylene leakage in the petrochemical industry is closely related to production safety and environmental pollution. Under infrared conditions, there are many factors that affect the texture characteristics of ethylene, such as ethylene concentration, background, and so on. We find that the detection criteria used in infrared imaging ethylene leakage detection research cannot fully reflect real-world production conditions, which is not conducive to evaluate the performance of current image-based target detection methods. Therefore, we create a new infrared image dataset of ethylene leakage with different concentrations and backgrounds, including 54275 images. We use the proposed dataset benchmark to evaluate seven advanced image-based target detection algorithms. Experimental results demonstrate the performance and limitations of existing algorithms, and the dataset benchmark has good versatility and effectiveness.