This paper presents a system for autonomous cooperative wall building with a team of Unmanned Aerial Vehicles (UAVs). The system was developed for Challenge 2 of the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020. The wall-building scenario of Challenge 2 featured an initial stack of bricks and wall structure where the individual bricks had to be placed by a team of three UAVs. The objective of the task was to maximize collected points for placing the bricks within the restricted construction time while following the prescribed wall pattern. The proposed approach uses initial scanning to find a priori unknown locations of the bricks and the wall structure. Each UAV is then assigned to individual bricks and wall placing locations and further perform grasping and placement using onboard resources only. The developed system consists of methods for scanning a given area, RGB-D detection of bricks and wall placement locations, precise grasping and placing of bricks, and coordination of multiple UAVs. The paper describes the overall system, individual components, experimental verification in demanding outdoor conditions, the achieved results in the competition, and lessons learned. The presented CTU-UPenn-NYU approach achieved the overall best performance among all participants to won the MBZIRC competition by collecting the highest number of points by correct placement of a high number of bricks.