Even infants have a remarkably sophisticated understanding of objects, agents, and how they interact. We investigated young children’s ability to reason about the relationship between complexity of physical structures created by agents, their perceived difficulty, and the time required for creating these structures. Seventy 4-5 year-olds were shown trials consisting of pairs of agents who had the same numbers of blocks but made different structures (e.g., horizontal line vs. vertical tower, castle structure vs. two piles of blocks). Children were asked which structure was easier to make (Difficulty condition) or who was done first (Time condition). Even the youngest participants were successful in determining which structure is more difficult, but their estimates of time showed improvement with age. These results offer novel insights into how an early understanding of difficulty and time shape young children’s beliefs about how agents intervene on the physical world to induce changes in their states.