United and Brighton both had opportunities early in the first half, with Antony firing wide in the second minute before Kaoru Mitoma hammered a fierce shot straight into the face of David De Gea.
However, the deadlock was not broken until the ninth minute of stoppage time, when Alexis MacAllister scored from the spot to secure victory after Luke Shaw had handled the ball in the box.
"Every defeat is a disappointment but in the end when you lose in the last second, that is of course annoying," Ten Hag told reporters.
"And I think the first minutes sum everything up. We create a good chance, we were not clinical enough, then in the attack after we concede a big chance by a giveaway from us, the ball on the head for David.
"And in the end, we also gave away the goal and that's annoying. If you can't win because you don't finish your opportunities then don't lose."
Asked if United's defeat was caused by fatigue, Ten Hag said: "I have seen our physical outputs in the last games, we still have high levels, performing really well.
"The last five games, we had four wins, one draw. We were in a good run, we are improving. But we still have a way to go. We have to be more clinical and stay focused until the end of the game."
The defeat left United in fourth place on 63 points. They next play at West Ham United on Sunday.
