Working Late or.......

Since the day I have been into Job I have observed that many people stay in the office till late evening. Initially I appreciated it and thought how hard work these guys are putting up to meet the project deadline and providing production support.
But gradually I started realising that more than 70% of these hard working guys really don't do such hard work. They stay late in the office either because:
1. They have got nothing else to do or no other place to go and spend time - These people gradually becomes accoustmed to staying late and feels better in office than at home.
2. Then their are other kinds of people; people who stays late in the office just because their bosses are staying late. Their bosses again may be staying late because of reason 1 or 2.

We do encounter people who stay late in the office for genuine reasons, but the percentage of such people is very low. The argument goes like this:
1. If the above mentioned 70% are genuinely staying late, then the manpower estimation has been done really bad by the project manager.
2. If we say that the manager has really done the estimation efficiently, then high probability for the root cause is that the skills have not been chosen efficiently.

Because if your manager argues that both the skills and the manpower estimation has been mapped efficiently, then how is it that more than 70% of the team has to do overtime with no pay? But the bigger question is, are these 70% of the team really doing work sitting late in the office?