Carlos Queiroz’s Iran side could have progressed to the second round with a win but their hopes were dented when Edin Dzeko gave the Bosnians a deserved lead with a crisp shot from 25 yards.
Roma’s Miralem Pjanic slid home a second after the break.
Reza Ghoochannejad pulled one back but Avdija Vrsajevic sealed it to give Bosnia their first World Cup win.
Their first ever victory in a major tournament at least gives the Bosnians something to show for their efforts in Brazil.
Safet Susic’s side were knocked out before Wednesday’s match after narrow defeats by Argentina and Nigeria in their opening two games, but had created enough chances to have taken something from both games.
Dzeko’s well-taken goal against the Iranians was scant consolation for the Manchester City forward after he had an effort ruled out incorrectly against Nigeria when the score was 0-0, and subsequently missed a number of opportunities to get his side back in to that game.
His 23rd-minute strike was little more than the Bosnians deserved after a positive start in which the City man had twice forced saves from Iran’s Alireza Haghighi.
With the Iranians needing to win and hope Nigeria lost if they were to progress, they went on the attack immediately and Masoud Shojaei saw his shot come back off the crossbar less than a minute after Bosnia had scored.
It would have been their first goal in a World Cup finals since a 1-1 draw with Angola in 2006, but the half ended with Susic’s side looking more likely to be the next team on the scoresheet.
They made it 2-0 with just under an hour gone when Iran conceded possession needlessly 40 yards from goal and Tino Susic moved forward before sliding through to the marginally offside Pjanic, who finished coolly into the corner.
Iran – under the management of former Manchester United coach Queiroz for the final time – pulled themselves back into it when Ghoochannejhad tapped home at the back post after Bosnia failed to clear a corner but any hopes of a comeback were short-lived.
Almost immediately after the restart, Sejad Salihovic played full-back Vrsajevic through on goal and he showed composure to score his first international goal.
It ensured that Iran have now gone out at the group stage in all four of their World Cup campaigns.
Bosnia coach Safet Susic: “The win against Iran is scant consolation, but I congratulate my players for a display of commitment and passion against a team needing a win to keep alive their hopes of progressing.
“We lacked experience and cool heads at crucial moments in the opening two games but I cannot complain about effort. That’s just football.”
Iran coach Carlos Queiroz: “The best team of the group did not qualify, Bosnia showed they have great players, great team with a lot of experience.
“Today they raised their levels and when that happened they gave us no chance.
“My players went to the limit of our mental capabilities, fitness and I’m very proud, very, very proud.”