Update February 13. 2019: This only works until OSX Yosemite (10.10). On Yosemite and up ipfw was replaced by pf.

In OSX you can limit the network bandwidth with ipfw rules. But this works on the whole machine, not only on some applications.

To limit the bandwidth for all targets/destination, execute

sudo ipfw pipe 1 config bw [Speed]
sudo ipfw add pipe 1 dst-ip 0.0.0.0/0

for example:

sudo ipfw pipe 1 config bw 300kbit/s
sudo ipfw add pipe 1 dst-ip 0.0.0.0/0

If you don't want to limit all kind of connections and just some IP addresses or ports, you can do:

sudo ipfw add pipe 1 dst-ip 192.168.1.0/24

sudo ipfw add pipe 1 dst-port http
sudo ipfw add pipe 1 dst-port https
sudo ipfw add pipe 1 dst-port pop3
sudo ipfw add pipe 1 dst-port pop3s
sudo ipfw add pipe 1 dst-port imap
sudo ipfw add pipe 1 dst-port imaps

You can remove all those rules by simple executing sudo ipfw flush. A reboot would also remove those rules.

For further information, have a look at the man page for ipfw. If you don't like executing commands, you can also have a look at WaterRoof.