Sounds like they were not pollinated. When a female bloom is unpollinated the fruit never really grows, shrivels, and dies off.

To help with pollination you can take a Q-tip from the male flowers to the female flowers. Paintbrushes work too.