Why is incest wrong?
There’s no such thing as something that is wrong outside the context of a particular moral framework. Sometimes many people in a culture feel something is wrong (e.g. there’s a common framework in those cultures), but there are always dissenters.
Even devout religious people who say, “No! Such-and-such simply is wrong” mean it’s wrong according to God. But since they believe God’s opinion trumps everything else (e.g. it’s the default), they sometimes omit mentioning the framework.
Most cultures have an incest taboo, and it’s been speculated (and I think it’s likely) that this is, to an extent, a product of Natural Selection. We tend to be biased against practices that hindered gene propagation back in our hunter-gatherer days, and since incest could lead to birth defects, it makes sense that Natural Selection would give us an aversion to it.
Almost always, when biology works this way—when it gives us a taste or a repulsion—it can be strengthened or weakened by culture and it varies greatly on the individual level. So even if there’s a built-in aversion to incest, we should expect some people to be able to overcome it and other to be born without it. The brain is fairly plastic. Natural Selection mostly gives us mental tendencies, not locked-in traits.
Once a taboo takes root in a culture, it tends to maintain itself via inertia. Parents raise their kids to believe it who raise their kids to believe it and so on.
I personal don’t think incest is wrong. I don’t believe any sex act is wrong, as long as it occurs between consenting adults and presents no clear-and-present danger to bystanders.
What stops us from it?
If someone wants to have incest and is stopped from it, it’s due to his own inner brakes, at least where I live. In American (and I think in most other Western countries), there’s no law against incest. (See the UPDATE, below.)
People don’t do it either because they don’t feel like it or because the taboo makes them fear they’re be censured if the engage in it.
And, of course, some people don’t stop themselves from doing it.
Why can’t we have recreational sex with our own family?
In my country, we can.
UPDATE: I just checked Wikipedia, and I’m wrong about that. Incest Law is complex in the US, and various acts are allowed and disallowed, depending on where you live.