To make it safer, the exchange site could act as an escrow for the bitcoin side of the payment. The seller puts the bitcoin payment in escrow, and the buyer sends the conventional payment directly to the seller. The exchange service doesn't handle any real world money.
This would be a step better than e-bay. E-bay manages to work fine even though shipped goods can't be recovered if payment falls through.