Generally speaking, electricity is more expensive to use for heating applications. I guess this has to do with having to "burn" the energy at the power plant, and transmit it to your house (transmission losses, etc.). With gas (natural or propane), you are only converting it to heat at the time of use, where as the electric (assuming fossil, not nuclear or hydro, or even wind/solar/geothermal), you burn the fuel and create heat which drives the electric generation process, incurring losses, then send it to your house.
Some people prefer gas for cooking because they can vary the temperature faster (electric elements cool off more slowly than turning the gas down, and usually can't put heat in as fast as a flame).
Propane, like oil, can be a pain (compared to nat. gas/electricity) because you have to worry about filling a tank and running out, not to mention that they can interfere with your landscaping/septic plans. Experiences? My sister lives in a rural area, and her propane is fine...except they did some clearing in their back yard, and wound up moving the propane tank to the other side of the house rather than use it as a lawn ornament (it had been nestled in some trees that they removed). I used to work with a guy who also lived in a rural area and had full electric...and used to moan to me all the time about his electric bills.