Building from source - disadvantages?

Are there any risks or disadvantages to building software from source, compared to installing a package? Can it mess with my system in any way?

I usually avoid it because I've found it to be a faff and often doesn't work anyway but in a couple of cases it has been necessary.