It should definitely only be up to the carrier. The only problem I think that can come from this is that so many unfortunate sqeaker souls will be thrown off the map.
LOL that would be such a punk move! Carrier: 'Yes I will carry you to safety my friend' Downed: 'Thank you good sir' 10 secs later Downed: 'You're going the wrong way' Carrier *silence* Edge of map is in sight Carrier *walks towards the ledge* Downed: 'Don't do it, don't...do...it!' Carrier *throws sad soul off of the map, turns his mic on, laughs meniacally and leaves the match*