It is a little bit of a shame to do it to what will be one of the most collectable Sandmans (if it is one) if they ever become really valuable (being an L31 Kingswood ute Sandman). It is the sort of thing lesser spec cars should be used for (6cyl and 253 examples), but it is your car and it will appeal to a different market and being a Sandman won't hurt its value.
I'd do it too, but I wouldn't use a manual so I wouldn't have to touch the floor. I'm even considering an LS1 for one of my GTS327's rather than a 427 as it is far easier and cheaper, but i'd be using a Muncie so no floor mods are required.