Should this also count as Clever? There is something similar about a weapon that can't be drawn unless the right circumstances are met, but the character still finds a way to make use of it.
Case in point, Thanos hitting Thor against his own hammer, instead of trying to use the hammer himself. It has to be seen to be believed.
I'm fairly certain that the Master Sword's ability of not being able to be drawn unless it's Link extends the charm to the pedestal of time as well even the whole temple of time and sacred grove around it.
Therefore the pedestal can not be destroyed either. It was created by Hylia a goddess to be part of the Master Sword to protect it from harm as well as to house it when it's not in use. Otherwise Ganon would have tried this after Ocarina of Time.