I believe you would need to know whether the installation being performed is the SP5 full-build and not a patch of SP5.
Again if you try to install the same service pack to another location I suspect that it would not get installed there as there are no full-builds SP3 is available.
The last update is not clear enough and confusing. Did you uninstall SP3 and tried to install SP5 full-build?