I tried the command on my development server to rename a site collection:
$site = Get-SPSite "https://sp2013/sites/Site2
Site1 no longer existed, but when browsing to Site2 I got errors. I have custom errors turned off on my dev system so I get to see the actual error, and the errors were coming from the Publishing cache indicating that Site1's url does not exist.
So, to clear the cache out, I ran
And that did it for me!