Princess Cays is actually on the island of Eleuthera, one of the Bahamas' larger islands.
Yes, George's is right outside the gates. When you get off the tender and come to the point where they take your picture, you'll notice a building with public restrooms to the right. Just beyond that is a big gate. If you go through the gate (there is a security officer there checking ship cards), there is a souvenir stand and at the far end is George's.
It's plenty safe to go there. When we went there were many cruisers, plus security is right there.
As for the OP, we have seen taxi drivers in this area before, and on our last cruise, we hired one to give us a tour. He took us as far north as Tarpum Bay before turning around and heading back to the ship. He also took us to a nice local beach bar/restaurant called Wild Orchids along the way for lunch.
We didn't see anywhere to rent a car, however.