When the Gangway goes up it pretty much means, "Game Over".
If Gangway is up at 4:00 and you come at 4:02 then you likely will become the victim of somebody's YouTube video to be seen by the whole world. Yes, people actually stand on their balconies and record what are called, "runners". In the end the majority of runners end up standing on the pier looking dazed and confused as the ship sails away. Should this scare people? Heck yes it should. For several reasons. This can cost you a lot! You will either have to fly to the next island at your expense or fly home at your expense. What is even more costly is for those traveling with a birth certificate and ID. Yes, this can be done in a closed loop cruise. There is just one problem with this. While you may have saved a few bucks on not getting a passport, you are going to pay dearly for this mistake if you miss the ship. Airlines require passports so you will be stuck on an island dealing with the Embassy trying to fix this big mistake and paying for a hotel, food and airfare for a while until the problem is resolved. If you missed your ship and it is no fault of your own, (an example would be if your transportation on the island broke down and you did not book the tour through the ship) then it may be covered by travel insurance (Which is cheap! Always buy it)
This is more than you asked for, but I can assume you are first time cruisers and so I'm trying to save you the trouble of costly mistakes. The short answer would be that when the Gangway goes up, nobody else will board and the ship is getting ready to sail. It is always a wise choice to leave your shoes excursion into plenty of time to accommodate for traffic and otherwise to give yourself plenty of time to get back to the ship. If you are on one of Royal's excursions they won't leave you behind unless you get off the excursion and go shopping or something and then become late for the ship. If the Gangway goes up at 4:00 I would recommend getting on the ship by 3:30 at the latest.