You don't give specifics as to what is "fairly late", so this is just general advice.
If you don't have a flight until late afternoon/evening, my plan would be to do the following.
1) Take a taxi from the port to the BSI bus terminal.
2) Store my luggage at the facility there
3) Spend the day in downtown Reykjavik, browsing as you wish
4) Return to the bus terminal, get your bags, then take the Flybus directly to KEF
This has a number of advantages. First, you aren't sitting around the airport for hours. It gives you time to wander through town on your own. And the local taxi + Flybus cost will be significantly less than a taxi all the way from the port to KEF.
You might also be able to get Flybus+ service from the port itself. I would check with Flybus to see if this option is available. That way, your port to terminal cost gets blended into your Flybus fare.