I wouldn’t have let him in at 1. That’s him telling you what to do and how to be. That’s him rushing you because his travel inconvenienced him.
You can be polite about it. Just say, oh sorry. We still won’t be done at 1, but thanks for asking. We will see you at 3.
Your house, your rules.
In my house directions, I say, sorry no early check ins. Here are some things to do in Kona should you arrive early… blah blah blah…