If you do all the cleaning, you could have your husband and the toddlers start their holiday trip a day or two before you, so that there’s no one around messing things up after you’ve cleaned them. However, I’m more inclined to think that your interpretation of clean is just not at the same level as your guests. To clarify, are you getting 3 and 4 star ratings in the cleaning category or for the overall rating? If it’s just the cleanliness category, I’m pretty sure you’re not going to get kicked off, but a low cleanliness rating might affect the type of guests that book your place, and probably not for the better.
Some good tips for a similar situation as yours in this thread.