Pack the gaps tightly until they are full and subsequently get rid of any excess grout.
Why not, they are annoying of course however it really is not destroying your homes structural foundation. The neighbors home sold right away for more money.