Wasps can't read house numbers, its quite common to have a nest at number 32 but the wasps are getting confused and flying into an identical location on number 28. All the returning wasps know that they live 3 tiles along from the drain pipe just above the bedroom window. Most of them will get it after a while and fly off to your neighbours house. However some of them won't and after flying into your home and wondering where their nest has disappeared to they will often end up in your bathroom or kids bedroom.
If your getting the occasional wasp in your home its a good idea to spend a few minutes looking up at the roofline of yours and your neighbours properties. If you can see a few wasps flying around and entering your home and loads of wasps flying around and entering the exact same location on your neighbours its likely thats where the problem is.
If you live in a semi detached house the nest won't be on the house thats attached to yours it will be on the house next to that. Obviously this isn't always the case it is possible to have nests in similar locations on neighbouring houses so before you go falling out with your neighbours its probably a good idea to get a pest control company to check it out.