Its buzzy flying thing time of year again, in April and May its more likely to be bees rather than wasps.
Are you seeing just a few going into holes in the brick work? This is likely to be Mason bees, these solitary bees don't build a nest and have a very short life, they will only be around for about 4 weeks. Solitary bees are not a pest they are friendly and won't harm you. (April, May)
If what you are seeing looks like thousands of even tens of thousands and if they have just suddenly turned up, what you have is a honey bee swarm. Call a local beekeeper who will be more than happy to re home them. (May, June)
Are there dozens of them buzzing around one point randomly? This is bumble bees, don't panic they are very friendly. The random flying about thing is the male bees waiting for a female to emerge from the nest this activity only last a few weeks. There will then be reduced activity until late summer. Bumbles bees are pretty chilled out and will not attack you or eat your children give them space and let them pollinate your flowers ( June, July)
I don't see active wasps nests until late June so anything before this time is unlikely to be anything to worry about. However there are plenty of queen wasps around in late April through May. The queen wasps will be looking for a suitable nesting site. Once a location has been found the queen will begin construction of the nest.
If you are still not sure if what you have is wasps or bees give me a call or send me a message.