A wasp nest in your upstairs exhaust vent may or may not be a problem, depending on what type of wasps are living there. Mud daubers are solitary insects that seldom sting, but paper wasps and bald-faced wasps are more social and aggressive. You should only attempt to exterminate them if the nest is close to the roof, and they are using the roof opening. If they are coming into your house through the fan opening, it's dangerous to try to exterminate them by yourself. Call an insect control professional.
Get a closeup view of one of the wasps using the nest and compare it with an online insect catalogue to identify it. If you only see one wasp, and it's very thin-waisted, it's probably a mud dauber. If you see several striped insects that resemble bees, they are probably paper wasps or yellowjackets.
Remove a mud dauber nest when the insect is away. It will simply make a new nest somewhere else. Turn off the exhaust, remove the fan cover and and look for the nest with a flashlight. Pull it out if you can reach it with a tongs or your fingers -- while wearing leather gloves. If you can't reach it, and it isn't interfering with the operation of the fan, cover both the fan cover and the exhaust opening on the roof with cardboard and leave them blocked for a few days. If the wasp can't get in, it will make another nest elsewhere.
Choose a time late at night or early in the morning to deal with yellowjackets or paper wasps. It's best to exterminate these wasps in June, before their nests get large, but you often don't notice nests until late summer, when they are teeming with insects.
Turn off the exhaust fan and put duct tape over the switch to prevent anyone from turning it on. Place cardboard over the fan cover and tape the cardboard to the ceiling, being sure not to leave any openings through which a wasp could crawl.
Put some insecticidal dust in a plastic ketchup container. Be sure the dust is safe for indoor use. Appropriate products contain bendiocarb, chlorpyrifos or boric acid. Boric acid works more slowly than the other two insecticides.
Drill a small hole -- about 1/8 inch in diameter -- and pump dust into the exhaust vent through hole by squeezing the container. Pump hard; the harder you pump, the farther the dust will travel. Cover the hole with duct tape as soon as you're done.
Go into the attic and locate the vent pipe coming from the bathroom and going through the roof. Drill a 1/8-inch hole in the metal and pump dust through the hole. Cover the hole immediately with duct tape.
Look for activity around the exhaust vent opening after three or four days and listen for buzzing in the vent pipe. Dust again if you see or hear wasps.
Remove the cardboard once you are sure there is no more wasp activity. It's best to wait for several days after activity around the roof vent ceases and you stop hearing buzzing.