The reason why water is backing up in the tub when you flush your toilet is simple: The sewer line is blocked. Now that that's out of the way, the more important issue is what to do about it, and that might depend on whether your waste pipes empty into a municipal sewage system or into a septic system.
If your home uses a septic system, the backup could be caused by problems in the septic tank or drainfield as well as a clog in the waste line. That doesn't necessarily mean the situation is easier to diagnose if you're on a municipal waste system. The blockage could be in the lateral pipes that connect your home to the main sewer. It's unlikely you can find such a blockage by yourself, much less clear it. Nevertheless, when you have a sewage backup in the tub, there are a few things you should try. You might get lucky.
Why Exactly Is the Toilet Backing Up into the Tub?
If the toilet and bathtub are in the same bathroom, their drains usually connect. When flush toilet water comes up in the tub, it means the blockage is located somewhere past the point of connection. Otherwise, water would simply back up in the toilet. The tub drain is usually at a slightly lower elevation than the toilet, so it provides a way for the flush water to get out of the pipe.
If this situation occurs suddenly, it's a good indication that the clog is caused by something bulky in the pipes – perhaps a diaper or some other item that you shouldn't have flushed. If the situation develops gradually over several weeks or months, it points to a more systemic problem, such as trees roots growing in the sewer pipes or a septic malfunction.
Dealing with a Septic System Backup
If your house is served by a septic system, check the drainfield. If it feels soggy, or you can detect sewer smells, it's probably blocked. Consult with a septic professional for remediation strategies. If the drainfield seems okay, check around the tank for soggy ground that could be the sign of an overflow. If your system uses a transfer pump, make sure the pump is operational. Sometimes septic backups are caused by something as simple as a tripped breaker. Finally, if it has been longer than five years since you've had the tank pumped, now is the time to do it. If needed, you can get a more thorough evaluation of your system by the professional who does the pumping.
Clearing a Clogged Sewer Pipe
If you're on a municipal system or you've determined your septic system is working normally, and you have a sewage backup in the tub or the toilet backing up into the shower, you have a clog to clear. A plunger can do wonders in such a situation, provided you use it properly. Try blocking the tub drain with duct tape and plunging the toilet with a bell-shaped or bellows-style toilet plunger. You may develop enough pressure to break up the clog.
You might also be able clear the clog with a sewer auger, but to do that, you'll either have to remove the toilet or find a cleanout valve on the sewer line through which to insert the auger. Don't try to insert a sewer auger through the toilet. You could break the toilet.
If you can't clear the clog yourself, your best bet is to call a plumber. A professional has tools, such as cameras and water jet augers, to find clogs and bust them. If you have roots in your pipes, the pro will be able to find them and advise you of the best course of action.