When you flush your toilet, the waste water should go down the toilet's drainpipe and eventually into the sewer system or septic tank. The bathtub's waste pipe connects to the same larger drainpipe as the toilet, meaning a clog in the plumbing will allow the toilet's waste water to come into the bathtub.
When your toilet's drain water no longer goes down the drainpipes the right way, you need to clear out any clogs from the toilet's drainpipe. Since the clog sits low enough that the water transfers from the toilet's drainpipe to the bathtub's drainpipe, you need to first remove the toilet to gain direct access to the pipe in the floor. Shut off the toilet's water supply and flush the toilet, and then sponge the rest of the water out of the tank. Back off the nuts from the bolts on the toilet's base, and carefully move the toilet to another location. Feed a drain auger down the pipe in the floor as you turn the crank on the handle clockwise. Wind the auger back up once you reach the end of the flexible line, spinning the crank counterclockwise.
Run a drain auger down the bathtub's drainpipe as well, ensuring that a clog in the bathtub's line is not complicating the problem. Remove the bathtub's overflow drain cover by backing out the anchor screws that hold it in place. Feed the flexible auger line down the overflow tube and into the main drainpipe below while you turn the auger's crank clockwise. Retrieve the auger once you have fed the entire flexible line into the tub's drainpipe, and spin the auger's crack counterclockwise as you do.
A clog or other problem with the vent pipe that connects to the main drainpipe for your bathroom may be the source of the problem. To inspect the vent pipe for problems, climb onto your roof and locate the vent pipe opening that sits over the bathroom. Any objects that sit on the vent openings, such as bird nests, you must remove so the vent pipes work correctly. Shining a flashlight down the vent pipes will allow you to see obstructions further down. If your flashlight does not give you a good look down the vent pipes, try lowering the flashlight down the vent pipes on the end of a rope or wire. A drain auger or even spraying water with a garden hose will help you dislodge the obstructions further down the vent pipes.
Sewer Drain Clog
All of the drainpipes in your house's plumbing feed into a large sewer drainpipe, which connects to the sewer system or your house's septic tank. Look for the cleanout covers between your house and the sewer line or septic tank. If you see sewage water running out around the cleanout covers, your sewer drainpipe has a serious clog. If you do not see water running out of the cleanout, use a pipe wrench, clamping it onto the cleanout cover knob to twist the cover counterclockwise until it comes free. If you see debris or water sitting inside the cleanout, your sewer drainpipe has a clog. You may feed a power auger down the cleanout to knock out the clog, or contact a plumber for professional help.