Rubber flooring that has been glued to the subfloor can be really difficult to scrape off, whether it is old carpet under-padding, rubber-backed tiles or some other thick rubber. You'll make more progress scraping if you can loosen the adhesive first. Plain water will sometimes work, but as the adhesion gets tougher, so must the solvent. You'll also do well to have a long-handled, 4-inch razor scraper on hand.
Heat thick rubber with a blow dryer and let the heat soften the adhesive. Work a scraper under the rubber and pry it up, pushing the scraper deeper in as you go. You should be able to remove most of the surface layer of the rubber in this way, leaving only spots and residue to clean up.
Wet down rubber carpet under-padding with warm water, then scrape it away with a floor scraper. The water will dissolve the under-padding, leaving spots and residue which you can clean up.
Spread mineral spirits on spots of rubber and adhesive that remain after the bulk of the rubber has been scraped away. Let the thinner soften the adhesive, then scrape the floor clean with a floor scraper. If mineral spirits don't loosen the adhesion, try lacquer thinner, latex paint remover or spray-on metal lubricant.
Clean the floor with warm soapy water to remove residue left by the solvents.