Noting that a bug is brown doesn't help much to identify it, given that such a large number of household pests display this color. As far as size goes, there is small, and there is smaller -- cockroaches are the only brown insects you wouldn't characterize as small. To determine whether to tolerate the brown bugs in your house or take immediate steps to exterminate them, you need more information about body features and habits.
Positive identification of the brown bugs that have moved onto your house is a prerequisite to extermination. You have to understand a pest's life cycle and eating habits to properly control it. The best way to positively identify a bug is to capture a specimen and compare it with online insect catalogues, but that's usually difficult to do. Simply noting important morphological features, such as whether it has wings or antennae, how many legs it has and its approximate size, can help with identification. You should also note the time of day and in which part of the house you see it. You can then use one of many online insect catalogues to narrow down the possibilities.
Ants and Termites
Ants are easy to identify when they form trails and head for any food you happen to leave out on the counter or the table. When you come across brown insects in the walls or basement though, you may be looking at a termite infestation, and that can have serious consequences for your house. You can distinguish ants from termites by their waists; ants have constricted, narrow waists while those of termites are broad. Moreover, ants have long front legs and elbowed antennae, whereas termites don't. Both ant and termite colonies produce winged individuals whose job it is to start new colonies. If you see one, you can identify it using the same guidelines.
Spiders.us lists 31 species of predominantly brown spiders that inhabit North America, and of these only a few pose a danger. The brown recluse, which has venom that causes skin necrosis, ranges in color from orange to dark brown, and it has long slender legs in proportion to its body. The Texas recluse is related, but it lives only in the southern part of Texas. Female black widows are usually jet black and have a red hourglass marking on their abdomens, while males may be brown and have red or light brown striping or small red or gray spots. The black widow's slightly less-venomous cousin, the brown widow, is predominantly brown. Some other brown spiders bite, but most are benign and, because they catch and eat insects, benefit the landscape.
Other Small Brown Insects
An infestation of brown bugs that are small enough to be difficult to see is often something to worry about. They could be bedbugs -- which are flat and oval and like to hide in your mattress. They may also be fleas, which are about 1/8 inch long and famously jump when you try to catch one. Insects with small reddish bodies that hitch rides on your pets could be ticks; these parasites ¼ inch long grow to as much as an inch when engorged with blood -- hopefully not yours. Microscopic brown bugs could be chiggers -- which bite -- or mites, which cause allergic reactions in sensitive individuals.