There is no evidence to suggest that any part of the olive tree (Olea europaea) is poisonous to animals. Olive trees grow in the warm climates of U.S. Department of Agriculture plant hardiness zones 8 through 10. Some other, unrelated tree species with olive in their common names or "olea" -- Latin for olive -- in their botanical or common names, are toxic and may be mistaken for olive trees. These include oleasters and oleanders.
Russian olive (Elaeagnus angustifolia), which grows in USDA zones 3 through 7, is a deciduous tree or large shrub, with silvery leaves and fruits that look like olives. Though deer avoid the plants, Russian olive is not toxic to animals and the fruits are attractive to some wildlife. The plants are exceptionally vigorous and have been reported as invasive in some areas. Two other common oleasters with olive-like fruits are also nontoxic: Ebbing's silverberry (Elaeagnus x ebbingei), hardy in USDA zones 7 through 9 and oleaster (Elaeagnus macrophylla), hardy in USDA zones 7 through 9.
Oleander (Nerium oleander), which grows in USDA zones 8 through 10 is a large, evergreen shrub that can also be pruned to have the form of a tree. Grown for its fragrant, five-petaled flowers, oleander has been reported as invasive in some areas. Its species and common name, "oleander," derives from the resemblance between oleander and members of the olive family. Unlike olive trees, all parts of the oleander shrub are toxic if eaten by people or pets. Although animals rarely eat oleander, consider carefully if you want it near your family or pets.
To make sure a specific tree is an olive, which is nontoxic to animals, rather than an olive lookalike or a plant with "olive" in its name, check for key features. True olives are medium-size spreading trees, growing to 30 feet tall and wide and has invasive qualities in some locations. Members of the oleaster and oleander families do not reach that size. Russian olive has silvery leaves, whereas olive trees bear foliage that is gray-green above and silvery on the underside. Olives and oleanders both have fragrant flowers, but olive flowers are tiny compared to those of oleander. Oleaster flowers are bell-shaped, while olive flowers are not.
Other Toxic Trees
University of California at Davis Veterinary School lists only one evergreen tree-like species among the most toxic plants commonly eaten by pets. That is the jade tree or plant (Crassula ovata), hardy in USDA zone 11, a succulent with a woody stem that can grow up to 12 feet tall and 6 feet wide. It bears tiny white flowers, somewhat like those of olives, and fleshy green leaves. Among evergreen trees that are toxic to animals, but rarely eaten, is Japanese yew (Taxus cuspidata), which grows in USDA zones 4 through 7. The needled evergreens produce fleshy, berrylike fruits.