Not all mold you may find in your how is toxic, most molds are not. Some people, however, do suffer from mold-related allergies, which is more common in those with asthma, infants and children, the elderly, people with chronic lung disease or compromised immune systems.