Invasive plants - what does it matter if you have a single invasive plant in your own garden? Maybe it doesn't. Maybe it does.
Invasive plants are non-natives which escape into the "wild" or natural areas and then outcompete native plants. Invasives REPLACE native plants because they don't have enough natural predators or diseases to keep them in check like they do in their own native habitat. Well-known examples include kudzu and mile-a-minute weed. None of us would consider planting those in our own gardens, though.
How do they escape? Primarily through seed dispersal by birds/wildlife/wind. Invasive plants may also spread from roots that have not been properly disposed of. I found some climbing nightshade (pictured) at the edge of my garden last summer, and then again at a client's home in my neighborhood. It's an invasive and poisonous to humans, pets, and livestock (but not some birds - which is how it spreads). I tried to get all the berries, but I'm sure I missed some and will have to watch out for it again next year.
Some very popular landscape plants become invasive if they escape your garden. Burning bush is invasive. Bradford Pear is invasive. Mimosas, Norway maples, and barberries are invasive. Butterfly bush. Spirea. English Ivy. Wisteria. Pachysandra. Shrub honeysuckle. All invasive. You can find a full list on the DCNR website. Normally I don't say that non-natives should be removed until it's necessary to do so , but these invasive plants are ones that I would encourage you to remove and replace.
Sources: "Bringing Nature Home" - Doug Tallamy
DCNR Invasive Plants Fact Sheet
carlisle.gardener