It's complicated and I'm not actually a scientific person in real life. But it depends. As I understand it, it's the actual molecules that matter. Organic or inorganic, the ions seem to be the thing. Once you get to the molecular level organic and inorganic, well the lines blur. For instance, in the case of food borne illness it's usually the toxins that the bacteria excrete that harm you. Whatever that compound might be. But this is not always true. And indeed organic things often evolve to harm other organic things. But, lead is inorganic and it can harm you. OTOH glass is not an organic poison. But it can be sharp and harm you that way.

Plastic is also a a very wide net. Some plastics are harmless or even non reactive. But certain plastic compounds can give off smaller molecules like Bisphenol A which can give of toxins that affect the human endocrine system.

Essentially, as I said, it depends. I hope that helped.

Actually this question made me look up certain things. It's always nice to learn. Organic poisons have carbon somewhere in their molecular structure. While inorganic poisons are usually heavy metals like lead or mercury. But combining certain metals with oxygen can make them even more toxic.

Not sure if you'll see this but thanks!