I'm more likely to discard the peels (than the cores) just for that reason, too.

It rather depends though - I'm more likely not to want to eat the peels if it's a shiny foreign apple with a shiny, flawless, skin, than when it's a local apple which has a funny shape and lots of little skin issues. Somehow I feel that if it looks so imperfect, then there might be a smaller chance of it having been poisoned too much!