We have squirrels getting into our bird feeders. We have added cayenne pepper to the seed, but it does not seem to help.
Adding pepper to the seed and then not having an alternative for the squirrels to chose will not help- they will just get used to eating spicy food! Have a site away from the bird food just for the squirrels that does not have pepper in it. You can also use porcupine wire to encompass the top of the feeder. Have your feeder 8 feet above the ground and 6 feet away from any tree or building, this should help to keep the critters away.