Goof Off works well for removing sticker or glue residue
http://www.valsparglobal.com/val/resident/goof-off.jsp
You might also try WD40 - I hear it works too just be careful not to get it on the bristles of the brush.
What are the handles made of? You could try to use an acetone nail polish remover, but, you may want to do a test on it first & be very careful when using it so you do not damage the handles.