Shopping Online
You can find safety pins just about anywhere, but buying them by the gross is definitely the cheapest option. 
Flat backed, or Bar Pins, are a great deal more expensive, so I only use them for super fancy SWAPS that a regular pin simply won't work with. 
They have many names, but my favorite is google eyes. It's best to have a variety of sizes on hand for SWAPS, and they even have fancy eyelashed ones, monster ones, and more now! I love the options. They can be found at crafts stores, Wal-Mart, and even the dollar store when you're lucky, but overall, it's probably cheaper to buy a big set than many bags of differing sizes. 
Glitter is a craft staple for SWAPS, and Amazon is by far the best place I've found so far, with so many options. You can get so many different colors from one purchase or get just the color you need in a large container. You'll use it. It's glitter. It's unavoidable. 
I love these little vials. They add something special to SWAPS and they'd darn cute! 
Embroidery floss is also a staple of SWAPS, used to hold things together in my booklets. I think the hole punched method is better, visually, than a booklet simply stapled together. 
I primarily used pony beads, because they're easy to thread on whatever you're using and also because they're low cost. They even have bags of solid colors, which come in handy when making things with a specific color scheme. Much easier than digging through a mixed bag looking for that last brownie brown bead for your craft. 
Craft foam has so many uses, and I love creating things out of it. Some colors are harder to find than others, and Junior green I've only found in the 9x11 sized sheets or smaller. Michaels is usually my go to place for foam. Though Wal-Mart carries some, Michaels has a much greater selection. Amazon is good for multipacks, but can get expensive for individual sheets. Always watch the quantity while pricing. 
I like to buy baggies in bulk. It reduces their per bag price and is also more convenient. More times than not, I've had trouble getting my hands on enough baggies in the actual stores. They're often out of stock or there in limited quantities. 
Kids glue and school glue may cut it with paper based SWAPS, but not with craft foam. For that, you need a stronger glue, or you risk things falling apart quickly. Aleene's Tacky Glue has never let me down.