Removing roof rack from rails.
 

I have a 96 4runner and I'm trying to remove my factory rack from the rails so I can install a Thule rack, butI can't seem to get the rail end caps off of the rails without feeling like they're going to break or I'm going to bend or gouge something. I almost used a hammer out of anger and frustration, but you know where that would've led me.....to the Toyota dealer for some new rails.
Any suggestions or advice would be very appreciated. Thanks.