I think the best way here is to just use RVM & Bundler if
you want to really control what gems you have and make it
repeatable. You could just uninstall every gem on your box without
these tools but it's going to be more maintainable and easier to
manage for others to help you hacking on it if you use either (or
both, preferably Bundler).