New techniques for integrable spin chains and their application to gauge theories