Towards Adaptive Active Visual Agents