ObjectPoseObserver: Refactor housekeeping tasks