Fix violated invariant in LayerInfoTree