s ground state is known to be very sensitive to its precise stoichiometry. Powder samples, produced by solid state synthesis at relatively low temperatures, tend to be stoichiometric, while single crystals grown from the melt tend to display weak "stuffing" wherein $\mathrm{\sim 2\%}$ of the $\mathrm{Yb^{3+}}$, normally at the $A$ site of the $A_2B_2O_7$ pyrochlore structure, reside as well at the $B$ site. In such samples $\mathrm{Yb^{3+}}$ ions should exist in defective environments at low levels, and be subjected to crystalline electric fields (CEFs) very different from those at the stoichiometric $A$ sites. New neutron scattering measurements of $\mathrm{Yb^{3+}}$ in four compositions of $\rm Yb_{2+x}Ti_{2-x}O_{7-y}$, show the spectroscopic signatures for these defective $\mathrm{Yb^{3+}}$ ions and explicitly demonstrate that the spin anisotropy of the $\mathrm{Yb^{3+}}$ moment changes from XY-like for stoichiometric $\mathrm{Yb^{3+}}$, to Ising-like for "stuffed" $B$ site $\mathrm{Yb^{3+}}$, or for $A$ site $\mathrm{Yb^{3+}}$ in the presence of an oxygen vacancy.