Social inference research demonstrates that when thinking about minds similar to our own, we anchor and adjust away from ourselves (Tamir & Mitchell, 2013). However, relational self theory (Andersen & Chen, 2002) suggests the possibility of using knowledge about others as an anchor when they are more similar than ourselves to a target. We investigated whether social inferences are made on the basis of significant other knowledge through an anchor and adjustment process, and whether this ability would be reduced under load. Participants answered questions about their likes and habits, as well as those of a significant other, a target similar to their significant other, and a yoked control. Differences between the significant other and similar target were related to participants’ reaction time, and found the opposite effect for self and target, suggesting anchoring and adjustment from the significant other rather than the self. However, inferences about the others tended to be more similar to the self under load, suggesting that the self serves as the primary source of information about others.