Updated to not attempt to drop the reference count on the last home directory object until we're about to actually give up the reference