The inverse scattering transform (IST) for the defocusing Manakov system is developed with non-zero boundary conditions at infinity comprising non-parallel boundary conditions — i.e., asymptotic polarization vectors. The formalism uses a uniformization variable to map two copies of the spectral plane into a single copy of the complex plane, thereby eliminating square root branching. The “adjoint” Lax pair is also used to overcome the problem of non-analyticity of some of the Jost eigenfunctions. The inverse problem is formulated in term of a suitable matrix Riemann-Hilbert problem (RHP). The most significant difference in the IST compared to the case of parallel boundary conditions is the asymptotic behavior of the scattering coefficients, which affects the normalization of the eigenfunctions and the sectionally meromorphic matrix in the RHP. When the asymptotic polarization vectors are not orthogonal, two different methods are presented to convert the RHP into a set of linear algebraic-integral equations. When the asymptotic polarization vectors are orthogonal, however, only one of these methods is applicable. Finally, it is shown that, both in the case of orthogonal and non-orthogonal polarization vectors, no reflectionless potentials can exist, which implies that the problem does not admit pure soliton solutions.