A new adaptive cell average spectral element method (SEM) is proposed to solve the time-dependent Wigner equation for transport in quantum devices. The proposed cell average SEM allows adaptive non-uniform meshes in phase spaces to reduce the high-dimensional computational cost of Wigner functions while preserving exactly the mass conservation for the numerical solutions. The key feature of the proposed method is an analytical relation between the cell averages of the Wigner function in the k-space (local electron density for finite range velocity) and the point values of the distribution, resulting in fast transforms between the local electron density and local fluxes of the discretized Wigner equation via the fast sine and cosine transforms. Numerical results with the proposed method are provided to demonstrate its high accuracy, conservation, convergence and a reduction of the cost using adaptive meshes.