This paper presents the development of parallel direct Vlasov solvers using the Spectral Element Method (SEM). Instead of the standard Particle-In-Cell (PIC) approach for kinetic space plasma simulation, i.e. solving the Vlasov-Maxwell equations, the direct method has been used in this paper. There are several benefits to solve the Vlasov equation directly, such as avoiding noise associated with the finite number of particles and the capability to capture the fine structure in the plasma, etc. The most challenging part of direct Vlasov solver comes from high dimension, as the computational cost increases as N^2d, where d is the dimension of the physical space. Recently due to fast development of supercomputers, the possibility of high dimensions becomes more realistic. A significant effort has been devoted to solve the Vlasov equation in low dimensions so far, now more interests focus on higher dimensions. Different numerical methods have been tried so far, such as finite difference method, Fourier spectral method, finite volume method, etc. In this paper SEM has been successfully applied to construct these solvers. SEM has shown several advantages, such as easy interpolation due to local element structure and long time integration due to its high order accuracy. Domain decomposition in high dimensions have been used for parallelization, these include scalable parallel 1D and 2D Poisson solvers. Benchmark results have been shown and simulation results have been reported for two different cases: one dimension (1P1V), and two dimensions (2P2V) in both physical and velocity spaces.