We propose a hybrid spectral element method for fractional two-point boundary value problem (FBVPs) involving both Caputo and Riemann-Liouville (RL) fractional derivatives. We first formulate these FBVPs as a second kind Volterra integral equation (VIEs) with weakly singular kernel, following a similar procedure in [16]. We then design a hybrid spectral element method with generalized Jacobi functions and Legendre polynomials as basis functions. The use of generalized Jacobi functions allow us to deal with the usual singularity of solutions at $t = 0$. We establish the existence and uniqueness of the numerical solution, and derive a $hp$-type error estimates under $L^2(I)$-norm for the transformed VIEs. Numerical results are provided to show the effectiveness of the proposed methods.