Based on a novel numerical flux involving jumps of even order derivatives of the numerical solution, a direct discontinuous Galerkin (DDG) method for diffusion problems was introduced in [H. Liu and J. Yan, SIAM J. Numer. Anal. 47(1) (2009), 475-698]. In this work, we show that higher order (k≥4) derivatives in the numerical flux can be avoided if some interface corrections are included in the weak formulation of the DDG method; still the jump of 2nd order derivatives is shown to be important for the method to be efficient with a fixed penalty parameter for all p^k elements. The refined DDG method with such numerical fluxes enjoys the optimal (k+1)th order of accuracy. The developed method is also extended to solve convection diffusion problems in both one- and two-dimensional settings. A series of numerical tests are presented to demonstrate the high order accuracy of the method.