A direct Arbitrary Lagrangian Eulerian (ALE) method based on multi-moment finite volume scheme is developed for the Euler equations of compressible gas in 1D and 2D space. Both the volume integrated average (VIA) and the point values (PV) at cell vertices, which are used for high-order reconstructions, are treated as the computational variables and updated simultaneously by numerical formulations in integral and differential forms respectively. The VIAs of the conservative variables are solved by a finite volume method in the integral form of the governing equations to ensure the numerical conservativeness; whereas, the governing equations of differential form are solved for the PVs of the primitive variables to avoid the additional source terms generated from moving mesh, which largely simplifies the solution procedure. Numerical tests in both 1D and 2D are presented to demonstrate the performance of the proposed ALE scheme. The present multi-moment finite volume formulation consistent with moving meshes provides a high-order and efficient ALE computational model for compressible flows.