This work presents two computational efficiency improvements for the hybridizable discontinuous Galerkin (HDG) fluid-structure interaction (FSI) model presented by Sheldon et al. A new formulation for the solid is presented that eliminates the global displacement, resulting in the velocity being the only global solid variable. This necessitates a change to the solid-mesh displacement coupling, which is accounted for by coupling the local solid displacement to the global mesh displacement. Additionally, the mesh basis and test functions are restricted to linear polynomials, rather than being equal-order with the fluid and solid. This change increases the computational efficiency dynamically, with greater benefit the higher order the computation, when compared to an equal-order formulation. These two improvements result in a 50% reduction in the number of global degrees of freedom for high-order simulations for both the fluid and solid domains, as well as an approximately 50% reduction in the number of local fluid domain degrees of freedom for high-order simulations. The new, more efficient formulation is compared against that from Sheldon et al. and negligible change of accuracy is found.