In this paper, we propose a new partitioned approach to compute fluid-structure interaction (FSI) by extending the original direct-forcing technique and integrating it with the immersed boundary method. The fluid and structural equations are calculated separately via their respective disciplinary algorithms, with the fluid motion solved by the immersed boundary method on a uniform Cartesian mesh and the structural motion solved by a finite element method, and their solution data only communicate at the fluid-structure interface. This computational framework is capable of handling FSI problems with sophisticated structures described by detailed constitutive laws. The proposed methods are thoroughly tested through numerical simulations involving viscous fluid flow interacting with rigid, elastic solid, and elastic thin-walled structures.