Consider the scattering of a time-harmonic electromagnetic plane wave by an arbitrarily shaped and filled cavity embedded in a perfect electrically conducting infinite ground plane. A method of symmetric coupling of finite element and boundary integral equations is presented for the solutions of electromagnetic scattering in both transverse electric and magnetic polarization cases. Given the incident field, the direct problem is to determine the field distribution from the known shape of the cavity; while the inverse problem is to determine the shape of the cavity from the measurement of the field on an artificial boundary enclosing the cavity. In this paper, both the direct and inverse scattering problems are discussed based on a symmetric coupling method. Variational formulations for the direct scattering problem are presented, existence and uniqueness of weak solutions are studied, and the domain derivatives of the field with respect to the cavity shape are derived. Uniqueness and local stability results are established in terms of the inverse problem.