This paper presents a fourth-order kernel-free boundary integral method for the time-dependent, incompressible Stokes and Navier-Stokes equations defined on irregular bounded domains. By the stream function-vorticity formulation, the incompressible flow equations are interpreted as vorticity evolution equations. Time discretization methods for the evolution equations lead to a modified Helmholtz equation for the vorticity, or alternatively, a modified biharmonic equation for the stream function with two clamped boundary conditions. The resulting fourth-order elliptic boundary value problem is solved by a fourth-order kernel-free boundary integral method, with which integrals in the reformulated boundary integral equation are evaluated by solving corresponding equivalent interface problems, regardless of the exact expression of the involved Green's function. To solve the unsteady Stokes equations, a four-stage composite backward differential formula of the same order accuracy is employed for time integration. For the Navier-Stokes equations, a three-stage third-order semi-implicit Runge-Kutta method is utilized to guarantee the global numerical solution has at least third-order convergence rate. Numerical results for the unsteady Stokes equations and the Navier-Stokes equations are presented to validate efficiency and accuracy of the proposed method.