The small matrix path integral (SMatPI) method is an efficient numerical approach to simulating the evolution of a quantum system coupled to a harmonic bath. The method relies on a sequence of kernel matrices that defines the non-Markovian dynamics of the quantum system. In the original SMatPI method, these kernels are computed indirectly through the QuAPI method. Instead, we focus on the definition of the kernel matrices and reveal a recurrence relation in these matrices. Using such a relationship, a tree based algorithm (t-SMatPI) is developed, which is shown to be much faster than straightforward computation of the kernel matrices based on their definitions. This algorithm bypasses the step to compute the SMatPI matrices by other path integral methods and provides more understanding of the SMatPI matrices themselves. Meanwhile, it keeps the memory cost and computational cost low. Numerical experiments show that the t-SMatPI algorithm gives exactly the same result as i-QuAPI and SMatPI. In spite of this, our method may indicate some new properties of open quantum systems, and has the potential to be generalized to higher-order numerical schemes.