We investigate the relationship between the gap between the energy of the ground state and the first excited state and the decay of correlation functions in harmonic lattice systems. We prove that in gapped systems, the exponential decay of correlations follows for both the ground state and thermal states. Considering the converse direction, we show that an energy gap can follow from algebraic decay and always does for exponential decay. The underlying lattices are described as general graphs of not necessarily integer dimension, including translationally invariant instances of cubic lattices as special cases. Any local quadratic couplings in position and momentum coordinates are allowed for, leading to quasi-free (Gaussian) ground states. We make use of methods of deriving bounds to matrix functions of banded matrices corresponding to local interactions on general graphs. Finally, implications on entanglement-area relationships in harmonic lattice systems are outlined.