In this paper, we investigate the effect of annealing gas compositions on the morphological evolution of heteroepitaxial silver island films grown on silicon substrates. We show that the presence of oxygen during thermal annealing leads to the formation of more isotropic silver islands with enhanced rates of dewetting and coarsening. This is explained by the results of morphology analysis and density functional theory calculation showing that a larger extent of oxygen adsorption under oxygen annealing ambient decreases the surface energy anisotropy of silver and the surface diffusion barrier of silver adatoms and clusters.