Speed of convergence of the average of certain random matrices