I mean the actual weights of the neural network. After the first iteration, each worker starts with the same weights but performs SGD updates on different subsets of the data. After each epoch, you re-seed the workers with the best weights.