The spectral form factor of quantum chaotic systems has the familiar `ramp$+$plateau' form, on top of which there are very strong fluctuations. Considered as the modulus of the partition function in complex temperature $\beta=\beta_R+i\tau=\beta_R+i\beta_I$, the form factor has regions of Fisher zeroes, the analogue of Yang-Lee zeroes for the complex temperature plane. Very close misses of the line parametrized by $\tau$ to these zeroes produce large, extensive spikes in the form factor. These are extremely sensitive to details, and are both exponentially rare and exponentially thin. Counting them over an exponential range of $\tau$ is a large-deviation problem. The smooth solution averaged over disorder does not allow us to locate the spikes, but is sufficient to predict the probability of a spike of a certain amplitude to occur in a given time interval. Motivated by this we here revisit the complex temperature diagram and density of zeros of the Random Energy Model, and a modified model in which we introduce level repulsion.