A test and measurement instrument has an input configured to receive a signal from a device under test, a memory, a user interface to allow the user to input settings for the test and measurement instrument, and one or more processors, the one or more processors configured to execute code that causes the one or more processors to: acquire a waveform representing the signal received from the device under test; generate one or more tensor arrays based on the waveform; apply machine learning to the one or more tensor arrays to produce equalizer tap values; and apply equalization to the waveform using the equalizer tap values to produce an equalized waveform; and perform a measurement on the equalized waveform to produce a value related to a performance requirement for the device under test. A method of testing a device under test includes acquiring a waveform representing a signal received from the device under test, generating one or more tensor arrays based on the waveform, applying machine learning to the one or more tensor arrays to produce equalizer tap values, applying the equalizer taps values to the waveform to produce an equalized waveform, performing a measurement on the equalized waveform to produce a value related to a performance requirement for the device under test.