Build an x-tal notch filter for 144 MHz.
Background
With an X-tal notch filter it's possible to attenuate the modulated SSB signal by 30 dB while leaving the lower noise sideband unaffected. The sensitivity can be increased 30 dB and noise sidebands measured directly using a spectrum analyzer.
Diagram of x-tal notch: X-tal is a 7'th overtone 144,300 MHz. L1 has 5 turns of 0,7 mm Cu wire on a 5 mm form with tuning slug. All components are mounted in a small metal box with 2 BNC connectors.
Filter response. Note the "ragged" response above the notch frequency,
due to parasitic resonances at the 7'th harmonic. Only the low frequency
side is useful for measurements.
Adjustment of L1 affects both notch depth and width, as well as response
on the lower side. When adjusting aim for a steep notch and relative
flat response the first 200 - 300 kHz lower in frequency.
Enlarged picture of low side response. The measuring range (-3dB) goes from -30 kHz to -300 kHz.
How to measure
Connect the transmitter to an attenuator or coupler, then through the notch into a spectrum analyzer. Key the transmitter and set the frequency at 144,24 MHz - where the notch filter has the least attenuation. Adjust for full scale with full PEP output. Now move to the notch frequency, and put the transmitted signal exactly in the notch. This must be done with care as the peak of the notch is only 2,5 - 3 kHz wide.
Now full PEP output will be 30 dB down the screen, so the sensitivity can be increased by 30 dB and it should be possible to see those noise sidebands and IMD products that are more than 30 kHz away. Only the lower sideband gives a well defined picture, but it can normally be assumed that the two sidebands are identical.
If the extra 30 dB sensitivity is not enough, it is possible to increase the analyzer gain by extra 10 dB. This will bring the SSB signal above top of the screen, but as we are - in principle - measuring on a single signal, no extra components are generated by the analyzer.