What happens when I do not drive the mixer LO with enough RF power?

See What happens when you underdrive a mixer? for information about this.

Additionally, the intermodulation distortion IP3 will become worse, phase matching will become more difficult, and performance over temperature becomes more difficult.

It is always better to select the lowest drive level mixer available and add efficient amplifiers to boost your LO signal level than to starve the mixer.