Is human evaluation through surveys common in ML research?[R][D]

Hi,

I am a master's student working on a project where our fine-tuned AI model provides feedback on images. We are also comparing its feedback with outputs from Qwen/OpenAI APIs on the same images.

My postdoc and PhD mentors suggested creating a Google Form where participants compare the feedback and select which one is more accurate or useful. They asked me to collect around 60 responses and perform statistical analysis to determine which model performs better.

I would like to know:

1. Is this type of human evaluation common in machine learning research?
2. Is a sample size of around 60 participants reasonable?
3. How do researchers deal with participants who answer randomly or do not read the questions carefully?
4. If reviewer feedback later requires additional evaluation, is it common to conduct another survey?

I would appreciate any advice on possible issue I should discuss with my PostDoc before proceeding.

Thank you.