I have a few things I try first when I am testing out a new model. I will give it a coding problem:
Write a python script that generates a MIDI file. The midi file should be at least 2 minutes in length and contain multiple coherent parts in a reasonable song structure that is harmonically consistent. Each MIDI file it generates should be completely random in style, key, tempo, instrumentation, etc.
l have a few Python scripts that gather timing data for a variety of prompts, and since I am job hunting, I like to ask it what it thinks about my latest resume.
That last one is where DeepSeek R1 fell apart for me.
The prompt I use is with an uploaded PDF is “please review this resume and tell me the good and bad.” Once complete, I give a second prompt ” Take your feedback and output a revised resume. Do not make up any achievements or metrics.”
The screenshot shows a few huge misses:
- That first line isn’t in my resume anywhere. I studied Computer Science and Graphic Design so I was very sure that wasn’t right.
- The grammatical feedback is even worse. Similar to the first “excellence for customer-drive outcomes” just isn’t how I talk. It’s not something I would say and a quick search confirmed that line isn’t in my resume anywhere.
- The last one is maybe the most glaring. In the words I didn’t write, it found fault with the use of the word “outcomes” and tells me I should probably use the same word because it will make more sense.
I love this technology and I use it for so many things now, but I sure hope this isn’t trusted without some confirmation. I’ve seen wild hallucinations in models before, but this is not wild. It’s benign and the kind of thing someone in a hurry wouldn’t think to double-check.
One other note, it’s pretty important to understand where the model comes from and what the biases are. DeepSeek was founded by Liang Wenfeng, one of China’s top investors. His hedge fund, High-Flyer, finances the company’s AI research. That’s probably why you get interesting answers when you ask who owns Taiwan.
