Which model are running for code assistance?

[Easy Rhino]

I have tried deepseek-coder-v2 for chat and codellama, starcoder2, and qwen2.5-coder 1.5B for code completion and all of them are horrible compared to straight up copilot.

 

[Vya Domus]

All the models I tried fail in pretty much the same way like down to the exact same mistakes, pretty sure they're all trained on the same data so it's kind of pointless trying to find the best one. Copilot is also just a wrapper for ChatGPT.

 

[Easy Rhino]

All the models I tried fail in pretty much the same way like down to the exact same mistakes, pretty sure they're all trained on the same data so it's kind of pointless trying to find the best one. Copilot is also just a wrapper for ChatGPT.

Yea, it is discouraging that the free models are all pretty bad. I mean, none of them are all that great at boiler plate. Copilot recognizes what I am trying to do and it adjusts on the fly. Pretty great for getting the boring stuff out of the way.

 

[R-T-B]

Keep in mind CoPilot scans and uses your code for model training purposes. That's a nonstarter for me.

The reason its better is it has used all of public github for training.

 

[Solaris17]

Yea, it is discouraging that the free models are all pretty bad. I mean, none of them are all that great at boiler plate. Copilot recognizes what I am trying to do and it adjusts on the fly. Pretty great for getting the boring stuff out of the way.

I have a sub to chatGPT. I havent had time to spin any of the more vram intensive models up, but I did go into some of the mid range ones that say require north of 8. They get better; but some like deepseek, just arent it for me. It seems to weigh certain answers above others. I will try to explain.

For example, one of my tests is to ask for a batch file. I specify batch in my prompt. Deepseek in everycase would try to give me powershell. Powershell is more modern, better documented and preffered today. The script itself was fine, but I pretty much give it a -100 because I specified BATCH. It would give it to me, but I had to insist a few more times.

The others specifically starcoder didnt have that "presumptuous" issue. But it started to fall flat when I started asking it more complex things.

I am hoping when I have time and start running the big big models it improves. Llama isnt bad, but it does start to poop syntax issues when things get hard. It loved putting ";" where it doesn't belong.

That said, I think a lot of these take a learning curve from those that use them too. You have to know how to "speak" to your model of choice.

 

[igormp]

I have tried deepseek-coder-v2 for chat and codellama, starcoder2, and qwen2.5-coder 1.5B for code completion and all of them are horrible compared to straight up copilot.

Given that the best models are the bigger ones, and I often don't have that much free ram/vram on a daily basis, I pretty much always use claude with cursor.
When offline or using a local model, I often use either codellama or deepseek-coder (the older model) because that's what i have downloaded already.

Keep in mind CoPilot scans and uses your code for model training purposes. That's a nonstarter for me.

The reason its better is it has used all of public github for training.

You can opt out of that, fwiw.

 

[R-T-B]

You can opt out of that, fwiw.

Aware but don't really buy that it works frankly. Once its in the model, they can't pull it out, and with the default being "on" its probably scanned nearly immediately.

But I will admit maybe it helps future commits, so I will admit I use that setting for my little bit of open source projects.

 

[igormp]

Aware but don't really buy that it works frankly.

You can monitor the network calls that are done.

Once its in the model, they can't pull it out, and with the default being "on" its probably scanned nearly immediately.

That's not really how it works. If you opt in, the data is sent back to OpenAI/MS, and they might do further fine-tunings with the new data in newer versions of the model, but it's not "shoved" into the model instantly, nor immediately used for the internal weights of the model.

 

[R-T-B]

Suffice to say, my confidence that they'd keep this transparent remains low. But we are getting offtopic. I did not mean to start AI fearmongering here, appologies.