I'm currently teaching a course on MIP, and out of interest I tried asking 4o about some questions I ask students. It could give the 'basic building blocks' (How to do x!=y, how to do a knapsack), but as soon as I asked it a vaguely interesting question that wasn't "bookwork", I don't think any of it's models were right.
I'm interested on how you seem to be getting better answers than me (or, maybe I just discard the answer once I can see it's wrong and write it myself, once I see it's wrong?)
In fact, I just asked it to do (and explain) x!=y for x,y integer variables in the range {1..9}, and while the constraints are right, the explanation isn't.
OK, but at that point you've told it basically everything, and this is a really basic book problem!
As another example I just gave it a network flow problem, and asked it to convert to maximum flow (I'm using the API, not chatGPT).
Despite numerous promptings, it never got it right -- it would not stop putting a limit on the source and sink (usually 1), which mean the flow was always exactly 1, here's the bit of wrong code (it's the last part, it's shouldn't be putting any restrictions on nmap['s'] and nmap['t'], as they represent the source and sink), and I couldn't pursade it this was wrong after several prods:
# Constraints: Ensure flow conservation at each vertex
A_eq = np.zeros((len(namelist), num_edges))
b_eq = np.zeros(len(namelist))
for i, (u, v, capacity) in enumerate(edges):
A_eq[nmap[u], i] = 1 # Outflow from u
A_eq[nmap[v], i] = -1 # Inflow to v
# Source 's' has a net outflow, and sink 't' has a net inflow
b_eq[nmap['s']] = 1
b_eq[nmap['t']] = -1
Sure, but that is nature of LLM prompting. It does take some doing to set up the right guardrails. It's still a good starting point.
Also a trick when the LLM fights you: start from scratch, and put guardrails in your initial prompt.
LLM prompting is a bit like gradient descent in a bumpy nonconvex landscape with lots of spurious optima and saddle points -- if you constrain it to the right locality, it does a better job at finding an acceptable local optimum.
I think this is just a case of different people wanting to work differently (and that's fine).
I can only tell this is wrong because I fully understand it -- and if I fully understand it, why not just write it myself rather than fight against an LLM. If I was trying to solve something I didn't know how to do, then I wouldn't know it was wrong, and where the bug was.
That's true, except an LLM can sometimes propose a formulation that one has never thought of. In nuanced cases, there is more than one formulation that works.
For MIPs, correctness can often (not always but usually) be checked by simply flipping the binaries and checking the inequalities. Coming up the inequalities from scratch are not always straightforward so LLMs often provide good starting points. Sometimes the formulation is something specific from a paper that that one has never read. LLMs are a way to "mine" those answers (some sifting required).
I think this the mindset that is needed to get value out of LLMs -- it's not about getting perfect answers on textbook problems, but working with an assistant to explore the space quickly at a fraction of the effort.
I'm interested on how you seem to be getting better answers than me (or, maybe I just discard the answer once I can see it's wrong and write it myself, once I see it's wrong?)
In fact, I just asked it to do (and explain) x!=y for x,y integer variables in the range {1..9}, and while the constraints are right, the explanation isn't.