v10 SFT checkpoint comparison
Six prompts, one reply each, from best.pt of every v10 SFT run.
temperature=0.8, top_k=40, max_tokens=60. Replies are cut at the first turn separator, exactly as chat_continuous would return them.
Seeds are fixed per (prompt, penalty) and shared across versions, so differences between versions are the model, not sampling luck.
The runs
| dir | best.pt iter | val | config |
|---|---|---|---|
model_v10_sft | 100 | 2.612 | 14,922 rows (incl. chat_unattr), warmup 50 / 175 iters |
model_v10_sft2 | 125 | 2.352 | 6,424 rows, warmup 50 / 175, stopped at 125 |
model_v10_sft3 | 125 | 2.359 | 6,424 rows, warmup 50 / 175 + keep-all, stopped at 125 |
model_v10_sft4 | 99 | 2.397 | 6,424 rows, warmup 25 / 100 (anti-repetition run) |
Val is not comparable between the first row and the rest (different dataset → different split), and is anti-correlated with quality within a run. It's here for identification only.
Repetition
Mean duplicate-4-gram rate across the six replies (0 = clean).
| dir | penalty 1.0 | penalty 1.15 |
|---|---|---|
model_v10_sft | 0.012 | 0.000 |
model_v10_sft2 | 0.000 | 0.000 |
model_v10_sft3 | 0.000 | 0.000 |
model_v10_sft4 | 0.017 | 0.000 |
Replies — no penalty
How are you this fine morning?
- sft — I'm afraid you'll have to travel.
- sft2 — I'm afraid you'll be tired, my dear.
- sft3 — I have been to see a friend of mine, and I think I shall like to have a little talk with you about the future. Is your brother alive? I think so.
- sft4 — I'm afraid you'll have a rather dull evening.
Have you ever seen a ghost?
- sft — Not in my life, but I have heard that there is such a place as the Firs to-night.
- sft2 — I have seen nothing, but a ghost.
- sft3 — I have seen nothing, but a ghost.
- sft4 — Not in my life, but I have seen one. I have seen one.
What do you like to do on a rainy afternoon?
- sft — I like to hear you talk to me; I like to listen; I like to hear you.
- sft2 — I like to do it. It is a grand old place, and I shall be always on the spot.
- sft3 — I like to see a large body of men waiting outside the house; I shall be able to see about a few.
- sft4 — I was very much interested in this rainy afternoon.
I am afraid I have had rather a trying week.
- sft — I am afraid I have had rather a trying week.
- sft2 — I am afraid I have had rather a trying week.
- sft3 — I am afraid I have had rather a trying week.
- sft4 — I am afraid I have had rather a trying week.
Do you have a favourite book?
- sft — I have found it, but I cannot read it now.
- sft2 — I have found a most charming one.
- sft3 — I have found a most charming one.
- sft4 — I think you will like it, when we get home.
Tell me something about your family.
- sft — It is a long story.
- sft2 — It is a sad story. I hope we shall be able to go through the ceremony.
- sft3 — It is a sad story. I hope we shall be able to go through the ceremony.
- sft4 — It is a sad story.
Replies — repetition penalty 1.15
How are you this fine morning?
- sft — I was very glad that you had come. And--and I am glad of the opportunity.
- sft2 — I am very well, thank you. My journey is to-day, and the house is quiet.
- sft3 — I am very well, thank you. My journey is to-day. It would be so pleasant for me to sleep here.
- sft4 — I was very glad to see you. There is plenty to tell; I shall be all right by three o'clock, I dare say.
Have you ever seen a ghost?
- sft — I have been one of the old women. I am the last. If I had lived in this place before I was married, I should have died there.
- sft2 — I have been one of the old women. I am the last. If I had lived in this place before I went to bed for I should not have known it! But we have been very happy here, haven’t we?
- sft3 — I have been one of the old women. I am the last. If I had lived in this place before I went to bed for I should not have known it! But we have been very happy here, haven’t we?
- sft4 — I am not sure of that, my dear; but it is very remarkable. Who could have been asleep before?
What do you like to do on a rainy afternoon?
- sft — I think it is a great deal more suitable for the occasion. It is quite in keeping with our time when it comes.
- sft2 — I think it is most kind of you to come and see me. It will be the greatest pleasure to me if you will allow me to say so.
- sft3 — I think it is most kind of you to come and see me. It will be the greatest pleasure to me if you will allow me to say so.
- sft4 — I think it is most kind of you to come and see me. It will be the greatest pleasure to you if you will allow me to say so.
I am afraid I have had rather a trying week.
- sft — It is a mercy that you were in Rome, and that I was not there.
- sft2 — It is a pity that you were in Rome, my dear.
- sft3 — It is a great trial to me, but I don't see how I can ever be sorry for it now.
- sft4 — It is a great trial to me, and I don't see how I can ever be sorry for anything now.
Do you have a favourite book?
- sft — I have no favourite book. You see I had my own thoughts; perhaps I had. I know that the best books are in my library.
- sft2 — I like it as much as my husband. How do you like your own works? They are all very nice, and I like you very well.
- sft3 — I like it as much as my husband. How do you like the book?
- sft4 — I like it as much as possible, but not quite so well. I have seen some of the best books, and I should like to see them again.
Tell me something about your family.
- sft — I have heard nothing of them.
- sft2 — I have heard from a member of the family, and he has given me a letter to show you--a very interesting old lady. He is not young at all. Perhaps this is one of his daughters.
- sft3 — I have heard from a member of the family, and he has been talking to you about it. He has been telling me what happened on the Barrow at Mossnil's ball.
- sft4 — I believe you don't like it, but you must give me the benefit of my information.
TEST FROM CHAT CONTINUOUS
model_v10_sft2/best.pt || repetition-penalty 1.15
==================================================================
Exchange 1 (Jane remembers the last 3 exchanges)
==================================================================
You : Have you ever been to a seance?
==================================================================
--- 6 candidates @ temp 0.8 ---
- I have, and I shall be back in a week.
- I have not been much of anatomised.
- I have been to a seance. I know you told me to. But I will not be so hard as to believe in him.
- I have been to one of the smaller-warehouses, and I have seen my name."
- It seems that you have been to a seance. There is no one in the house but me, and I live with my sister-in-law.
- I am not aware of it. I have never been in the house. And I have always been with your mother as well. I hope you will enjoy yourself.
model_v10_sft3/best.pt || repetition-penalty 1.15
==================================================================
Exchange 1 (Jane remembers the last 3 exchanges)
==================================================================
You : Have you ever been to a seance?
==================================================================
--- 6 candidates @ temp 0.8 ---
- I know not about that, but I have heard of one very lovely girl who was very much admired. I am very happy to say I saw her.
- I was an old friend of the family, and I have been very happy.
- I have been there two years, and my name is Roxhall--Lucas!--but it's no good. It'll be all right----
- And I have been in the habit of going there every evening.
- I have always been a seance. It was very quiet and so lonely, so far away from the street and the street. I heard you say once that you never would go to the house again.
- I have been to one, and we were all together. Oh, how beautiful it is! It's a great pleasure to me to be there.
model_v10_sft4/best.pt || epetition-penalty 1.15
==================================================================
Exchange 1 (Jane remembers the last 3 exchanges)
==================================================================
You : Have you ever been to a seance?
==================================================================
--- 6 candidates @ temp 0.8 ---
- I have been at a seance.
- No. I have never seen one. It is the same with me. I'd like to live in another world.
- I have lived in a very little way. It was very pleasant, and I am glad of it. I never enjoyed anything so much as that evening.
- I have only just come in from the other place. But I was waiting for that opportunity, and that time is not yet. It seems that the seances are over.
- I have not been in the place for a long time. My sister, the dean's wife, brought me here.
- It’s all very well to tell us the right way to go now, but we might as well have been in the drawing-room. I don’t think we shall be here a whole week.