You are here

minor errors and time bonus, help?

On the current round, I made a mistake on the Oasis puzzle which was, for now, ruled non-minor in the sense of ruling out time bonus. In a way I'm fine with that, because it was a real mistake, not just a code entry error. However, my understanding of the (spirit of) the rule is that if you've made a true effort to finish the test, handing in solutions to every puzzle, and just happened to break one of them, that shouldn't deprive you of time bonus.

I see that the fact that there were just 5 huge puzzles on this test skews things here, but in that case I'd rather argue that it's a mistake to run the test like this.

Anyway, for clarification, I'd love to know what kind of errors other players had ruled minor in the first round (e.g. Ken Endo, Philipp Weiß and Prasanna Seshadri in round 1A). Would some of you be willing to share? For me, this is not primarily about getting those points (though I'd be happy to have them) and more about clarifying and discussing the rule.

My mistake by the way: I isolated a white cell in row 3 of the Oasis puzzle when finshing things up in the top left corner, resulting in black cells in 13 instead of 25.

I totally share your opinion on the matter. I was similarly frustrated with some rulings in sudoku GP.

During the Snyder´s supervision, decisions were made exactly in the spirit you describe. When you check the results for years 2014 and 2015, everyone with time bonus higher than 10 minutes received partial credit. It makes no sense to intentionally submit a wrong solution, especially for the low-point puzzles, you lose points for the puzzle plus 20% of the time bonus. You will be better off when you actually solve the puzzle, or at least the expected value is always higher.

However, if you have only 5 minutes left and you feel you will be unable to solve the last puzzle before the limit, you could fill the answer code 123456789 in order to claim 80% partial bonus. This practice needs to be prevented.

I think that the rules were originally written so vaguely so that the GP director had a tool to prevent this type of angle shooting. Later new directors started to interpret the rules in their own way. This was a big problem in 2016 sudoku season where some players lost 20-30 minutes bonuses due to a single mistake.

As there is a new director this year, I would like to hear his interpretation of the "typo or small mistake" rule.

Regarding the question from the previous post, I can only say that avoiding any disadvantages I am trying to have the same criteria in such cases. The answer key 123456789 certainly will not be accepted for bonus, except if this is correct key or I determined that there is a typo or small mistake.
My conclusion after the first two rounds is that it will be much easier for me with more limited space for interpretation.

Nikola Zivanovic
Competition Director,
WPF Sudoku GP 2017

For what it's worth, i don't think that losing 20-30 minutes bonus is necessarily a problem. The main argument is that although bonus time is important, you still have plenty of time to check your answers - you don't accidentally hit the claim bonus button.

What would you say to the solvers who do sacrifice a bit of bonus time to check their answers properly? An overly generous partial bonus policy is completely unfair to these solvers.

I cannot speak for 2017, but in 2016 a consistent policy, which I though was fair, was applied to determining 'minor' errors. I chose not to reveal the exact details of this, and simply apply it uniformly to all possible cases of n-1 out of n correct. Given the 2017 season is live, I'll only give more details if Nikola/Wei-Hwa are happy for me to do so.

Well, if you check your solutions and find a mistake, you avoid losing points for the puzzle plus 20% of the time bonus. Isnt this enough for you?

Do you realize that for some puzzle types it is hard to check that the solution doesnt break any constraint? Wouldnt it be weird to solve non-consecutive or antiknight in 2 minutes and then spend another 2 minutes checking?

If there were rules for awarding partial bonuses, what was the point of keeping them secret? It only makes the competition less transparent.

If everyone made mistakes consistently, I might buy your argument that checking is hard. But it's clear there are some solvers who make very few mistakes and there are others that don't. Of course anyone might make a mistake every now and again, but that's why the minor error rule is there (certainly many mistakes do qualify as 'minor' and partial bonus is awarded), and even if it's not a minor error then you still have 2 chances to discard.

As Nikola says, it is not the easiest thing to try and determine when all you have are two answer codes.

I would also suggest to solvers who are losing bonus time regularly that this isn't normal compared to other solvers, and that perhaps they should perhaps take responsibility for improving this aspect of their solving.

To address the point about secrecy, I think you are thinking about this the wrong way. To determine minor errors is up to the discretion of the director. Whilst I used a rule of thumb to help me, there is no obligation to reveal this. The main thing is that the spirit of the rule is clear: typos and small mistakes.

In practice with only 2 answer codes to go by this is not a trivial thing to determine. Certainly, no simple algorithmic rule can work in 100% of cases. If such a rule was publicised then this would only be unhelpful because there may be qualifying cases excluded by the rule or disqualifying cases that are included. Leaving this to the discretion of the director means that a fair judgement can be made taking all factors into account. It's what the director is there for.

If you have specific examples in mind, perhaps try looking at just:

The answer codes with an error
The correct answer codes

This might help illustrate why this element of director discretion is necessary

As I see it, the purpose of having the Time Bonus is to give a reward to the players who solve all the puzzles.

In an ideal world, we would have a team of graders who would magically look at everyone's papers and know whether they are solved correctly or not within seconds. But we don't have the budget for that.

So, there is no practical way to know for certain whether a puzzle is solved -- instead, we have players convert their solutions into "answer"s. A player might solve a puzzle correctly but end up with the wrong answer. A player might make a mistake when solving a puzzle but end up with the correct answer.

Given that, the only fair way is to redefine what "solving a puzzle" means -- it means getting a valid answer. For the GP, you are scored based on whether you have a valid answer. I sympathize with you if you solved the puzzle but made an transcription error and have the wrong answer, but I cannot help you without being unfair to the other solvers.

Having the time bonus for perfect solvers is of course also fair, and allows us to distinguish between the best solvers while still keeping the competition somewhat balanced for the middle solvers (so that puzzle selection skills is not as strong a factor).

Having the "80% rule" is inherently UNFAIR. It always has been, regardless of how loose or tight we make the rules. Since we've redefined "solving the puzzle" as "submitting the right answer", if you submit the wrong answer, you are just wrong, no matter whether it is one digit off or completely incorrect in every digit. Giving you ANY points would be unfair to other solvers who checked their work, or other solvers who made small mistakes on two puzzles.

The only reason we have it, in my opinion, is that it mitigates one inherent problem that time bonuses cause -- which is that the last puzzle you solve is worth a disproportionately high amount of points when compared to the other puzzles. If the round is 500 points, and you are so fast there is 200 points of time bonus, then your last 30-point puzzle is really a 230-point puzzle. That is a lot of pressure on just that one puzzle. This creates a big discontinuity in scores, where just a small amount of puzzle-solving ability creates a cliff in the overall scores.

So in this one case, we give a small concession to try to smooth that discontinuity. In past GPs, it has been vague where that concession is. For this year, I tightened up the vagueness a little bit: which is that if you haven't solved the original puzzle, you don't get the points. The points only come if it is plausible that you solved the original puzzle but made an error when converting to an answer key. It's not always about number of symbols you get wrong -- if I asked for rows 2 and 7, and the solver submitted a matching answer key for rows 2 and 6, I'd likely give them the points. Because the puzzles are so different in nature and how the answer extraction works, I can't have a consistent rule that applies even within the same round, let alone between rounds or years.

But as I see it, the 80% rule has always been unfair. If you got the 80% time bonus, it's not because you deserved it. It's a consolation prize to solvers who happen to get lucky in a rather specific situation, and it should be something that is truly about luck -- it should be impossible for a player to aim for it on purpose. It is like the 7th tie-breaker in FIFA rules, where it comes down to a coin toss. We can hope that in the long run the luck averages out. Any argument that says "well, I *should* have received the 80% points" is missing the reason for having that rule in the first place.

This is the second time I benefit from this rule of 80% bonus.
(It seems that I loose accuracy with age and perhaps I should have to resign and admit that the peak of my "career" as a sudoku player was 2015-2016, one of these year with no proper WSC and the other one I didn't take part in WSC because I had no guarantee 2015 issues were fixed... that's life)

I have to say I'm happy that the rule exists, but I understand Wei-Hwa's point when he said "the 80% rule has always been unfair". I understand players who ask some clarifications about the rule, too.
The discomfort comes from that a player who makes a mistake and solved all other puzzles correctly will not loose the same amount of points (he looses points of the puzzle + bonus points) than a player who haven't solved all puzzles (he only looses points of incorrect puzzle).

I think we should think about the possibilities offered by the fact that these tournaments are online.
I'm not a big fan of the "instant grading" system on LMI, because I think it rewards too much players who make mistakes. However possibilities exist to improve the rules.

Here is my proposal of change for next year:

Once you claimed bonus,
-Either all puzzles are correct and there is no change with actual rule,
-Either some codes are wrong and in this cases the system tells you which code(s) is wrong, and you can continue to submit answers.
When you finished to fix mistake(s) and submitted what you think are correct answers, you can click the "claim bonus" button again (or another one, perhaps it would be clearer) and if then all puzzles are correct you'll earn 100% bonus points according to this time submission BUT NO POINT FOR PUZZLE(S) SOLVED INCORRECTLY THE FIRST TIME YOU CLAIMED THE BONUS.

-I think it should be limited to only one chance to fix mistakes (you can only click once the new button), hence players are not encouraged to guess the right code.
-With this system there is no more rule depending on the judgement of the director.
-It means that if you solved all puzzles correctly when you claim bonus (first time) you'll know that it's correct.
-If your mistake is a typo or a small mistake that you can fix quickly, you'll loose only very few bonus time, but you'll loose the points allotted for this puzzle,
-If in contrary the mistake is more serious, you'll have to solve the puzzle again and loose not only points allotted to the puzzle, but bonus points according to the time you need to fix the puzzles.
-The grace time will allow players to score bonus points only and in no case to have points for incorrect submission on the first click on "claim button"
-In all cases, bonus points are given according to the time needed to solve all puzzles correctly !

I'm curious to know if my proposal sounds a good idea to players.
I don't think it would be very hard to make technically.

The only problem I see now is that there are still issues with the claim bonus button (and there is a risk the same problems will recur with the second button, too): On this round, I had to submit 2 times my answer before the claim bonus button was active, but I heard that a player had to refresh the page more than 5 times before it works.


Hi Fred,

I think perhaps your proposal needs a more thoughtful response than I can muster at this late hour, but one comment I'd like to make is that the more I think about the LMI instant grading system, the more I like it.

I've rationalised it as a bit like a world championships play-off, where each solver has his or her own personal checker. As long as the instant grading penalty for a wrong submission is sufficiently penalised (it doesn't feel the balance here is always right at LMI) then it seems like a fair system to me. It doesn't quite replicate the offline experience where you hand in your paper - but then on the other hand there are many other differences between an offline competition and an online competition.

Since we are online we have better chances to use technology - so if it's possible, and it's fair, then why not?

I agree, the Instant Grading is the greatest invention since the discovery of Sudoku :)
Another indisputable advantage of the LMI site is the preliminary results table, but I am not sure whether it will be proof against cheating here as well.

Without going up to preliminary results, I would be happy to see the "my solution" page right after finishing to play.
I think it's the case on UKPA online tournament system, so it should be technically possible.
A non-negligible advantage would be that protests would be possible during the tournament days, and the final ranking could be published earlier.

Instant grading could work here, yes.
It would be a big change of paradigm, I don't know if GP organizers are ready to do this big change.

It's just a matter of personal preference, but I prefer when there is no interaction with the system during the tournament. Perhaps it's also linked with my way of playing: normally I solve all or a maximum of puzzles before entering the codes. "Instant grading" is perhaps better if you submit codes after each puzzle solved and I think it is more appropriate for an online solving mode (like the existing one on LMI) than a paper solving mode. But I say again this is just personal preference.

The real question is: On which basis can you fix the number of points a player should earn if he needs to submit solution twice? third? I think this is a highly arbitrary factor, what sounds like a small issue in my opinion.

There is a simple solution how to deal with time bonuses, just add few more puzzles into each GP round, so nobody can solve all puzzles in time. :)

Adding more puzzles so that nobody can solve all puzzles in time is generally a bad idea, in my opinion. It means that the competition becomes less about general-solving skill, and more about puzzle-selection strategy.

It is also rather demoralizing, especially for the lower-ranked players.

I'd love to have automatic grading, but I'm not sure we have the budget to hire someone who can write the requisite software.

Puzzle-selection strategy
- for 95% of the players it already works that way in the current system. Players have to choose what to solve. Just top competitiors "are forced" to solve all.

Is more puzzles demoralizing for lower-ranked players?
I see it exactly opposite - now after solving few puzzles suitable for their skill level, they may get to the point, where is nothing else to solve (remaining puzzles in the set are too difficult for them, so they quit solving). If we add more puzzles (and these could be mostly easier ones), it will keep motivating them to continue solving. Also this works nicely for better solvers, added easier puzzles can works as filler for last minutes of given time limit.

But main point is, there wont't be problems with:
-problematic claim bonus button (all that players have to do is just submit answers)
-losing time bonus value after mistake (then there won't be crazy situations in rankings like: player1 claiming bonus with 30 spare minutes and one error has same score as player2 submiting with 1 spare minute and one unsolved puzzle. Player1 should have chance to score points in these 30mins)

I am not entirely against time bonuses. It is nice challenge for players to try finish all puzzles. But in my opinion, rankings and overall solving experience will be better without them.

In my case, it was an 1-digit typo, and I PREDICTED this would be credited. If this was called as a kind of coin-toss, then I would feel coins had been thrown too often than expected.

Rule is rule, and I am okay as long as judgement is consistent. But even if I was saved by partial credit, I do not like it in that current system works only for finished players, not everyone.

I don't think it's an issue that the rule applies only for players who have finished, because there is no partial credit for the incorrect puzzle (in all cases you'll have 0 points for incorrect puzzles), it is partial credit concerning bonus points linked to time saved (thus it's absolutely fine that it concerns only players who saved time).

TiiT's picture

I think the best solution here for everyone is to try entering the answer keys more carefully each time. and problem solved :)