PE issues

The forum to report every bug you find or tell us what you'd like to find in UVa OJ

Moderator: Board moderators

Locked
User avatar
little joey
Guru
Posts: 1080
Joined: Thu Dec 19, 2002 7:37 pm

PE issues

Post by little joey » Mon Dec 11, 2006 5:57 pm

Great! My list of solved problems is shorter by 60 in one swoop! If only FIFA had the same rejudging policy for their tournaments, Holland would become World Champion for 1978, for sure!

I agree that P.E. should be no longer accepted, but I think it would have been just to the users, who spent many hours on trying to decode faulty, ambiguous or incomplete output diescriptions, or to comply with wrong judge files, to first make sure the errors were corrected, before confronting them with these long lists of unaccepted solutions.

I'm only halfway through my list, and these are my notes:

132: Output format not specified.

139: ? I think I follow the specs, but get PE.

175: The sample output has a space after the ':' in case of an empty list, but to get rid of the PE omit the space.

221: There must be a blank line between cases, but to get rid of PE print no blank lines.

312: ? I think I follow the specs, but get PE.

324: Quote: "The output format isn't too critical", but I get PE by using the exact format of the sample output.

334: To get rid of the PE print an extra space at the end of some lines.

447: Quote: "Output of two cases will be separated by a blank line", but to get AC also print a blank line after the last case.

483: Don't put a newline at the end of the last line.

526: Although nothing is said about blank lines, you should print one between each case.

531: ? I think I follow the specs, but get PE.

There may be more; I haven't resolved all my PE's and I didn't mention the ones already discussed here.

I know you're working hard to make things better, but IMO this action was premature.

Jan
Guru
Posts: 1334
Joined: Wed Jun 22, 2005 10:58 pm
Location: Dhaka, Bangladesh
Contact:

Post by Jan » Mon Dec 11, 2006 8:08 pm

I want to mention some problems, too.

186 - I think I have done it correctly, but PE.

276 - The outout format is not clear.

612 - The output format is not mentioned. But to get acc we have to print a blank line between two consecutive cases.

621 - I dont know why I m getting PE.

637 - PE, but havent found any reason.

814 - I did everything right, but getting PE.

And obviously there are a lot more.
Ami ekhono shopno dekhi...
HomePage

tobby
Learning poster
Posts: 98
Joined: Fri Dec 30, 2005 3:31 pm

Post by tobby » Mon Dec 11, 2006 8:16 pm

How about

697 Jack and Jill - Nothing is said about whether we should print a blank line after the last case. Now I got PE for that problem.

And about the sample I/O:

376 More Triangles ... - The second line of sample output has a trailing space. I followed that and got PE.

400 Unix ls - Some spaces missing after the end of the last line of sample output

EDIT : I have just got AC for 697. No blank lines should be printed after the last case.

User avatar
Carlos
System administrator
Posts: 1286
Joined: Sat Oct 13, 2001 2:00 am
Location: Valladolid, Spain
Contact:

Post by Carlos » Wed Dec 13, 2006 12:25 am

please, post them in separated topics or i'll never be sure all of this issues are solved!

Anyway, if you knew those problems had errors, why didn't you report them? :-D

Btw, sorry for this change, but as a few users reported PE issues last year, we though that it wouldn't affect too much.
DON'T PM ME --> For any doubt, suggestion or error reporting, please use the "Contact us" form in the web.

Per
A great helper
Posts: 429
Joined: Fri Nov 29, 2002 11:27 pm
Location: Sweden

Re: PE issues

Post by Per » Wed Dec 13, 2006 1:49 am

little joey wrote:Great! My list of solved problems is shorter by 60 in one swoop! If only FIFA had the same rejudging policy for their tournaments, Holland would become World Champion for 1978, for sure!
Hmm, was this announced somewhere? I can't recall having received a mail about it, and I can't see it mentioned anywhere on the site?
I agree that P.E. should be no longer accepted, [...]
I don't agree. For a majority of all problems, output format is completely irrelevant, and small whitespace errors should not be a problem. For me, presentation error is an artefact of "old" times when many problems had unnecessarily complicated output. (Of course, there are exceptions where output format is important, e.g. 10467, but this type of problem is a lot more uncommon than it's counterpart).

Just my two cents worth of opinions.

User avatar
little joey
Guru
Posts: 1080
Joined: Thu Dec 19, 2002 7:37 pm

Post by little joey » Wed Dec 13, 2006 3:48 pm

Carlos wrote:Anyway, if you knew those problems had errors, why didn't you report them? :-D
?
Many, if not all of these errors were reported in the forums through the years, unnoticed by the system administrators who had no time to listen to their users' feedback.

And for me personally: I didn't bother too much about PEs because there were so many of them and a PE counted as 'problem solved' anyway. In that respect I think many other users had that same attitude.
Per wrote:Quote:
I agree that P.E. should be no longer accepted, [...]

I don't agree. For a majority of all problems, output format is completely irrelevant, and small whitespace errors should not be a problem. For me, presentation error is an artefact of "old" times when many problems had unnecessarily complicated output. (Of course, there are exceptions where output format is important, e.g. 10467, but this type of problem is a lot more uncommon than it's counterpart).
Yes, I agree, but faced with dilemma of either accepting all formatting errors or rejecting them all, I would opt for the latter. In an ideal judging system a special corrector would be used for all problems, tailored to the relevance of the specific formatting requirements. In cases where judging is done based on simple text-comparison, no realy fair method exists that covers all cases, IMO.

BTW: not only 'ancient' problems cause formatting issues; it's quite common in 'modern days' problems to require the case-number to be printed in the output with a manyfold of different formats (capitalisation, hash-marks, spaces, colons, newlines, etc.), and it would be good if the judging system were more tolerant in these cases.

Darko
Guru
Posts: 580
Joined: Fri Nov 11, 2005 9:34 am
Location: Calgary, Canada

Post by Darko » Wed Dec 13, 2006 8:12 pm

OK, I missed the discussion (if there was such thing), but how did they decide to do this again?

Guessing the output format is not fun. Before this decision, they should've gone through all (yes, all) problem statements and checked if they were ambiguous or not, if judge's solutions followed the same specifications, that sort of thing.

It would've been much easier if the issues were addressed as they were encountered. It was only recently that it was pointed to us that people that run this site read only Bugs and Suggestions. Rejudging seemed mostly random in the past, it was probably because the reasons were not publicly posted, now it's better - but now we know that they are not aware of the majority of the issues with the problems.

And, I agree with Per, unless output formatting is what the problem is about, it shouldn't be an issue.

User avatar
Carlos
System administrator
Posts: 1286
Joined: Sat Oct 13, 2001 2:00 am
Location: Valladolid, Spain
Contact:

Post by Carlos » Thu Dec 14, 2006 9:38 am

I sure agree with you, but you know, it takes about 2 hours to make a special corrector program for each problem. We have so many problems that this would be unreliable.

As I already said, no PE was reported to us in the last few months. I don't agree at all when you say that
Many, if not all of these errors were reported in the forums through the years, unnoticed by the system administrators who had no time to listen to their users' feedback.
Forums were not intended to be for system administrator's feedback. You had a support email address, problemset@acm.uva.es, were we listened to all, I mean ALL of your suggestions. I spent a lot of time replying emails, correcting bugs and maintaining the judge. It's good to know what it worths to you. A few time ago, I realized that most users were complaining in the forum, then I opened that "Bugs and suggestions" forum. And have you ever seen an ignored thread?

I sure know that this project surpasses our (VOJ team's) posibilities, but we do our best. In UVa nobody is interested in helping us, not sponsoring us for contracting someone. Until now we've relied on unvaluable users' help, YOUR help, that was a really great help. But if it keeps like that, VOJ will disappear as soon as one of us gives up. So please, understand the situation and be a bit patient with every change we do.

We'll fix PE issues as soon as we can (and as soon as the judging machine finishes rejudging all wrong judged submission, about 2500 a day).
Guessing the output format is not fun. Before this decision, they should've gone through all (yes, all) problem statements and checked if they were ambiguous or not, if judge's solutions followed the same specifications, that sort of thing.
We should have done it, sure, but it with the new statistics it was easier to look for PEs in user's info. We hoped each one of you to report every single problem with PE. And it worked. What I didn't expect was that "who the hell had that f*** idea" reaction. It's my idea, it's my fault. I'm sorry.
DON'T PM ME --> For any doubt, suggestion or error reporting, please use the "Contact us" form in the web.

User avatar
little joey
Guru
Posts: 1080
Joined: Thu Dec 19, 2002 7:37 pm

Post by little joey » Thu Dec 14, 2006 12:49 pm

OK. I'm sorry. I take it back. I was describing the situation as it existed for a long time, and in no way I meant to belittle the great amount of work you and Miguel and others put into the OJ. I know you always promptly and accurately respond to postings in "Bugs and Suggestions", and I and many others truely appreciate that. It hasn't always been that way, however, and many bugs have been discussed in the forum by people who were unaware that their messages were not read by the system administrators or had no idea how to communiate with them. That was what I tried to say.

Ala. Don't get distracted by the clumsy and too explicit critisisms of this grumpy old f@rt. I'll try to refrain from them in the future. If you implement big changes out of the blue, like this PE thing, you should be prepared to stir up some reactions; not all of them being positive.

Jan
Guru
Posts: 1334
Joined: Wed Jun 22, 2005 10:58 pm
Location: Dhaka, Bangladesh
Contact:

Post by Jan » Thu Dec 14, 2006 1:39 pm

Carlos, don't be sorry. But the main problem is I got P.E in several problems, and couldn't figure out the error. I checked again and again but in vain. I think I have followed each and every instruction from those problems. The problem is that there are many people who got accepted. So, what to do with these problems? Should I report?

And there are some problems(I got accepted), but I think that to get accepted some luck is needed, too. Like 312. For this problem remove all trailing spaces, but every line should be present in the output. No blank lines should be removed. So, what to do with these problems? Should I post the necessary informations to get those accepted? My opinion is that for these problems PE should be considered as accepted.
Ami ekhono shopno dekhi...
HomePage

shahriar_manzoor
System administrator & Problemsetter
Posts: 399
Joined: Sat Jan 12, 2002 2:00 am

hmm

Post by shahriar_manzoor » Fri Dec 15, 2006 6:56 am

To solve this problem one generalized special preprocessor has to be written that will

(a) Remove all new lines in the user output file.
(b) Remove all consecutive spaces (More than one spaces) and replace it with one space.
(c) Remove all leading and trailing spaces in a line.

Then compare the output with judge output. The judge output will already have gone through this process. The problems which have strict output descriptions or have special judge already can be excluded from this process.

Have I missed any points?

This is what is done in world finals. I can write one if needed but did not dare to write one without request :).

-Shahriar

User avatar
Carlos
System administrator
Posts: 1286
Joined: Sat Oct 13, 2001 2:00 am
Location: Valladolid, Spain
Contact:

Post by Carlos » Sun Dec 17, 2006 3:59 pm

Actually, I prefer to fix every single dataset (or even description) than applying a special corrector program for all of them.

Anyway, for some specific problems that corrector would be useful. I've just made a special corrector program for 324 that allows extra blank spaces, but not extra blank lines. Having a customizable special corrector program for this kind of situations (applying the 3 rules you mention) would be great, but I don't know it the effort worths it...it would be used in 3-4 more problems, not in the whole problemset as you propose. So, if you feel like doing it, go on; if not, it doesn't matter, I'll just make problem specific corrector programs (324 corrector program took 30 lines of code, not a great effort...).

whether you make it or not, thanks for your proposal and your help! :-D
DON'T PM ME --> For any doubt, suggestion or error reporting, please use the "Contact us" form in the web.

Locked

Return to “Bugs and suggestions”