Project

General

Profile

Defect #123

sqs-reader hanging

Added by Michele Vivoda over 8 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
medium
Assignee:
Category:
MarkReader
Target version:
Start date:
2012-04-25
Due date:
2012-05-05
% Done:

100%

Estimated time:
3.00 h
Spent time:

Description

The current version of the sqs-reader hangs after processing starts (a folder is dropped on the pane).

Debugging in eclipse I found it hangs in method MarkReaderSession#receiveAllPageTaskResults(), in call to taskCommitFuture.get();
I looked in the code but for me is quite complex the threading code and didn't manage to understand what is happening.

The compiled version I have in the self-executable jar made by Milos is working (very well also, 3000 images in 4 mins)
but I would like to run a source version to modify the code to include parsing of qrcode and empty image check for free text.

History

#1 Updated by Hiroya Kubo over 8 years ago

Would you please check the current version(newer than reported one) of the sqs-reader if it hangs or not... I think it has fixed.

#2 Updated by Michele Vivoda over 8 years ago

Hi,
I am afraid the problem persists, I tried many times but swing app still hangs. Tests hang as well after these lines are printed:

INFO: - (src\test\resources\test0::form.pdf)
0 a001.tif 1335306056128 0 1
1 a002.tif 1335306056131 0 1
0 a003.tif 1335306056134 0 1
1 a004.tif 1335306056136 0 1

BTW: to build utils I had to change the repository to

<repository>
<id>dev.sqs2.net</id>
<name>SQS Project Repository</name>
<url>http://sqs2.net/maven-repository&lt;/url>
</repository>

#3 Updated by Michele Vivoda over 8 years ago

Hmm.. looks like I was getting the project from old mercurial, now I got it from bitbucket where the repository URL is correct, sorry, forget about the <repository> comment.

Swing and tests still hanging though, will look in code where.

#4 Updated by Hiroya Kubo over 8 years ago

Please tell me your JDK type and version.

In my environment, all tests are passed...

http://jenkins.sqs2.net/job/sqs-reader/lastSuccessfulBuild/consoleText

#5 Updated by Michele Vivoda over 8 years ago

I manage to run the packaged reader in Windows
with all SUN JDK java 6 I have installed (1.6.0_15, 1.6.0_23)

so I was always trying junit tests with 1.6.0_23.. now I tried v.15 and
the test does not halt there (good news) but ..

..in changing to vm v. 0_15 the code gives compilation errors because
the dom.Element interface of the referenced lib does not define the getTextContent() method..
so I added a newer jaxp to the endorsed lib of the jre and it fixed it

Now the tests completed but with errors.

http://pastebin.com/pb8pNNXp

#6 Updated by Hiroya Kubo over 8 years ago

The hanging is not reproduced on jdk1.6.0_32 but on jdk1.7.0_04 in my environment(Linux x86 (32-bit)) just as you pointed. I start debugging...

#7 Updated by Hiroya Kubo over 8 years ago

  • Project changed from sqs-translator to sqs-reader

This issue was moved to sqs-reader project.

#8 Updated by Hiroya Kubo over 8 years ago

  • Due date set to 2012-05-05
  • Category set to MarkReader
  • Assignee set to Hiroya Kubo
  • Target version set to MarkReader2.1

#9 Updated by Hiroya Kubo over 8 years ago

  • % Done changed from 0 to 10
  • Estimated time set to 3.00 h

Michele Vivoda wrote:

I manage to run the packaged reader in Windows
with all SUN JDK java 6 I have installed (1.6.0_15, 1.6.0_23)

so I was always trying junit tests with 1.6.0_23.. now I tried v.15 and
the test does not halt there (good news) but ..

..in changing to vm v. 0_15 the code gives compilation errors because
the dom.Element interface of the referenced lib does not define the getTextContent() method..
so I added a newer jaxp to the endorsed lib of the jre and it fixed it

I pushed my commit on pom.xml of sqs-util to update versions of dependencies: xercesImpl, xml-apis, xml-resolver.
The errors such as "getTextContent() method is undefined" are eliminated.
Now you may be able to remove your jaxp related jars from your endorsed lib directory.

#10 Updated by Hiroya Kubo over 8 years ago

I pushed my commit on pom.xml of sqs-util to update versions of dependencies: xercesImpl, xml-apis, xml-resolver.
The errors such as "getTextContent() method is undefined" are eliminated.
Now you may be able to remove your jaxp related jars from your endorsed lib directory.

A dependency of sqs-reader is updated, too.

#11 Updated by Hiroya Kubo over 8 years ago

  • % Done changed from 10 to 60

Redesign class SessionSourcePhase and debug event model of MarkReaderSession.

The All of prepared TestNG testcases are passed!

However, Swing GUI is still hanging. I'll fix it soon.

#12 Updated by Michele Vivoda over 8 years ago

I managed to run the tests without any major problem apart one,
I was getting many test failures like those in the pastebin link of previous post, something like:

afterClass(net.sqs2.omr.session.service.PeerExecutableOMRPageTaskTest): Error on remove: C:\Users\mik\AppData\Local\Temp\MarkReaderAppTest6908160978042087966X\resources\MarkReaderAppTest1309954595382017100X_test3\RESULT

I looked in code and all are due to call to SessionSources.closeAll(true);
I don't know why is failing in deleting the RESULT directory.

However at higher level I think is better to not touch the source dirs so I set up a TestFolderManager class to make two things:
1) copy the src/test/resources dir to a temp dir
2) create further copies of the test1, test2.. dirs for individual tests
Then I changed all the references to the resource file with something like

public final File sourceDirectoryRoot0 = TestFolderManager.getFolder( "test0");

In this way there are 2 advantages: sources are not touched and each test has its own folder to work on.

Then, as final workaround? step I replaced all the calls to SessionSources.closeAll(true) with SessionSources.closeAll(false); Since the files are temporary there is no need to remove them.. but I am not sure if this part is correct or not.

I will submit a pull request.

Swing UI I checked and is still hanging but later, repeating in endless loop the following:

INFO: execute remote task in debug mode
5-mag-2012 19.40.28 net.sqs2.omr.session.daemon.TaskTracker$1 run

Ah, last detail one test is failing for PeerExecutableOMRPageTaskTest#testExecuteOnTest3,
says result is 18 and not 2 like you lately changed to in assertEquals(pageAreaResultList.size(), 2);
Does this mean I have a wrong version ?

#13 Updated by Hiroya Kubo over 8 years ago

Thank you for your response, Michele.

Your reported test failures, such as "Error on remove: C:\eclipsework\sqs-reader3\src\test\resources\test3\RESULT", may caused by remaining open files at the end of MarkReaderSession. If a directory contains open files, the removal of the directory will fail in a Microsoft Windows system, but will not fail in Linux and MacOS systems. I'll check it.

BTW, your idea about TestFolderManager sounds nice! I totally agree with the 2 advantages your suggested.

Swing UI have been fixed on my local environment, please wait for a moment.

#14 Updated by Hiroya Kubo over 8 years ago

Michele Vivoda wrote:

Ah, last detail one test is failing for PeerExecutableOMRPageTaskTest#testExecuteOnTest3,
says result is 18 and not 2 like you lately changed to in assertEquals(pageAreaResultList.size(), 2);
Does this mean I have a wrong version ?

This result seems very strange for me. Your test failure might be also caused by some kind of runtime environment differences between us... I'll manage to reproduce the test failure on my Windows environment...

#15 Updated by Hiroya Kubo over 8 years ago

  • % Done changed from 60 to 90

Hiroya Kubo wrote:

Swing UI have been fixed on my local environment, please wait for a moment.

I pushed and built http://jenkins.sqs2.net/job/sqs-reader/70/

Please try it!

#16 Updated by Hiroya Kubo over 8 years ago

  • Status changed from Assigned to Closed
  • % Done changed from 90 to 100

#17 Updated by Michele Vivoda over 8 years ago

Yes my friend, Swing UI works nicely!

I see UI even more responsive and fast..but may be is an impression, what is sure is that in my tests papers I scanned today the new version could read one document that looked apparently normal and that the old version was not reading..so maybe a secret bug fixed as well ;-)

For the "cannot delete" error, I presume is like you say but I have no files open so maybe is an issue of the new db lib ? Not sure. I can delete the dir afterwards, no problem, I put also a long sleep before but no luck.

The PeerExecutableOMRPageTaskTest#testExecuteOnTest3 is still failing.

#18 Updated by Michele Vivoda over 8 years ago

I created a pull request for the tests modifications.

Additionally, not only one PageGuideBlockMissing error is fixed as I wrote in previous post, but also one paper (of 87) has been parsed in a "more accurate" way since one more, opinably written though, mark has been detected. Did you change also something about the parsing ? This is the paper, http://postimage.org/image/adkr5dqfh/ previously the parser was not getting option "Topo" (first of question 2).

Also available in: Atom PDF