java.sql.SQLRecoverableException: No more data to read from socket

[jgebal]

I get the following error when running tests using utplsql-cli:

java.sql.SQLRecoverableException: No more data to read from socket
	at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:119)
	at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534)
	at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485)
	at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
	at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
	at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:223)
	at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:56)
	at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:907)
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1119)
	at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)
	at oracle.jdbc.driver.T4CCallableStatement.executeInternal(T4CCallableStatement.java:1300)
	at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3887)
	at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4230)
	at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1079)
	at io.github.utplsql.api.TestRunner.run(TestRunner.java:144)
	at io.github.utplsql.cli.RunCommand.lambda$run$0(RunCommand.java:141)
	at io.github.utplsql.cli.RunCommand$$Lambda$7/1647809929.run(Unknown Source)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

The trouble is, the utplsql-cli does not stop/throw.
It seems to be stuck, doing nothing.

This leads to two potential problems:

1. Exceptions are not handled properly and the cli is doing nothing
2. timeout for getting data from output buffer is set to a low (too low) value.
   Single test can run for quite some time.

[pesse]

@jgebal experienced this issue at any time since 3.0.4 version?

[c0dw]

We're using the most recent version 3.1.2 in a custom docker container (alpine based). About one in 25 calls (in a loop, each iteration using a freshly initialized container) get's blocked and only after waiting one full hour get's terminated eventually.

The utplsql command generates no output at all in these (rare) cases.

Furthermore, as already mentioned the exception/error exit code, due to the timeout seems to get lost on the way. Our call does get exit-code zero even after such a timeout. Therefore only the missing output (and the delay) can be used a indirect means of error handling...

Any progress avoiding this blockade and/or at least a proper error escalation in case it can't be avoided would be greatly appreciated.

[pesse]

Hi @c0dw ,

are you using Oracle 11.2?
We have identified similar issues with a 11.2 setup (see #98)

Samuel

[c0dw]

Hi @pesse
No, we're running the utplsql Framework (serverside) on Oracle 12 (Enterprise Edition) and from using 'lsof' I know the utplsql cli uses a version 12 jdbc driver on client side as well:

/mnt/workspace # lsof -p 24
...
/opt/jdk1.8.0_141/bin/java	/opt/utPLSQL-cli/lib/ojdbc8-12.2.0.1.jar
...

[pesse]

I'll soon have a new version of cli ready with logging possibilities - I would then aks you to try it with this new version and provide the logs, if possible

[c0dw]

Absolutely! I look forward to verify the new version, thanks!
BTW, in case you're still open to feature suggestions for the new version: IMHO it would also help if the timeout were not hardcoded to 60 minutes, but could be customized when invoking utplsql depending on the expected runtime of the test. Our unittests usually take no langer than 1 or 2 minutes.

[pesse]

That's a good point - want to make an issue for it?
It's actually not much work to include that.

[c0dw]

OK, done! -> configurable timeout #124
... and many thanks for your swift response!

[pesse]

Hey @c0dw , can you try to reproduce it with this version of cli?
https://bintray.com/utplsql/utPLSQL-cli/download_file?file_path=utPLSQL-cli-develop-201902142127.zip

use
utplsql run user/pw@connectstring -d
for debug output

[c0dw]

Hi @pesse
I built a new Dockerimage with the utplsqlcli version 201902142127 you provided and tried to reproduce the bug while using the debug flag. It did not happen. After 100 successful iterations I broke out of the loop. For reference I include the debug output of the last iteration here anyway. And FYI, at the end I listed the invoking loop

iteration100.txt

After that, I tried to reproduce the error again with the official release Version 3.1.2 of utplsqlcli. It also took significantly more iterations until the error occured: More than 70 iterations in a first pass and 56 iterations in the second attempt. As you can see iteration 56 took one hour (3603 seconds) to timeout and it produced no log (zero lines in srqoauth2_doc.log). In the shebang of the script I set the '-e' flag to cause immediate termination in case of an error, which obviously never made it up the callstack...

15.02.2019 12:42:29 - Iteration 50, duration 6 seconds, number of output-lines: 24 target/srqoauth2_doc.log
15.02.2019 12:42:37 - Iteration 51, duration 5 seconds, number of output-lines: 24 target/srqoauth2_doc.log
15.02.2019 12:42:46 - Iteration 52, duration 6 seconds, number of output-lines: 24 target/srqoauth2_doc.log
15.02.2019 12:42:54 - Iteration 53, duration 5 seconds, number of output-lines: 24 target/srqoauth2_doc.log
15.02.2019 12:43:03 - Iteration 54, duration 6 seconds, number of output-lines: 24 target/srqoauth2_doc.log
15.02.2019 12:43:11 - Iteration 55, duration 5 seconds, number of output-lines: 24 target/srqoauth2_doc.log
15.02.2019 13:43:17 - Iteration 56, duration 3603 seconds, number of output-lines: 0 target/srqoauth2_doc.log
15.02.2019 13:43:26 - Iteration 57, duration 6 seconds, number of output-lines: 24 target/srqoauth2_doc.log
15.02.2019 13:43:34 - Iteration 58, duration 5 seconds, number of output-lines: 24 target/srqoauth2_doc.log

I managed to enter the running Docker container and took a threaddump with jstack, if this is any help understanding the problem:

jstack_threaddump.txt

I will try again to reproduce the failure and collect debugging info using your working version 201902142127. Next time I will be more patient and wait for more than 100 iterations just to be sure, the error does indeed not occur anymore.

[pesse]

thanks a lot for your thorough analysis, @c0dw
We did some tweaks in the newer version, too, so it might be that the main problem is resolved.

[c0dw]

That's the least I can do!
Meanwhile I switched back to the utplsqlcli working version 201902142127 and started a new session expecting it to keep running smoothly all day and next night. (Note to self: Stop expecting!)

As a matter of fact, it happend already in Iteration 17, this time I can provide the full debug log and for comparison the one before (iteration16), where all was well:

Iteration16-16_DEBUG.txt

As you can see, the process is stuck while 'Initializing reporters'. I get one line of output every 30 seconds stating...
2019-02-15 14:25:23 [HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Pool stats (total=3, active=3, idle=0, waiting=0)

...and here is the according jstack threaddump of the utplsqlcli again:
jstack_threaddump_utplsqlcli201902142127.txt

[c0dw]

Hi @pesse

Please let me know, if I can contribute anything else to resolve this blocking issue. Apparently it persists in the development snapshot 201902142127 you provided, as outlined in my last posting.

I'd also like to point out, we still face the additional problem, that the timeout does not trigger a proper error escalation. I still don't get an exitcode after a timeout. My java skills are slightly rusty - so correct me if I'm wrong, but from your code (lines 203-209) I understand you're using java.util.concurrent.ExecutorService.shutdown() which causes an orderly shutdown and after that in line 209 you return 'returnCode[0]' which I suppose is initialized to literally zero as well. Therefore an invoking process does not get notified, that an irregular termination (due to a timeout) occurred, which makes error handling cumbersome. IMHO a timeout should push an error/exception up the call-stack.
Update: I digged a little deeper and I think the solution is using the return Value of the awaitTermination Method to determine the returnValue of your own function, like ...

if (!executorService.awaitTermination(timeoutInMinutes, TimeUnit.MINUTES)) {
        returnCode[0] = 1; // or whatever exitcode you might assign a timeout - except zero!
}

According to the API doc awaitTermination() returns true if this executor terminated and false if the timeout elapsed before termination.