Migration Solutions for ColdFusion Applications to ASP.NET
      
New Atlanta Product Forums Profile | Search | Login | RSS
New Topic Reply   Previous Page  Page: 1   Next Page

Thread: Failed to connect to ServletExec
Created on: 04/27/09 10:31 AM Replies: 12
fgasbarra


Joined: 04/27/09
Posts: 2
Failed to connect to ServletExec
04/27/09 10:31 AM

Good evening,
we are experiencing the error number 146 "Failed to connect to ServletExec" issued by apache web server.
This error happens during the day without any reason.
We use the module servletexec 5.0 within apache v2.0.61 web server.
The webadapter is configured in the following way:

servletexec.aliasCheckInterval=10
servletexec.ttsweb.hosts=all
servletexec.ttsweb.instances=127.0.0.1:8080
servletexec.ttsweb.pool-increment=5
servletexec.ttsweb.pool-max-idle=10

ServletExec is used to host and execute servlets of a BMC Remedy application.

In the same way this problem starts it desappears end has no effect restarting the ServletExec engine and the web server.
Has anyone idea of the reason why this kind of error is happening?
Link | Top | Bottom
mattm


Joined: 10/10/07
Posts: 266
RE: Failed to connect to ServletExec
04/27/09 11:35 AM

That message is coming from the SE AS native adapter (the part of SE that snaps into the web server software).

If you were to turn off your SE AS instance (the java side of SE) and then try to request a servlet or JSP you could expect to see this message.

So the cause of the problem is likely one of these:

1. You or someone else has turned off the SE AS instance, in which case turning it back on would fix it.

2. Your SE AS instance crashed, in which case turning it back on would fix it.

3. Something in your network (a firewall perhaps) is blocking the communication between the SE AS native adapter and the SE AS instance.

Basically, if the adapter can't connect to the instance (for any reason) then you'll see this message.

Here are something things you might consider checking:

1. Examine your ServletExec.log files for clues.

2. Check your network to see if there might be something that is blocking traffic on port 8080 (perhaps only during certain times of the day??)

3. Check to make sure that you are starting up the SE AS instance (StartServletExec.sh) in a manner that it will continue running even after you log off the machine.
For further details about that, please see:
http://www.newatlanta.com/c/support/servletexec/self_help/faq/detail?faqId=115

I hope this helps.

Matt McGinty
Software Engineer
NAC
Matt McGinty, New Atlanta Technical Support
Link | Top | Bottom
fgasbarra


Joined: 04/27/09
Posts: 2
RE: Failed to connect to ServletExec
04/27/09 11:49 AM

Matt, thanks for the rapid response.
The doubt that I have is on how the network could be envolved in the problem if the web server and the se instance are hosted both on the same machine.

Sincerely

Fabrizio Gasbarra
Link | Top | Bottom
mattm


Joined: 10/10/07
Posts: 266
RE: Failed to connect to ServletExec
04/27/09 12:47 PM

A firewall running on that same machine is one possibility.
Matt McGinty, New Atlanta Technical Support
Link | Top | Bottom
dcarmo


Joined: 04/27/09
Posts: 4
RE: Failed to connect to ServletExec
04/27/09 1:38 PM

Hi Matt,
Fgasbarra is my coworker... and we are together in this case. :))
No, we don't have any firewalls running in our machines.
No changes were made in this machine in the last 6 months...

Thanks!
Darci do Carmo
Link | Top | Bottom
mattm


Joined: 10/10/07
Posts: 266
RE: Failed to connect to ServletExec
04/27/09 2:18 PM

Are there any clues in any of your ServletExec.log files?

Please confirm a statement that was made in the original post by answering these questions in the manner requested:

Does restarting Apache fix this problem? (yes or no)
Does restarting SE fix this problem? (yes or no)
Does this problem seem to fix itself? (yes or no)


Was there ever a time when this problem did not occur? (yes or no)
If yes, for how long (how many months) did it not occur?

What might have changed from the time when the problem did not occur, to the time when it began to occur?

I understand you are using SE 5.0 AS.
Precisely which version of SE 5.0?
http://www.newatlanta.com/c/support/servletexec/self_help/faq/detail?faqId=94

Matt
Matt McGinty, New Atlanta Technical Support
Link | Top | Bottom
dcarmo


Joined: 04/27/09
Posts: 4
RE: Failed to connect to ServletExec
04/28/09 11:02 AM

Are there any clues in any of your ServletExec.log files? No, but I would like to send this logfile, How can I do it? Who knows, you or someone can see something that we cannot see.

Does restarting Apache fix this problem? No.
Does restarting SE fix this problem? No.
Does this problem seem to fix itself? yes, this problem does not occur since last friday.

Was there ever a time when this problem did not occur? No. this problem never occurred before.

What might have changed from the time... Nothing.

version is 5.0.0.13 running in the UNIX Solaris 10 (SPARC)
thanks!
Darci
Link | Top | Bottom
mattm


Joined: 10/10/07
Posts: 266
RE: Failed to connect to ServletExec
04/28/09 12:24 PM

Darci,

So this problem has never occurred before... not at all over the past several months... and now, it suddenly starts to occur.

I understand that you don't know of anything that might have changed.
However, there must be something that has changed.
Certainly the behavior has changed, so it only seem reasonable that something has changed to cause it to occur.

Figuring out what that something is may be the hard part.

Please send any log files you have to support@servletexec.com

Matt
Matt McGinty, New Atlanta Technical Support
Link | Top | Bottom
mattm


Joined: 10/10/07
Posts: 266
RE: Failed to connect to ServletExec
04/28/09 12:24 PM

Darci,

I understand you are using SE 5.0 AS.
Precisely which version of SE 5.0?
http://www.newatlanta.com/c/support/servletexec/self_help/faq/detail?faqId=94

Matt
Matt McGinty, New Atlanta Technical Support
Link | Top | Bottom
dcarmo


Joined: 04/27/09
Posts: 4
RE: Failed to connect to ServletExec
04/28/09 3:02 PM

Matt,
I sent to you...

Darci
Link | Top | Bottom
mattm


Joined: 10/10/07
Posts: 266
RE: Failed to connect to ServletExec
04/28/09 3:47 PM

Darci,

Please check the address to which you sent your email(s).
Nothing from you has arrived at support@servletexec.com

I checked the Spam folder and the Junk folder.
Nothing from you is there.

Can you send something with *no* attachment, just to see if that works?

Matt
Matt McGinty, New Atlanta Technical Support
Link | Top | Bottom
mattm


Joined: 10/10/07
Posts: 266
RE: Failed to connect to ServletExec
04/28/09 4:43 PM

Darci,
I received the 10 ServletExec.log files you sent.
I looked over the first 4 of them.
They all contain what looks like a stacktrace that is repeated over and over.

While the trace occurs at different timestamps... each time it *does* occur it repeats many times within the same second.

Each trace mentions this Remedy ARWEB error:

-----
ARERR [9093] User is currently connected from another machine
-----

which occurs in this Remedy ARWEB code:

----
at com.remedy.arsys.goat.Form.checkAccess(Unknown Source)
----

The trace does not give line numbers of the Remedy code... possibly because of how the Remedy code was compiled.

Another possibly relevant stacktrace I saw in the log files was:

-----
Compiled formsCache: Failed to write element to disk '57900,57400,56110,52100,50999,50055,50050,50000|tts/TBR_Summary Panel/536875261/N'. Initial cause was null
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:841)
...
...   
at com.remedy.arsys.goat.field.FieldGraph.writeObject(Unknown Source)
...
...   
at com.remedy.arsys.share.GoatCacheManager$SaveThread.run(Unknown Source)
-----

All these traces considered together lead me to believe that your application may be having some Threading issues.
That perhaps your Remedy app is not setup to function properly in a multi-threaded environment.

It also seems that when the application encounters the "ARERR [9093]" error it may be trying to recover from that and retry the same functionality once again which again cause the error (a loop).
Without knowing what the Remedy app is trying (and retrying) to do I can only guess that the task *may* be using up "low level" System Resources (a network socket and thus a file descriptor perhaps)???

These are all just guesses.

But to learn more you should contact the makers of your webapp (Remedy) and ask them to help you determine what is going wrong inside the ARWEB webapp code that happens to be running inside ServletExec.

Please know that it is possible for java code running inside any JVM to destabilize that JVM to the point that it can no longer function as needed.
It is also possible for that JVM to then recover from the problem at some later point.

If the Remedy webapp is using up all the file descriptors on the system then that might explain
why the SE native adapter can't connect to the SE AS java instance. And might also explain why restarting the JVM or Apache has no effect.

There is a time period before the OS will clean up such sockets/file-descriptors. Normally this is all transparent but if tons and tons of sockets get allocated and must then wait to be cleaned up, it may mean that the OS can't give the JVM any new sockets until old ones have been cleaned up.

You might also consider using
> netstat -a
from a Linux command window to examine the state of the OS sockets.

Start with a fresh reboot of the OS and then issue that command and take note of the output.

Then start SE AS and Apache.
Then issue the command again and take note of the differences in the output as compared to the last time you issued that command.

Then try to make the problem occur again.

Then issue that command again to see if there are an inordinate number of sockets that have not be released for use yet.

Then wait for the problem to magically correct itself.

Then issue that command again and examine the output again, comparing with previous results.

Here is a FAQ to read:
http://www.newatlanta.com/c/support/servletexec/self_help/faq/detail?faqId=4

But just realize that increasing the number of OS FD's may only delay the problem from occurring rather than fix it (*if* the Remedy code is doing the wrong thing).

Matt
Matt McGinty, New Atlanta Technical Support
Link | Top | Bottom
dcarmo


Joined: 04/27/09
Posts: 4
RE: Failed to connect to ServletExec
04/28/09 6:38 PM

Ok.
About error 9093: this is a normal event... because a user cannot log in to Remedy Aplication from two computers at the same time. So, the system records this event.

On next days we will check and do it. I will post my results here.
If you have more news, let us know!


Darci
Link | Top | Bottom

New Post
Please login to post a response.


company media information terms of use privacy policy contact us
This page was dynamically built on the BlueDragon CFML Engine