mod_mono on strike, troubleshoot ?

[zzz]

My setup is rapidscada+simplejson+grafana on linux. Simplejson extension is running with apache2+mod_mono4 as suggested in documentation.

There is an array of thinclients displaying the dashboard 7×24, as the number of clients increasing, grafana starts to show backend error (red exclamation mark on upper left corner of each graph) on dashboard. Further error message displayed in grafana is as below:

// on Error tab
Object
status:500
statusText:”Internal Server Error”
data:Object
Message:”An error has occurred.”
config:Object
url:”api/datasources/proxy/1/query”
data:Object
method:”POST”
withCredentials:undefined
hideFromInspector:false
message:”Query error: 500 Internal Server Error”

// on Query tab
response:Object
Message:”An error has occurred.”

The setup was running ok for months, until recently, after we added some new clients (<30 in total).
I myself is familar with Neither mono Nor apache2 (nginx is always my primary choice). I saw an post from stackoverflow, with similar problem
https://stackoverflow.com/questions/49118701/mono-server-freezing-after-2-days
which seems to suggest load and concurrency issue. Since the config suggested by the documentation was already using unix socket, the solution suggested in that post is not helpful in my case.

# Default configuration, don’t edit it!
<IfModule mod_mono.c>
MonoUnixSocket default /tmp/.mod_mono_server4
MonoServerPath default /usr/bin/mod-mono-server4
AddType application/x-asp-net .aspx .ashx .asmx .ascx .asax .config .ascx
MonoApplicationsConfigDir default /etc/mono-server4
MonoPath default /usr/lib/mono/4.5:/usr/lib:
# start /etc/mono-server4/conf.d/monodoc-http/10_monodoc-http
Alias /monodoc “/usr/share/monodoc/web”
AddMonoApplications default “/monodoc:/usr/share/monodoc/web”
<Directory /usr/share/monodoc/web>
SetHandler mono
<IfModule mod_dir.c>
DirectoryIndex index.aspx
</IfModule>
</Directory>
# end /etc/mono-server4/conf.d/monodoc-http/10_monodoc-http
</IfModule>

The rapidscada web frontend was once also down together with the SimpleJson extension. But this is rare, currently the SimpleJson extension will only work for a couple of hours and then cease working with “internal error” above. I am not sure how to proceed with the troubleshoot, since I can’t find any log. except the repeated error like below in apache2/error.log.

// endless repeatation of
[Sun Apr 10 15:54:35.855352 2022] [:error] [pid 13688] Failed to connect to mod-mono-server after several attempts to spawn the process.

// occasional
[Sun Apr 10 00:00:23.539564 2022] [mpm_prefork:notice] [pid 1683958] AH00163: Apache/2.4.52 (Debian) mod_mono/3.13 mod_wsgi/4.7.1 Python/3.9 configured — resuming normal operations
[Sun Apr 10 00:00:23.539649 2022] [core:notice] [pid 1683958] AH00094: Command line: ‘/usr/sbin/apache2’
WARNING: WebConfigurationManager’s LRUcache evictions count reached its max size
Cache Size: 100 (overridable via MONO_ASPNET_WEBCONFIG_CACHESIZE)

[Mikhail]

Hello,

Please check RAM consumption.
Are there any errors in /opt/scada/ScadaWeb/log/ScadaWeb.log?

Mono is not stable enough. For this reason, we use .NET6 instead of Mono in Rapid SCADA 6. The Grafana plugin has not be ported yet. When we do that, you could update.

[zzz]

Hi Mik,

Odd enough, /opt/scada/ScadaWeb/log/ScadaWeb.log doesn’t exist :-(. /opt/scada/ScadaWeb/log/ is empty. Does it need further configuring ?

I don’t have a system monitor tracking the server, not sure what the ram look like at the moment of the crash. But in my past observation, it seldom exceeds 50%, usually 30~45%

Is v6 in beta now ? I’d like the give it a try if so. Is there any compile guide for dotnet-core noobs?

I think it really a good idea to move away from mono, the resources for mod_mono is getting really scarce…

Regards

[Mikhail]

Hi,
After uploading the project, writing rights on the log folder often disappear.
If you provide writing access for all on this folder, I expect that log will be created. This is a known issue, it is fixed in the v6.
Hope to catch a error with details.

Is v6 in beta now ?

Yes. It’s available. But currently not all modules have been ported.

Is there any compile guide for dotnet-core noobs?

A guide will be created later. In short, switch to Release configuration before build. Build *Common projects first. VS 2022 Community is OK.

[zzz]

Hi Mik,

I can touch /opt/scada/ScadaWeb/log/tmp.1. So I looked around and noticed that mod-mono is running as www-data. Do you mean I should chown -R www-data ...../log?

One thing I forgot to the mention, the mod-mono’s RAM use is low but the CPU is now most of the time 80~140% (this is a two core node, 100%=1core). ScadaServer 10~20% (not constant), ScadaComm 20~40%.

Ok, I will try once I have VS2022 install. Thanks.

[zzz]

After the log folder is chowned, I can see scadaweb.log now, but 99% of it seems from Rapidscada’s web frontend, mostly sutff like below

Scada.ScadaException: Error getting view with ID=238 from the cache: The view is not found in the cache
at Scada.Client.ViewCache.GetViewFromCache (System.Int32 viewID, System.Boolean throwOnFail) [0x0008f] in <bed4d89bd9d74517a
be54cd0f8c8e979>:0
at Scada.Web.ClientApiSvc.GetViewFromCache (System.Int32 viewID, Scada.Web.UserRights userRights) [0x00026] in <ba66a674522d
470295eecd6524152467>:0
at Scada.Web.ClientApiSvc.GetCnlList (System.String cnlNums, System.String viewIDs, System.Int32 viewID, Scada.Web.UserRight
s userRights) [0x00032] in <ba66a674522d470295eecd6524152467>:0
at Scada.Web.ClientApiSvc.GetHourCnlData (System.Int32 year, System.Int32 month, System.Int32 day, System.Int32 startHour, S
ystem.Int32 endHour, System.String cnlNums, System.String viewIDs, System.Int32 viewID, System.Boolean existing, System.String
dataAge) [0x0000e] in <ba66a674522d470295eecd6524152467>:0
2022-04-12 20:56:00 <wsta00><www-data><EXC> Error getting view with ID=238 from the cache: The view is not found in the cache:
Scada.ScadaException: The view is not found in the cache
at Scada.Client.ViewCache.GetViewFromCache (System.Int32 viewID, System.Boolean throwOnFail) [0x00049] in <bed4d89bd9d74517a
be54cd0f8c8e979>:0
2022-04-12 20:56:00 <wsta00><www-data><EXC> Error getting events by the filter where channels= , view id=238:
Scada.ScadaException: Error getting view with ID=238 from the cache: The view is not found in the cache
at Scada.Client.ViewCache.GetViewFromCache (System.Int32 viewID, System.Boolean throwOnFail) [0x0008f] in <bed4d89bd9d74517a
be54cd0f8c8e979>:0
at Scada.Web.ClientApiSvc.GetViewFromCache (System.Int32 viewID, Scada.Web.UserRights userRights) [0x00026] in <ba66a674522d
470295eecd6524152467>:0
at Scada.Web.ClientApiSvc.GetCnlSet (System.String cnlNums, System.String viewIDs, System.Int32 viewID, Scada.Web.UserRights
userRights) [0x00040] in <ba66a674522d470295eecd6524152467>:0

Not sure why, I haven’t seen crash yet after the change, will post log once it crashed again. Thanks.

[Mikhail]

I suppose that “view is not found in the cache” are not critical errors. It usually means that user session is closed but web app is still open in browser.

If crash occurs again, please check the log for the details.

[Author]

Posts