Forum Home › Forums › Runtime Bugs › Server Bugs › Silent server crash
Tagged: Auto Control, crash, ModAutoControl, server
- This topic has 21 replies, 4 voices, and was last updated 3 years, 7 months ago by Mikhail.
-
AuthorPosts
-
January 19, 2021 at 10:46 am #8083zzzParticipant
Everything stopped working after 5:15:00. Retart manually around 10:00.
Server Log tail 01-16 05:15:30 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 05:15:30 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0 01-16 05:15:33 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 05:15:37 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 05:15:39 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 05:15:41 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0 01-16 05:15:42 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 05:15:44 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 05:15:47 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 10:05:02 <***C2><SYSTEM><ACT> Connection listener is stopped 01-16 10:05:02 <***C2><SYSTEM><EXC> Error processing new current data: System.Threading.ThreadAbortException: Thread was being aborted. at System.Threading.Monitor.ReliableEnter(Object obj, Boolean& lockTaken) at Scada.Server.Engine.MainLogic.RaiseOnCurDataProcessing(Srez receivedSrez) at Scada.Server.Engine.MainLogic.ProcCurData(Srez receivedSrez) 01-16 10:05:02 <***C2><SYSTEM><EXC> Error receiving and processing data from the client 127.0.0.1: Thread was being aborted. 01-16 10:05:02 <***C2><SYSTEM><EXC> Error communicating with the client 127.0.0.1: Thread was being aborted. 01-16 10:05:12 <***C2><SYSTEM><EXC> Error executing actions on current data calculated in module ModAutoControl: Thread was being aborted. 01-16 10:05:12 <***C2><SYSTEM><ACT> Server is aborted 01-16 10:05:12 <***C2><SYSTEM><ACT> ScadaServerService is stopped --------------------------------------------------------------------------------
Communicator Log (... log before 10:00:29 all rolled out, missing ...) 01-16 10:00:29 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. 01-16 10:00:29 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:00:30 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000 01-16 10:00:30 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:00:30 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. . 01-16 10:00:39 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. 01-16 10:00:39 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:00:40 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000 01-16 10:00:40 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:00:40 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. . 01-16 10:00:49 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. 01-16 10:00:49 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:00:50 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000 01-16 10:00:50 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:00:50 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. . 01-16 10:00:59 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. 01-16 10:00:59 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:01:00 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000 01-16 10:01:00 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:01:00 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. . 01-16 10:01:09 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. 01-16 10:01:09 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:01:10 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000 01-16 10:01:10 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:01:10 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. . . . 01-16 10:05:50 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
ModAutoControl Log 01-16 05:15:30 Trigger "F14\F14 MCnt Sync" fires 01-16 05:15:30 Command is enqueued to be sent at 05:15:30. Out channel 1402 01-16 05:15:30 Sending trigger commands completed 01-16 05:15:30 Command is sent. Out channel 1402 01-16 05:15:30 Trigger "F14\CCuMix AccWt Sync" fires 01-16 05:15:30 Command is enqueued to be sent at 05:15:30. Out channel 9001 01-16 05:15:30 Sending trigger commands completed 01-16 05:15:30 Command is sent. Out channel 9001 01-16 05:15:33 Trigger "F14\F14 MCnt Sync" fires 01-16 05:15:33 Command is enqueued to be sent at 05:15:33. Out channel 1402 01-16 05:15:33 Sending trigger commands completed 01-16 05:15:33 Command is sent. Out channel 1402 01-16 05:15:37 Trigger "F14\F14 MCnt Sync" fires 01-16 05:15:37 Command is enqueued to be sent at 05:15:37. Out channel 1402 01-16 05:15:37 Sending trigger commands completed 01-16 05:15:37 Command is sent. Out channel 1402 01-16 05:15:39 Trigger "F14\F14 MCnt Sync" fires 01-16 05:15:39 Command is enqueued to be sent at 05:15:39. Out channel 1402 01-16 05:15:39 Sending trigger commands completed 01-16 05:15:39 Command is sent. Out channel 1402 01-16 05:15:41 Trigger "F14\CCuMix AccWt Sync" fires 01-16 05:15:41 Command is enqueued to be sent at 05:15:41. Out channel 9001 01-16 05:15:41 Sending trigger commands completed 01-16 05:15:41 Command is sent. Out channel 9001 01-16 05:15:42 Trigger "F14\F14 MCnt Sync" fires 01-16 05:15:42 Command is enqueued to be sent at 05:15:42. Out channel 1402 01-16 05:15:42 Sending trigger commands completed 01-16 05:15:42 Command is sent. Out channel 1402 01-16 05:15:44 Trigger "F14\F14 MCnt Sync" fires 01-16 05:15:44 Command is enqueued to be sent at 05:15:44. Out channel 1402 01-16 05:15:44 Sending trigger commands completed 01-16 05:15:44 Command is sent. Out channel 1402 01-16 05:15:47 Trigger "F14\F14 MCnt Sync" fires 01-16 05:15:47 Command is enqueued to be sent at 05:15:47. Out channel 1402 01-16 05:15:47 Sending trigger commands completed 01-16 05:15:47 Command is sent. Out channel 1402 01-16 10:05:12 Module ModAutoControl is stopped --------------------------------------------------------------------------------
Restart
-------------------------------------------------------------------------------- 01-16 10:05:45 <***C2><SYSTEM><ACT> ScadaServerService 5.1.4.0 is started 01-16 10:05:45 <***C2><SYSTEM><ACT> Module is loaded from the file C:\ProgramData\RapidSCADA\581\ScadaServer\Mod\ModAutoControl.dll 01-16 10:05:45 <***C2><SYSTEM><ACT> Check the existence of the data directories is completed successfully 01-16 10:05:45 <***C2><SYSTEM><ACT> Check the existence of the configuration database files is completed successfully 01-16 10:05:45 <***C2><SYSTEM><ACT> Input channels are read from the configuration database. Active channel count: 129 01-16 10:05:45 <***C2><SYSTEM><ACT> Ouput channels are read from the configuration database 01-16 10:05:45 <***C2><SYSTEM><ACT> Users are read from the configuration database 01-16 10:05:45 <***C2><SYSTEM><ACT> Formulas are read from the configuration database 01-16 10:05:46 <***C2><SYSTEM><ACT> The formulas source code has been compiled 01-16 10:05:46 <***C2><SYSTEM><ACT> Connection listener is started 01-16 10:05:46 <***C2><SYSTEM><ACT> Start server 01-16 10:05:46 <***C2><SYSTEM><ACT> Current data are loaded 01-16 10:05:46 <***C2><SYSTEM><ACT> Command: out channel = 1405, user ID = 0 01-16 10:05:50 <***C2><SYSTEM><ACT> Connect to client 127.0.0.1 01-16 10:14:22 <***C2><SYSTEM><ACT> Connection listener is stopped 01-16 10:14:22 <***C2><SYSTEM><EXC> Error receiving and processing data from the client 127.0.0.1: Thread was being aborted. 01-16 10:14:22 <***C2><SYSTEM><EXC> Error communicating with the client 127.0.0.1: Thread was being aborted. 01-16 10:14:32 <***C2><SYSTEM><EXC> Error executing actions on current data calculated in module ModAutoControl: Thread was being aborted. 01-16 10:14:32 <***C2><SYSTEM><ACT> Server is aborted 01-16 10:14:32 <***C2><SYSTEM><ACT> ScadaServerService is stopped --------------------------------------------------------------------------------
ModAutoControl -------------------------------------------------------------------------------- 01-16 10:05:46 Start ModAutoControl module 01-16 10:05:46 Check "Automatic control module" registration: Registration key is valid 01-16 10:05:46 Module state loaded from file 01-16 10:05:46 Variables in the commands are not used 01-16 10:05:46 Trigger state changed "F14\F14 AccWtStop": Repeat firing 2/2 (every 1 sec) 01-16 10:05:46 Command is enqueued to be sent at 10:05:46. Out channel 1405 01-16 10:05:46 Sending trigger commands completed 01-16 10:05:46 Command is sent. Out channel 1405 01-16 10:14:32 Module ModAutoControl is stopped --------------------------------------------------------------------------------
Tried restart CommSvc since, ServerLog didn’t show anything wrong. It’s jut CommSvc can’t connect to ServerSvc.
CommSvc Log 01-16 10:05:50 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:05:55 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 01-16 10:05:55 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:05:55 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. .(repeat) 01-16 10:06:00 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:06:05 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 01-16 10:06:05 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:06:05 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. .(repeat) 01-16 10:08:10 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. 01-16 10:08:10 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:08:15 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 01-16 10:08:15 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server .(repeat from 10:08:10) . . 01-16 10:08:16 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. 01-16 10:08:16 <***C2><SYSTEM><ACT> Stop communication lines 01-16 10:08:16 <***C2><SYSTEM><ACT> Receiving commands aborted 01-16 10:08:17 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. 01-16 10:08:17 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. 01-16 10:08:17 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. 01-16 10:08:27 <***C2><SYSTEM><ACT> ScadaCommService is stopped -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- 01-16 10:08:29 <***C2><SYSTEM><ACT> ScadaCommService 5.2.1.1 is started 01-16 10:08:29 <***C2><SYSTEM><ACT> Load device library: KpEmail.dll 01-16 10:08:29 <***C2><SYSTEM><ACT> Load device library: KpSnmp.dll 01-16 10:08:29 <***C2><SYSTEM><ACT> Load device library: KpModbus.dll 01-16 10:08:29 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:08:35 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 01-16 10:08:35 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:08:35 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error. Delay before trying again .(repeat from 10:08:29) . . 01-16 10:11:05 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:11:11 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 01-16 10:11:11 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:11:11 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error. Delay before trying again 01-16 10:11:20 <***C2><SYSTEM><ACT> ScadaCommService is stopped -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- 01-16 10:11:23 <***C2><SYSTEM><ACT> ScadaCommService 5.2.1.1 is started 01-16 10:11:23 <***C2><SYSTEM><ACT> Load device library: KpEmail.dll 01-16 10:11:23 <***C2><SYSTEM><ACT> Load device library: KpSnmp.dll 01-16 10:11:23 <***C2><SYSTEM><ACT> Load device library: KpModbus.dll 01-16 10:11:23 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:11:28 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 01-16 10:11:28 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:11:28 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error. Delay before trying again 01-16 10:11:38 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:11:44 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 01-16 10:11:44 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:11:44 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error. Delay before trying again 01-16 10:11:50 <***C2><SYSTEM><ACT> ScadaCommService is stopped -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- 01-16 10:11:52 <***C2><SYSTEM><ACT> ScadaCommService 5.2.1.1 is started 01-16 10:11:52 <***C2><SYSTEM><ACT> Load device library: KpEmail.dll 01-16 10:11:52 <***C2><SYSTEM><ACT> Load device library: KpSnmp.dll 01-16 10:11:52 <***C2><SYSTEM><ACT> Load device library: KpModbus.dll 01-16 10:11:52 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:11:58 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 01-16 10:11:58 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:11:58 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error. Delay before trying again 01-16 10:12:08 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:12:13 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 01-16 10:12:13 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:12:13 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error. Delay before trying again 01-16 10:12:24 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:12:26 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:12:26 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Thread was being aborted. 01-16 10:12:26 <***C2><SYSTEM><EXC> Error restoring connection with SCADA-Server: Thread was being aborted. 01-16 10:12:26 <***C2><SYSTEM><EXC> Error receiving configuration database table from Server: Thread was being aborted. 01-16 10:12:26 <***C2><SYSTEM><EXC> Error starting operation: System.Threading.ThreadAbortException: Thread was being aborted. at Scada.Client.ServerComm.ReceiveBaseTable(String tableName, IBaseTable baseTable) at Scada.Comm.Engine.Manager.ReceiveConfigBase(ConfigBaseSubset& configBase) at Scada.Comm.Engine.Manager.StartThreads() 01-16 10:12:26 <***C2><SYSTEM><ACT> ScadaCommService is stopped -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- 01-16 10:12:29 <***C2><SYSTEM><ACT> ScadaCommService 5.2.1.1 is started 01-16 10:12:29 <***C2><SYSTEM><ACT> Load device library: KpEmail.dll 01-16 10:12:29 <***C2><SYSTEM><ACT> Load device library: KpSnmp.dll 01-16 10:12:29 <***C2><SYSTEM><ACT> Load device library: KpModbus.dll 01-16 10:12:29 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:12:35 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 01-16 10:12:35 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:12:35 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error. Delay before trying again .(repeat from 10:12:29) . . 01-16 10:14:32 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:14:33 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000 01-16 10:14:33 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:14:33 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error. Delay before trying again
============= Restart
-------------------------------------------------------------------------------- 01-16 10:14:35 <***C2><SYSTEM><ACT> ScadaServerService 5.1.4.0 is started 01-16 10:14:35 <***C2><SYSTEM><ACT> Module is loaded from the file C:\ProgramData\RapidSCADA\581\ScadaServer\Mod\ModAutoControl.dll 01-16 10:14:35 <***C2><SYSTEM><ACT> Check the existence of the data directories is completed successfully 01-16 10:14:35 <***C2><SYSTEM><ACT> Check the existence of the configuration database files is completed successfully 01-16 10:14:35 <***C2><SYSTEM><ACT> Input channels are read from the configuration database. Active channel count: 129 01-16 10:14:35 <***C2><SYSTEM><ACT> Ouput channels are read from the configuration database 01-16 10:14:35 <***C2><SYSTEM><ACT> Users are read from the configuration database 01-16 10:14:35 <***C2><SYSTEM><ACT> Formulas are read from the configuration database 01-16 10:14:35 <***C2><SYSTEM><ACT> The formulas source code has been compiled 01-16 10:14:35 <***C2><SYSTEM><ACT> Connection listener is started 01-16 10:14:35 <***C2><SYSTEM><ACT> Start server 01-16 10:14:35 <***C2><SYSTEM><ACT> Current data are loaded 01-16 10:14:37 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0 01-16 10:14:43 <***C2><SYSTEM><ACT> Connect to client 127.0.0.1 01-16 10:17:12 <***C2><SYSTEM><EXC> Error receiving and processing data from the client 127.0.0.1: Thread was being aborted. 01-16 10:17:12 <***C2><SYSTEM><EXC> Error communicating with the client 127.0.0.1: Thread was being aborted. 01-16 10:17:12 <***C2><SYSTEM><ACT> Connection listener is stopped 01-16 10:17:22 <***C2><SYSTEM><EXC> Error executing actions on current data calculated in module ModAutoControl: Thread was being aborted. 01-16 10:17:22 <***C2><SYSTEM><ACT> Server is aborted 01-16 10:17:22 <***C2><SYSTEM><ACT> ScadaServerService is stopped --------------------------------------------------------------------------------
-------------------------------------------------------------------------------- 01-16 10:14:35 Start ModAutoControl module 01-16 10:14:35 Check "Automatic control module" registration: Registration key is valid 01-16 10:14:35 Module state loaded from file 01-16 10:14:35 Variables in the commands are not used 01-16 10:14:35 Trigger state changed "F14\F14 AccWtStop": Waiting 01-16 10:14:35 Trigger "F14\CCuMix AccWt Sync" fires 01-16 10:14:35 Command is enqueued to be sent at 10:14:35. Out channel 9001 01-16 10:14:35 Sending trigger commands completed 01-16 10:14:35 Command is sent. Out channel 9001 01-16 10:17:23 Module ModAutoControl is stopped --------------------------------------------------------------------------------
Delay before trying again 01-16 10:17:20 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:17:21 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000 01-16 10:17:21 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:17:21 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error. Delay before trying again
============ Restart
-------------------------------------------------------------------------------- 01-16 10:17:23 <***C2><SYSTEM><ACT> ScadaServerService 5.1.4.0 is started 01-16 10:17:23 <***C2><SYSTEM><ACT> Module is loaded from the file C:\ProgramData\RapidSCADA\581\ScadaServer\Mod\ModAutoControl.dll 01-16 10:17:23 <***C2><SYSTEM><ACT> Check the existence of the data directories is completed successfully 01-16 10:17:23 <***C2><SYSTEM><ACT> Check the existence of the configuration database files is completed successfully 01-16 10:17:23 <***C2><SYSTEM><ACT> Input channels are read from the configuration database. Active channel count: 129 01-16 10:17:23 <***C2><SYSTEM><ACT> Ouput channels are read from the configuration database 01-16 10:17:23 <***C2><SYSTEM><ACT> Users are read from the configuration database 01-16 10:17:23 <***C2><SYSTEM><ACT> Formulas are read from the configuration database 01-16 10:17:23 <***C2><SYSTEM><ACT> The formulas source code has been compiled 01-16 10:17:23 <***C2><SYSTEM><ACT> Connection listener is started 01-16 10:17:23 <***C2><SYSTEM><ACT> Start server 01-16 10:17:23 <***C2><SYSTEM><ACT> Current data are loaded 01-16 10:17:23 <***C2><SYSTEM><ACT> Command: out channel = 1405, user ID = 0 01-16 10:17:23 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0 01-16 10:17:31 <***C2><SYSTEM><ACT> Connect to client 127.0.0.1 01-16 10:17:31 <***C2><SYSTEM><ACT> The user ScadaComm is successfully authenticated 01-16 10:17:33 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 10:17:37 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 10:17:39 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 10:17:42 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 10:17:44 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 10:17:47 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 10:17:49 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 10:17:51 <***C2><SYSTEM><ACT> Connection listener is stopped 01-16 10:17:51 <***C2><SYSTEM><ACT> Server is stopped 01-16 10:17:51 <***C2><SYSTEM><ACT> ScadaServerService is stopped --------------------------------------------------------------------------------
-------------------------------------------------------------------------------- 01-16 10:17:23 Start ModAutoControl module 01-16 10:17:23 Check "Automatic control module" registration: Registration key is valid 01-16 10:17:23 Module state loaded from file 01-16 10:17:23 Variables in the commands are not used 01-16 10:17:23 Trigger state changed "F14\F14 AutoStop": Repeat firing 2/2 (every 1 sec) 01-16 10:17:23 Command is enqueued to be sent at 10:17:23. Out channel 1405 01-16 10:17:23 Sending trigger commands completed 01-16 10:17:23 Command is sent. Out channel 1405 01-16 10:17:23 Trigger "F14\CCuMix AccWt Sync" fires 01-16 10:17:23 Command is enqueued to be sent at 10:17:23. Out channel 9001 01-16 10:17:23 Sending trigger commands completed 01-16 10:17:23 Command is sent. Out channel 9001 01-16 10:17:32 Trigger state changed "F14\F14 Stop: PH3": Waiting 01-16 10:17:32 Trigger state changed "F14\F14 Stop: Flow1": Waiting 01-16 10:17:32 Trigger state changed "F14\F14 Stop: Temp1": Waiting 01-16 10:17:33 Trigger state changed "F14\F14 Stop: Temp2": Waiting 01-16 10:17:33 Trigger "F14\F14 MCnt Sync" fires 01-16 10:17:33 Command is enqueued to be sent at 10:17:33. Out channel 1402 01-16 10:17:33 Sending trigger commands completed 01-16 10:17:33 Command is sent. Out channel 1402 01-16 10:17:37 Trigger "F14\F14 MCnt Sync" fires 01-16 10:17:37 Command is enqueued to be sent at 10:17:37. Out channel 1402 01-16 10:17:37 Sending trigger commands completed 01-16 10:17:37 Command is sent. Out channel 1402 01-16 10:17:39 Trigger "F14\F14 MCnt Sync" fires 01-16 10:17:39 Command is enqueued to be sent at 10:17:39. Out channel 1402 01-16 10:17:39 Sending trigger commands completed 01-16 10:17:39 Command is sent. Out channel 1402 01-16 10:17:42 Trigger "F14\F14 MCnt Sync" fires 01-16 10:17:42 Command is enqueued to be sent at 10:17:42. Out channel 1402 01-16 10:17:42 Sending trigger commands completed 01-16 10:17:42 Command is sent. Out channel 1402 01-16 10:17:44 Trigger "F14\F14 MCnt Sync" fires 01-16 10:17:44 Command is enqueued to be sent at 10:17:44. Out channel 1402 01-16 10:17:44 Sending trigger commands completed 01-16 10:17:44 Command is sent. Out channel 1402 01-16 10:17:47 Trigger "F14\F14 MCnt Sync" fires 01-16 10:17:47 Command is enqueued to be sent at 10:17:47. Out channel 1402 01-16 10:17:47 Sending trigger commands completed 01-16 10:17:47 Command is sent. Out channel 1402 01-16 10:17:49 Trigger "F14\F14 MCnt Sync" fires 01-16 10:17:49 Command is enqueued to be sent at 10:17:49. Out channel 1402 01-16 10:17:49 Sending trigger commands completed 01-16 10:17:49 Command is sent. Out channel 1402 01-16 10:17:51 Module ModAutoControl is stopped --------------------------------------------------------------------------------
CommSvc Log (Delay before trying again)
==== Restart after ModAutoControl is stopped
-------------------------------------------------------------------------------- 01-16 10:17:54 <***C2><SYSTEM><ACT> ScadaServerService 5.1.4.0 is started 01-16 10:17:54 <***C2><SYSTEM><ACT> Check the existence of the data directories is completed successfully 01-16 10:17:54 <***C2><SYSTEM><ACT> Check the existence of the configuration database files is completed successfully 01-16 10:17:54 <***C2><SYSTEM><ACT> Input channels are read from the configuration database. Active channel count: 129 01-16 10:17:54 <***C2><SYSTEM><ACT> Ouput channels are read from the configuration database 01-16 10:17:54 <***C2><SYSTEM><ACT> Users are read from the configuration database 01-16 10:17:54 <***C2><SYSTEM><ACT> Formulas are read from the configuration database 01-16 10:17:54 <***C2><SYSTEM><ACT> The formulas source code has been compiled 01-16 10:17:54 <***C2><SYSTEM><ACT> Connection listener is started 01-16 10:17:54 <***C2><SYSTEM><ACT> Start server 01-16 10:17:54 <***C2><SYSTEM><ACT> Current data are loaded 01-16 10:18:01 <***C2><SYSTEM><ACT> Connect to client 127.0.0.1 01-16 10:18:01 <***C2><SYSTEM><ACT> The user ScadaComm is successfully authenticated 01-16 10:18:17 <***C2><SYSTEM><ACT> Connection listener is stopped 01-16 10:18:17 <***C2><SYSTEM><ACT> Server is stopped 01-16 10:18:17 <***C2><SYSTEM><ACT> ScadaServerService is stopped --------------------------------------------------------------------------------
CommSvc Log 01-16 10:17:31 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:17:31 <***C2><SYSTEM><ACT> Start communication lines 01-16 10:17:31 <***C2><SYSTEM><ACT> Start receiving commands 01-16 10:17:33 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1 01-16 10:17:37 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1 01-16 10:17:39 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1 01-16 10:17:43 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1 01-16 10:17:44 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1 01-16 10:17:48 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1 01-16 10:17:49 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1 01-16 10:17:51 <***C2><SYSTEM><EXC> Error requesting telecommand from SCADA-Server: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. 01-16 10:17:51 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:17:51 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:17:52 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000 01-16 10:17:52 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:17:52 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. 01-16 10:17:53 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. .(repeat) 01-16 10:18:01 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:18:17 <***C2><SYSTEM><EXC> Error requesting telecommand from SCADA-Server: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. 01-16 10:18:17 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:18:17 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:18:18 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000 01-16 10:18:18 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server 01-16 10:18:18 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again. .(repeat)
==== Restart after ModAutoControl is turned back on. Back to Normal.
-------------------------------------------------------------------------------- 01-16 10:18:19 <***C2><SYSTEM><ACT> ScadaServerService 5.1.4.0 is started 01-16 10:18:20 <***C2><SYSTEM><ACT> Module is loaded from the file C:\ProgramData\RapidSCADA\581\ScadaServer\Mod\ModAutoControl.dll 01-16 10:18:20 <***C2><SYSTEM><ACT> Check the existence of the data directories is completed successfully 01-16 10:18:20 <***C2><SYSTEM><ACT> Check the existence of the configuration database files is completed successfully 01-16 10:18:20 <***C2><SYSTEM><ACT> Input channels are read from the configuration database. Active channel count: 129 01-16 10:18:20 <***C2><SYSTEM><ACT> Ouput channels are read from the configuration database 01-16 10:18:20 <***C2><SYSTEM><ACT> Users are read from the configuration database 01-16 10:18:20 <***C2><SYSTEM><ACT> Formulas are read from the configuration database 01-16 10:18:20 <***C2><SYSTEM><ACT> The formulas source code has been compiled 01-16 10:18:20 <***C2><SYSTEM><ACT> Connection listener is started 01-16 10:18:20 <***C2><SYSTEM><ACT> Start server 01-16 10:18:20 <***C2><SYSTEM><ACT> Current data are loaded 01-16 10:18:20 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0 01-16 10:18:27 <***C2><SYSTEM><ACT> Connect to client 127.0.0.1 01-16 10:18:27 <***C2><SYSTEM><ACT> The user ScadaComm is successfully authenticated 01-16 10:18:31 <***C2><SYSTEM><ACT> Command: out channel = 1405, user ID = 0 01-16 10:18:32 <***C2><SYSTEM><ACT> Command: out channel = 1405, user ID = 0 01-16 10:18:34 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0 01-16 10:18:34 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0 01-16 10:18:34 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0 01-16 10:19:00 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0 01-16 10:19:04 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0 01-16 10:19:30 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0 01-16 10:19:34 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0 01-16 10:19:46 <***C2><SYSTEM><ACT> Connect to client 127.0.0.1 01-16 10:19:46 <***C2><SYSTEM><ACT> The user ScadaWeb is successfully authenticated 01-16 10:19:47 <***C2><SYSTEM><ACT> Check username and password for admin. Result: passed
-------------------------------------------------------------------------------- 01-16 10:18:20 Start ModAutoControl module 01-16 10:18:20 Check "Automatic control module" registration: Registration key is valid 01-16 10:18:20 Module state loaded from file 01-16 10:18:20 Variables in the commands are not used 01-16 10:18:20 Trigger "F14\CCuMix AccWt Sync" fires 01-16 10:18:20 Command is enqueued to be sent at 10:18:20. Out channel 9001 01-16 10:18:20 Sending trigger commands completed 01-16 10:18:20 Command is sent. Out channel 9001 01-16 10:18:27 Trigger state changed "F14\F14 AutoStop": Waiting 01-16 10:18:27 No trigger commands to send 01-16 10:18:29 Trigger state changed "F14\F14 Stop: Flow1": Firing 01-16 10:18:29 Sending commands skipped 01-16 10:18:29 Trigger state changed "F14\F14 Stop: Temp1": Waiting 01-16 10:18:30 Trigger state changed "F14\F14 Stop: Temp2": Firing 01-16 10:18:30 Sending commands skipped 01-16 10:18:31 Trigger state changed "F14\F14 ": Firing 01-16 10:18:31 Command is enqueued to be sent at 10:18:31. Out channel 1405 01-16 10:18:31 Sending trigger commands completed 01-16 10:18:31 Command is sent. Out channel 1405 01-16 10:18:32 Trigger state changed "F14\F14 ": Repeat firing 2/2 (every 1 sec) 01-16 10:18:32 Command is enqueued to be sent at 10:18:32. Out channel 1405 01-16 10:18:32 Sending trigger commands completed 01-16 10:18:32 Command is sent. Out channel 1405 01-16 10:18:32 Trigger state changed "F14\F14 Stop: PH3": Waiting 01-16 10:18:34 Trigger "F14\F14 MCnt Sync" fires 01-16 10:18:34 Command is enqueued to be sent at 10:18:34. Out channel 1402 01-16 10:18:34 Sending trigger commands completed 01-16 10:18:34 Command is sent. Out channel 1402 01-16 10:18:34 Trigger "F14\CCuMix AccWt Sync" fires 01-16 10:18:34 Command is enqueued to be sent at 10:18:34. Out channel 9001 01-16 10:18:34 Sending trigger commands completed
CommSvc Log 01-16 10:18:27 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost" 01-16 10:18:31 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=4 01-16 10:18:32 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=4 01-16 10:18:34 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1 01-16 10:18:34 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=190, number=1 01-16 10:18:34 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=190, number=1 . . .
Could you privde a produre to troubleshoot such problem in general ?
It just happend to me that I might try disableing AutomaticControl first.
That’s pure luck though.- This topic was modified 3 years, 9 months ago by Mikhail.
January 19, 2021 at 1:24 pm #8093MikhailModeratorHi,
How often does the problem occur?
What is your scenario of using Auto Control Module?It just happend to me that I might try disableing AutomaticControl first.
Agree. This is a possible cause.
January 19, 2021 at 1:25 pm #8094MikhailModeratorAlso provide a screenshot of the trigger settings that sends the last command:
01-16 05:15:47 <***C2>Command: out channel = 1402, user ID = 0 January 21, 2021 at 3:18 am #8119zzzParticipant> How often does the problem occur?
Can’t say, sometimes it run fine for days w/o problem. Maybe could continue if not restarted? Sometimes it dies silently hours after restart.Yesterday moring, I found it stopped working around 01/19/23:30, about 6 hours after a restart. I can’t find the log anymore, it’s been overwritten. But everytime the server is restart after the silent death, it always give the
ThreadAbortException
message like shown in the above logs.I suspect some of the values in the channels got AutoControl crazy.
Every time I restart the ServerSrv w/ autocontrol disabled, I have to wait for the communicator to fully work. Or the next restsart with ServerSrv w/ autocontrol enabled, will usually fail. Pity, I just thought about this, can’t find snapshot of those values anymore. The archive is at 30 secs interval.
Mybe this could be an option for some troubleshoot feature to save a snapshot of current values.Could you shad some light, how could autocontrol module block the server thread processing communicator communiations?
> What is your scenario of using Auto Control Module?
– Sync data from sensor bus line to PLC bus lines.
– Unified control for process metrics.
– Execution of control message from production system.AutoControl Rules
<TrGroup active="true" name="F14"> <Trigger id="9" active="true" name="F14 WtReached" type="CnlDataTrigger"> <CnlDataTriggerProps> <CnlNum>11410</CnlNum> <ChkCnlVal>true</ChkCnlVal> <CnlValCond> <CompareOperator1>GreaterThan</CompareOperator1> <CompareArgument1>905</CompareArgument1> <LogicalOperator>None</LogicalOperator> <CompareOperator2>LessThan</CompareOperator2> <CompareArgument2>0</CompareArgument2> </CnlValCond> <Deadband>0</Deadband> <ChkCnlStat>false</ChkCnlStat> <CnlStat /> <Delay>0</Delay> <Repeat>1</Repeat> <RepeatCnt>2</RepeatCnt> </CnlDataTriggerProps> <Commands> <Command cmdTypeID="0"> <KPNum>0</KPNum> <CmdNum>4</CmdNum> <CtrlCnlNum>1405</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>false</CopyCmdVal> <CmdVal>1</CmdVal> </Command> </Commands> </Trigger> <Trigger id="3" active="true" name="Reset MCounter & MES CtrlBits On PreStart" type="EvTrigger"> <EvTriggerProps> <ChkObjNum>false</ChkObjNum> <ObjNum>2</ObjNum> <ChkKPNum>false</ChkKPNum> <KPNum>114</KPNum> <ChkParamID>false</ChkParamID> <ParamID>0</ParamID> <ChkCnlNum>true</ChkCnlNum> <CnlNum>11404</CnlNum> <ChkNewCnlVal>true</ChkNewCnlVal> <NewCnlValCond> <CompareOperator1>Equal</CompareOperator1> <CompareArgument1>1</CompareArgument1> <LogicalOperator>None</LogicalOperator> <CompareOperator2>LessThan</CompareOperator2> <CompareArgument2>0</CompareArgument2> </NewCnlValCond> <ChkCnlStat>false</ChkCnlStat> <NewCnlStat>0</NewCnlStat> <DescrCompareOper>None</DescrCompareOper> <Descr /> <DataCompareOper>None</DataCompareOper> <Data>1</Data> </EvTriggerProps> <Commands> <Command cmdTypeID="0"> <KPNum>0</KPNum> <CmdNum>1</CmdNum> <CtrlCnlNum>1401</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>false</CopyCmdVal> <CmdVal>1</CmdVal> </Command> <Command cmdTypeID="0"> <KPNum>0</KPNum> <CmdNum>1</CmdNum> <CtrlCnlNum>1402</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>false</CopyCmdVal> <CmdVal>0</CmdVal> </Command> <Command cmdTypeID="0"> <KPNum>0</KPNum> <CmdNum>1</CmdNum> <CtrlCnlNum>1403</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>false</CopyCmdVal> <CmdVal>0</CmdVal> </Command> </Commands> </Trigger> <Trigger id="16" active="true" name="F14 Flow" type="CnlDataTrigger"> <CnlDataTriggerProps> <CnlNum>1403</CnlNum> <ChkCnlVal>true</ChkCnlVal> <CnlValCond> <CompareOperator1>LessThan</CompareOperator1> <CompareArgument1>*00</CompareArgument1> <LogicalOperator>None</LogicalOperator> <CompareOperator2>LessThan</CompareOperator2> <CompareArgument2>0</CompareArgument2> </CnlValCond> <Deadband>1</Deadband> <ChkCnlStat>false</ChkCnlStat> <CnlStat /> <Delay>60</Delay> <Repeat>0</Repeat> <RepeatCnt>0</RepeatCnt> </CnlDataTriggerProps> <Commands> <Command cmdTypeID="0"> <KPNum>0</KPNum> <CmdNum>1</CmdNum> <CtrlCnlNum>1403</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>false</CopyCmdVal> <CmdVal>9</CmdVal> </Command> <Command cmdTypeID="1"> <KPNum>0</KPNum> <CmdNum>1</CmdNum> <CtrlCnlNum>1404</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>false</CopyCmdVal> <EditMode>Str</EditMode> <CmdDataStr>Flow overlimit</CmdDataStr> </Command> </Commands> </Trigger> <Trigger id="18" active="true" name="F14 Temp" type="CnlDataTrigger"> <CnlDataTriggerProps> <CnlNum>1407</CnlNum> <ChkCnlVal>true</ChkCnlVal> <CnlValCond> <CompareOperator1>LessThan</CompareOperator1> <CompareArgument1>*00</CompareArgument1> <LogicalOperator>Or</LogicalOperator> <CompareOperator2>GreaterThan</CompareOperator2> <CompareArgument2>*00</CompareArgument2> </CnlValCond> <Deadband>1</Deadband> <ChkCnlStat>false</ChkCnlStat> <CnlStat /> <Delay>60</Delay> <Repeat>0</Repeat> <RepeatCnt>0</RepeatCnt> </CnlDataTriggerProps> <Commands> <Command cmdTypeID="0"> <KPNum>0</KPNum> <CmdNum>1</CmdNum> <CtrlCnlNum>1403</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>false</CopyCmdVal> <CmdVal>9</CmdVal> </Command> <Command cmdTypeID="1"> <KPNum>0</KPNum> <CmdNum>1</CmdNum> <CtrlCnlNum>1404</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>false</CopyCmdVal> <EditMode>Str</EditMode> <CmdDataStr>Temp over limit</CmdDataStr> </Command> </Commands> </Trigger> <Trigger id="19" active="true" name="F14 Temp2" type="CnlDataTrigger"> <CnlDataTriggerProps> <CnlNum>1406</CnlNum> <ChkCnlVal>true</ChkCnlVal> <CnlValCond> <CompareOperator1>LessThan</CompareOperator1> <CompareArgument1>*0</CompareArgument1> <LogicalOperator>Or</LogicalOperator> <CompareOperator2>GreaterThan</CompareOperator2> <CompareArgument2>*0</CompareArgument2> </CnlValCond> <Deadband>1</Deadband> <ChkCnlStat>false</ChkCnlStat> <CnlStat /> <Delay>60</Delay> <Repeat>0</Repeat> <RepeatCnt>0</RepeatCnt> </CnlDataTriggerProps> <Commands> <Command cmdTypeID="0"> <KPNum>0</KPNum> <CmdNum>1</CmdNum> <CtrlCnlNum>1403</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>false</CopyCmdVal> <CmdVal>9</CmdVal> </Command> <Command cmdTypeID="1"> <KPNum>0</KPNum> <CmdNum>1</CmdNum> <CtrlCnlNum>1404</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>false</CopyCmdVal> <EditMode>Str</EditMode> <CmdDataStr>Temp2 over limit</CmdDataStr> </Command> </Commands> </Trigger> <Trigger id="21" active="true" name="F14 PH3" type="CnlDataTrigger"> <CnlDataTriggerProps> <CnlNum>1402</CnlNum> <ChkCnlVal>true</ChkCnlVal> <CnlValCond> <CompareOperator1>LessThan</CompareOperator1> <CompareArgument1>6</CompareArgument1> <LogicalOperator>None</LogicalOperator> <CompareOperator2>LessThan</CompareOperator2> <CompareArgument2>0</CompareArgument2> </CnlValCond> <Deadband>0</Deadband> <ChkCnlStat>false</ChkCnlStat> <CnlStat /> <Delay>90</Delay> <Repeat>0</Repeat> <RepeatCnt>0</RepeatCnt> </CnlDataTriggerProps> <Commands> <Command cmdTypeID="0"> <KPNum>0</KPNum> <CmdNum>1</CmdNum> <CtrlCnlNum>1403</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>false</CopyCmdVal> <CmdVal>9</CmdVal> </Command> <Command cmdTypeID="1"> <KPNum>0</KPNum> <CmdNum>1</CmdNum> <CtrlCnlNum>1404</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>false</CopyCmdVal> <EditMode>Str</EditMode> <CmdDataStr>PH3 over limit</CmdDataStr> </Command> </Commands> </Trigger> <Trigger id="5" active="true" name="F14 MCnt Sync" type="CnlDataChangeTrigger"> <CnlDataChangeTriggerProps> <CnlNum>1410</CnlNum> </CnlDataChangeTriggerProps> <Commands> <Command cmdTypeID="0"> <KPNum>0</KPNum> <CmdNum>1</CmdNum> <CtrlCnlNum>1402</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>true</CopyCmdVal> <CmdVal>0</CmdVal> </Command> </Commands> </Trigger> <Trigger id="14" active="true" name="CCuMix AccWt Sync" type="CnlDataChangeTrigger"> <CnlDataChangeTriggerProps> <CnlNum>19001</CnlNum> </CnlDataChangeTriggerProps> <Commands> <Command cmdTypeID="0"> <KPNum>0</KPNum> <CmdNum>1</CmdNum> <CtrlCnlNum>9001</CtrlCnlNum> <UserID>0</UserID> <Delay>0</Delay> <SendOnFiring>true</SendOnFiring> <SendOnNorm>false</SendOnNorm> <CopyCmdVal>true</CopyCmdVal> <CmdVal>0</CmdVal> </Command> </Commands> </Trigger> </TrGroup>
January 21, 2021 at 10:50 am #8124JWParticipantI had experienced similar things before when:
Using Auto Control Module to send commands to one external modbus device.
The cause I suspect:
1 trigger send multiple commands;
or several triggers send commands at the same time;It happens quite frequently when doing stress test, e.g. toggling the trigger every a few second.
as I remember in detail log, it’s similar to the issue I mentioned in another post. (Modbus Comm Bug)
it looks like scada mis-align the reply from the modbus slave software.
https://drive.google.com/file/d/11y8e6zyzzKWXJovF0uj8TM22hCG9TJkH/view?usp=sharingI am going to find my records, as see if I can make a minimum project that reproduce this problem.
January 21, 2021 at 2:15 pm #8131MikhailModeratorI was asking for a screenshot, not XML. I can check the settings if you provide it.
January 22, 2021 at 5:29 am #8135zzzParticipantNot that I don’t want to provide, but all file hosting service here require registration with cellphone which is now required by law, almost all foreign free img hosting are blocked.
January 22, 2021 at 6:26 am #8137manjey73ParticipantUse a VPN in your browser, which should help you get access to image hosting services
January 22, 2021 at 6:31 am #8138zzzParticipant@MikHail: I have sent you the img via email. Unless someone has some not-yet-blocked less-known image hosting to recommend. I’v nowhere public to upload img.
@123abc56: You may be right about the racing condition. But anyhow, I guess we need thatcurrent data
table at that moment to inspect the problem. Too bad I didn’t snapshot that file.January 22, 2021 at 1:57 pm #8141MikhailModeratorUse a VPN in your browser, which should help you get access to image hosting services
It may be complicated in China. If somebody has experience with that, please share.
January 22, 2021 at 1:59 pm #8142MikhailModeratorAs a workaround of the hanging problem, I suggest to set delay of the command to 1 instead of 0. In this case command is send using a queue that is more reliable. Let us know about the result.
January 23, 2021 at 12:06 pm #8147JWParticipantHi, I have a few findings.
1. when the server crashed, there is no log.
the following part of the error message shows up when manually stop/restart the server.System.Threading.ThreadAbortException: Thread was being aborted. at System.Threading.Monitor.ReliableEnter(Object obj, Boolean& lockTaken) at Scada.Server.Engine.MainLogic.RaiseOnCurDataProcessing(Srez receivedSrez) at Scada.Server.Engine.MainLogic.ProcCurData(Srez receivedSrez) 01-16 10:05:02 <***C2><SYSTEM><EXC> Error receiving and processing data from the client 127.0.0.1: Thread was being aborted. 01-16 10:05:02 <***C2><SYSTEM><EXC> Error communicating with the client 127.0.0.1: Thread was being aborted. 01-16 10:05:12 <***C2><SYSTEM><EXC> Error executing actions on current data calculated in module ModAutoControl: Thread was being aborted. 01-16 10:05:12 <***C2><SYSTEM><ACT> Server is aborted 01-16 10:05:12 <***C2><SYSTEM><ACT> ScadaServerService is stopped
2. the server will crash even the device it’s sending command to is not online, so it may not be the communicator problem I mentioned.
3. Adding 1s delay will reduce the probability / frequency of the server crash, but can not complete solve it.
January 23, 2021 at 3:10 pm #8149JWParticipantHi,
I made a test project that will trigger this crash issue consistently within 1~2min. It can be downloaded in the link below.
Test project
https://drive.google.com/file/d/1l5Z4y8vRrVlked83ZzkT9AtmZpPZ61LI/view?usp=sharingProject settings
The data is generated from simulator driver of scada, with 1s update interval.
The alarm generated from simulator will trigger commands that send to a modbus relay device. Commands are with 1s delay already.
It will crash no matter the relay is connected or not. (You can connect a modbus slave/server simulator with 16 DO at 127.0.0.1:502 to see the command in action.)Web UI
https://drive.google.com/file/d/17proisKbU5Gf3xLQ0bRhxrgphJ8h9Sr7/view?usp=sharing
Administrator and server Log
https://drive.google.com/file/d/1RcClfPdw3ozifkd5IMGiUuf_xvVgoSeb/view?usp=sharing
Auto Control settings
https://drive.google.com/file/d/1S5FPVlr9OMcmyV1tDK0NPWtcWE_2ftz-/view?usp=sharingOther notes
If turn off auto control module, it will not crash on my 1.5 hour test.
If add 1s more delay after request cycle to simulator, or set the delay of simulator device from 1s to 2s, the crash will happen less frequently. (but I have no license of auto control module on my this computer, I can only test for 10mins per round. I tested 4-5 rounds, only 1 round crashed at about 3 minutes.)January 24, 2021 at 7:36 am #8151MikhailModeratorHi,
Thanks a lot. I will try to reproduce the problem using you project and write here.
Adding 1s delay will reduce the probability / frequency of the server crash, but can not complete solve it.
I mean adding a delay to a command in the module settings, not to request. If delay is added, commands are added to a queue instead of being sent directly. The queue should protect the server app from hanging.
but I have no license of auto control module on my this computer
Get a trial key https://rapidscada.net/trial/TrialKeyGen.aspx
January 25, 2021 at 3:52 am #8154JWParticipantI mean adding a delay to a command in the module settings, not to request. If delay is added, commands are added to a queue instead of being sent directly. The queue should protect the server app from hanging.
Yes, 1s delay to commands in the auto module are added already in the above test project, which do reduce the crash probability / frequency. But it will still crash on this setting.
Then I also try to test the effect of different request delay.
seems they both have effect on this issue.
-
AuthorPosts
- You must be logged in to reply to this topic.