Silent server crash

Forum Home Forums Runtime Bugs Server Bugs Silent server crash

Viewing 15 posts - 1 through 15 (of 22 total)
  • Author
    Posts
  • #8083
    zzz
    Participant

    Everything stopped working after 5:15:00. Retart manually around 10:00.

    
    Server Log tail
    
    01-16 05:15:30 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 05:15:30 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0
    01-16 05:15:33 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 05:15:37 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 05:15:39 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 05:15:41 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0
    01-16 05:15:42 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 05:15:44 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 05:15:47 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 10:05:02 <***C2><SYSTEM><ACT> Connection listener is stopped
    01-16 10:05:02 <***C2><SYSTEM><EXC> Error processing new current data:
    System.Threading.ThreadAbortException: Thread was being aborted.
       at System.Threading.Monitor.ReliableEnter(Object obj, Boolean& lockTaken)
       at Scada.Server.Engine.MainLogic.RaiseOnCurDataProcessing(Srez receivedSrez)
       at Scada.Server.Engine.MainLogic.ProcCurData(Srez receivedSrez)
    01-16 10:05:02 <***C2><SYSTEM><EXC> Error receiving and processing data from the client 127.0.0.1: Thread was being aborted.
    01-16 10:05:02 <***C2><SYSTEM><EXC> Error communicating with the client 127.0.0.1: Thread was being aborted.
    01-16 10:05:12 <***C2><SYSTEM><EXC> Error executing actions on current data calculated in module ModAutoControl: Thread was being aborted.
    01-16 10:05:12 <***C2><SYSTEM><ACT> Server is aborted
    01-16 10:05:12 <***C2><SYSTEM><ACT> ScadaServerService is stopped
    --------------------------------------------------------------------------------
    
    
    Communicator Log
    
    (... log before 10:00:29 all rolled out, missing ...)
    01-16 10:00:29 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    01-16 10:00:29 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:00:30 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000
    01-16 10:00:30 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:00:30 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    .
    01-16 10:00:39 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    01-16 10:00:39 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:00:40 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000
    01-16 10:00:40 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:00:40 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    .
    01-16 10:00:49 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    01-16 10:00:49 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:00:50 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000
    01-16 10:00:50 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:00:50 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    .
    01-16 10:00:59 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    01-16 10:00:59 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:01:00 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000
    01-16 10:01:00 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:01:00 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    .
    01-16 10:01:09 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    01-16 10:01:09 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:01:10 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000
    01-16 10:01:10 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:01:10 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    .
    .
    .
    01-16 10:05:50 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    
    
    ModAutoControl Log
    
    01-16 05:15:30 Trigger "F14\F14 MCnt Sync" fires
    01-16 05:15:30 Command is enqueued to be sent at 05:15:30. Out channel 1402
    01-16 05:15:30 Sending trigger commands completed
    01-16 05:15:30 Command is sent. Out channel 1402
    01-16 05:15:30 Trigger "F14\CCuMix AccWt Sync" fires
    01-16 05:15:30 Command is enqueued to be sent at 05:15:30. Out channel 9001
    01-16 05:15:30 Sending trigger commands completed
    01-16 05:15:30 Command is sent. Out channel 9001
    01-16 05:15:33 Trigger "F14\F14 MCnt Sync" fires
    01-16 05:15:33 Command is enqueued to be sent at 05:15:33. Out channel 1402
    01-16 05:15:33 Sending trigger commands completed
    01-16 05:15:33 Command is sent. Out channel 1402
    01-16 05:15:37 Trigger "F14\F14 MCnt Sync" fires
    01-16 05:15:37 Command is enqueued to be sent at 05:15:37. Out channel 1402
    01-16 05:15:37 Sending trigger commands completed
    01-16 05:15:37 Command is sent. Out channel 1402
    01-16 05:15:39 Trigger "F14\F14 MCnt Sync" fires
    01-16 05:15:39 Command is enqueued to be sent at 05:15:39. Out channel 1402
    01-16 05:15:39 Sending trigger commands completed
    01-16 05:15:39 Command is sent. Out channel 1402
    01-16 05:15:41 Trigger "F14\CCuMix AccWt Sync" fires
    01-16 05:15:41 Command is enqueued to be sent at 05:15:41. Out channel 9001
    01-16 05:15:41 Sending trigger commands completed
    01-16 05:15:41 Command is sent. Out channel 9001
    01-16 05:15:42 Trigger "F14\F14 MCnt Sync" fires
    01-16 05:15:42 Command is enqueued to be sent at 05:15:42. Out channel 1402
    01-16 05:15:42 Sending trigger commands completed
    01-16 05:15:42 Command is sent. Out channel 1402
    01-16 05:15:44 Trigger "F14\F14 MCnt Sync" fires
    01-16 05:15:44 Command is enqueued to be sent at 05:15:44. Out channel 1402
    01-16 05:15:44 Sending trigger commands completed
    01-16 05:15:44 Command is sent. Out channel 1402
    01-16 05:15:47 Trigger "F14\F14 MCnt Sync" fires
    01-16 05:15:47 Command is enqueued to be sent at 05:15:47. Out channel 1402
    01-16 05:15:47 Sending trigger commands completed
    01-16 05:15:47 Command is sent. Out channel 1402
    01-16 10:05:12 Module ModAutoControl is stopped
    --------------------------------------------------------------------------------
    

    Restart

    
    --------------------------------------------------------------------------------
    01-16 10:05:45 <***C2><SYSTEM><ACT> ScadaServerService 5.1.4.0 is started
    01-16 10:05:45 <***C2><SYSTEM><ACT> Module is loaded from the file C:\ProgramData\RapidSCADA\581\ScadaServer\Mod\ModAutoControl.dll
    01-16 10:05:45 <***C2><SYSTEM><ACT> Check the existence of the data directories is completed successfully
    01-16 10:05:45 <***C2><SYSTEM><ACT> Check the existence of the configuration database files is completed successfully
    01-16 10:05:45 <***C2><SYSTEM><ACT> Input channels are read from the configuration database. Active channel count: 129
    01-16 10:05:45 <***C2><SYSTEM><ACT> Ouput channels are read from the configuration database
    01-16 10:05:45 <***C2><SYSTEM><ACT> Users are read from the configuration database
    01-16 10:05:45 <***C2><SYSTEM><ACT> Formulas are read from the configuration database
    01-16 10:05:46 <***C2><SYSTEM><ACT> The formulas source code has been compiled
    01-16 10:05:46 <***C2><SYSTEM><ACT> Connection listener is started
    01-16 10:05:46 <***C2><SYSTEM><ACT> Start server
    01-16 10:05:46 <***C2><SYSTEM><ACT> Current data are loaded
    01-16 10:05:46 <***C2><SYSTEM><ACT> Command: out channel = 1405, user ID = 0
    01-16 10:05:50 <***C2><SYSTEM><ACT> Connect to client 127.0.0.1
    01-16 10:14:22 <***C2><SYSTEM><ACT> Connection listener is stopped
    01-16 10:14:22 <***C2><SYSTEM><EXC> Error receiving and processing data from the client 127.0.0.1: Thread was being aborted.
    01-16 10:14:22 <***C2><SYSTEM><EXC> Error communicating with the client 127.0.0.1: Thread was being aborted.
    01-16 10:14:32 <***C2><SYSTEM><EXC> Error executing actions on current data calculated in module ModAutoControl: Thread was being aborted.
    01-16 10:14:32 <***C2><SYSTEM><ACT> Server is aborted
    01-16 10:14:32 <***C2><SYSTEM><ACT> ScadaServerService is stopped
    --------------------------------------------------------------------------------
    
    
    ModAutoControl
    --------------------------------------------------------------------------------
    01-16 10:05:46 Start ModAutoControl module
    01-16 10:05:46 Check "Automatic control module" registration:
    Registration key is valid
    01-16 10:05:46 Module state loaded from file
    01-16 10:05:46 Variables in the commands are not used
    01-16 10:05:46 Trigger state changed "F14\F14 AccWtStop": Repeat firing 2/2 (every 1 sec)
    01-16 10:05:46 Command is enqueued to be sent at 10:05:46. Out channel 1405
    01-16 10:05:46 Sending trigger commands completed
    01-16 10:05:46 Command is sent. Out channel 1405
    01-16 10:14:32 Module ModAutoControl is stopped
    --------------------------------------------------------------------------------
    

    Tried restart CommSvc since, ServerLog didn’t show anything wrong. It’s jut CommSvc can’t connect to ServerSvc.

    
    CommSvc Log
    
    01-16 10:05:50 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:05:55 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
    01-16 10:05:55 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:05:55 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    .(repeat)
    01-16 10:06:00 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:06:05 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
    01-16 10:06:05 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:06:05 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    .(repeat)
    01-16 10:08:10 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    01-16 10:08:10 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:08:15 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
    01-16 10:08:15 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    .(repeat from 10:08:10)
    .
    .
    01-16 10:08:16 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    01-16 10:08:16 <***C2><SYSTEM><ACT> Stop communication lines
    01-16 10:08:16 <***C2><SYSTEM><ACT> Receiving commands aborted
    01-16 10:08:17 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    01-16 10:08:17 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    01-16 10:08:17 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    01-16 10:08:27 <***C2><SYSTEM><ACT> ScadaCommService is stopped
    --------------------------------------------------------------------------------
    --------------------------------------------------------------------------------
    01-16 10:08:29 <***C2><SYSTEM><ACT> ScadaCommService 5.2.1.1 is started
    01-16 10:08:29 <***C2><SYSTEM><ACT> Load device library: KpEmail.dll
    01-16 10:08:29 <***C2><SYSTEM><ACT> Load device library: KpSnmp.dll
    01-16 10:08:29 <***C2><SYSTEM><ACT> Load device library: KpModbus.dll
    01-16 10:08:29 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:08:35 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
    01-16 10:08:35 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:08:35 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error.
    Delay before trying again
    .(repeat from 10:08:29)
    .
    .
    01-16 10:11:05 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:11:11 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
    01-16 10:11:11 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:11:11 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error.
    Delay before trying again
    01-16 10:11:20 <***C2><SYSTEM><ACT> ScadaCommService is stopped
    --------------------------------------------------------------------------------
    --------------------------------------------------------------------------------
    01-16 10:11:23 <***C2><SYSTEM><ACT> ScadaCommService 5.2.1.1 is started
    01-16 10:11:23 <***C2><SYSTEM><ACT> Load device library: KpEmail.dll
    01-16 10:11:23 <***C2><SYSTEM><ACT> Load device library: KpSnmp.dll
    01-16 10:11:23 <***C2><SYSTEM><ACT> Load device library: KpModbus.dll
    01-16 10:11:23 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:11:28 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
    01-16 10:11:28 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:11:28 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error.
    Delay before trying again
    01-16 10:11:38 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:11:44 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
    01-16 10:11:44 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:11:44 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error.
    Delay before trying again
    01-16 10:11:50 <***C2><SYSTEM><ACT> ScadaCommService is stopped
    --------------------------------------------------------------------------------
    --------------------------------------------------------------------------------
    01-16 10:11:52 <***C2><SYSTEM><ACT> ScadaCommService 5.2.1.1 is started
    01-16 10:11:52 <***C2><SYSTEM><ACT> Load device library: KpEmail.dll
    01-16 10:11:52 <***C2><SYSTEM><ACT> Load device library: KpSnmp.dll
    01-16 10:11:52 <***C2><SYSTEM><ACT> Load device library: KpModbus.dll
    01-16 10:11:52 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:11:58 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
    01-16 10:11:58 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:11:58 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error.
    Delay before trying again
    01-16 10:12:08 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:12:13 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
    01-16 10:12:13 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:12:13 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error.
    Delay before trying again
    01-16 10:12:24 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:12:26 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:12:26 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Thread was being aborted.
    01-16 10:12:26 <***C2><SYSTEM><EXC> Error restoring connection with SCADA-Server: Thread was being aborted.
    01-16 10:12:26 <***C2><SYSTEM><EXC> Error receiving configuration database table from Server: Thread was being aborted.
    01-16 10:12:26 <***C2><SYSTEM><EXC> Error starting operation:
    System.Threading.ThreadAbortException: Thread was being aborted.
       at Scada.Client.ServerComm.ReceiveBaseTable(String tableName, IBaseTable baseTable)
       at Scada.Comm.Engine.Manager.ReceiveConfigBase(ConfigBaseSubset& configBase)
       at Scada.Comm.Engine.Manager.StartThreads()
    01-16 10:12:26 <***C2><SYSTEM><ACT> ScadaCommService is stopped
    --------------------------------------------------------------------------------
    --------------------------------------------------------------------------------
    01-16 10:12:29 <***C2><SYSTEM><ACT> ScadaCommService 5.2.1.1 is started
    01-16 10:12:29 <***C2><SYSTEM><ACT> Load device library: KpEmail.dll
    01-16 10:12:29 <***C2><SYSTEM><ACT> Load device library: KpSnmp.dll
    01-16 10:12:29 <***C2><SYSTEM><ACT> Load device library: KpModbus.dll
    01-16 10:12:29 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:12:35 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
    01-16 10:12:35 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:12:35 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error.
    Delay before trying again
    .(repeat from  10:12:29)
    .
    .
    
    01-16 10:14:32 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:14:33 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000
    01-16 10:14:33 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:14:33 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error.
    Delay before trying again
    

    ============= Restart

    
    --------------------------------------------------------------------------------
    01-16 10:14:35 <***C2><SYSTEM><ACT> ScadaServerService 5.1.4.0 is started
    01-16 10:14:35 <***C2><SYSTEM><ACT> Module is loaded from the file C:\ProgramData\RapidSCADA\581\ScadaServer\Mod\ModAutoControl.dll
    01-16 10:14:35 <***C2><SYSTEM><ACT> Check the existence of the data directories is completed successfully
    01-16 10:14:35 <***C2><SYSTEM><ACT> Check the existence of the configuration database files is completed successfully
    01-16 10:14:35 <***C2><SYSTEM><ACT> Input channels are read from the configuration database. Active channel count: 129
    01-16 10:14:35 <***C2><SYSTEM><ACT> Ouput channels are read from the configuration database
    01-16 10:14:35 <***C2><SYSTEM><ACT> Users are read from the configuration database
    01-16 10:14:35 <***C2><SYSTEM><ACT> Formulas are read from the configuration database
    01-16 10:14:35 <***C2><SYSTEM><ACT> The formulas source code has been compiled
    01-16 10:14:35 <***C2><SYSTEM><ACT> Connection listener is started
    01-16 10:14:35 <***C2><SYSTEM><ACT> Start server
    01-16 10:14:35 <***C2><SYSTEM><ACT> Current data are loaded
    01-16 10:14:37 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0
    01-16 10:14:43 <***C2><SYSTEM><ACT> Connect to client 127.0.0.1
    01-16 10:17:12 <***C2><SYSTEM><EXC> Error receiving and processing data from the client 127.0.0.1: Thread was being aborted.
    01-16 10:17:12 <***C2><SYSTEM><EXC> Error communicating with the client 127.0.0.1: Thread was being aborted.
    01-16 10:17:12 <***C2><SYSTEM><ACT> Connection listener is stopped
    01-16 10:17:22 <***C2><SYSTEM><EXC> Error executing actions on current data calculated in module ModAutoControl: Thread was being aborted.
    01-16 10:17:22 <***C2><SYSTEM><ACT> Server is aborted
    01-16 10:17:22 <***C2><SYSTEM><ACT> ScadaServerService is stopped
    --------------------------------------------------------------------------------
    
    
    --------------------------------------------------------------------------------
    01-16 10:14:35 Start ModAutoControl module
    01-16 10:14:35 Check "Automatic control module" registration:
    Registration key is valid
    01-16 10:14:35 Module state loaded from file
    01-16 10:14:35 Variables in the commands are not used
    01-16 10:14:35 Trigger state changed "F14\F14 AccWtStop": Waiting
    01-16 10:14:35 Trigger "F14\CCuMix AccWt Sync" fires
    01-16 10:14:35 Command is enqueued to be sent at 10:14:35. Out channel 9001
    01-16 10:14:35 Sending trigger commands completed
    01-16 10:14:35 Command is sent. Out channel 9001
    01-16 10:17:23 Module ModAutoControl is stopped
    --------------------------------------------------------------------------------
    
    
    Delay before trying again
    01-16 10:17:20 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:17:21 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000
    01-16 10:17:21 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:17:21 <***C2><SYSTEM><ERR> Unable to start operation due to SCADA-Server communication error.
    Delay before trying again
    

    ============ Restart

    
    --------------------------------------------------------------------------------
    01-16 10:17:23 <***C2><SYSTEM><ACT> ScadaServerService 5.1.4.0 is started
    01-16 10:17:23 <***C2><SYSTEM><ACT> Module is loaded from the file C:\ProgramData\RapidSCADA\581\ScadaServer\Mod\ModAutoControl.dll
    01-16 10:17:23 <***C2><SYSTEM><ACT> Check the existence of the data directories is completed successfully
    01-16 10:17:23 <***C2><SYSTEM><ACT> Check the existence of the configuration database files is completed successfully
    01-16 10:17:23 <***C2><SYSTEM><ACT> Input channels are read from the configuration database. Active channel count: 129
    01-16 10:17:23 <***C2><SYSTEM><ACT> Ouput channels are read from the configuration database
    01-16 10:17:23 <***C2><SYSTEM><ACT> Users are read from the configuration database
    01-16 10:17:23 <***C2><SYSTEM><ACT> Formulas are read from the configuration database
    01-16 10:17:23 <***C2><SYSTEM><ACT> The formulas source code has been compiled
    01-16 10:17:23 <***C2><SYSTEM><ACT> Connection listener is started
    01-16 10:17:23 <***C2><SYSTEM><ACT> Start server
    01-16 10:17:23 <***C2><SYSTEM><ACT> Current data are loaded
    01-16 10:17:23 <***C2><SYSTEM><ACT> Command: out channel = 1405, user ID = 0
    01-16 10:17:23 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0
    01-16 10:17:31 <***C2><SYSTEM><ACT> Connect to client 127.0.0.1
    01-16 10:17:31 <***C2><SYSTEM><ACT> The user ScadaComm is successfully authenticated
    01-16 10:17:33 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 10:17:37 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 10:17:39 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 10:17:42 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 10:17:44 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 10:17:47 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 10:17:49 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 10:17:51 <***C2><SYSTEM><ACT> Connection listener is stopped
    01-16 10:17:51 <***C2><SYSTEM><ACT> Server is stopped
    01-16 10:17:51 <***C2><SYSTEM><ACT> ScadaServerService is stopped
    --------------------------------------------------------------------------------
    
    
    --------------------------------------------------------------------------------
    01-16 10:17:23 Start ModAutoControl module
    01-16 10:17:23 Check "Automatic control module" registration:
    Registration key is valid
    01-16 10:17:23 Module state loaded from file
    01-16 10:17:23 Variables in the commands are not used
    01-16 10:17:23 Trigger state changed "F14\F14 AutoStop": Repeat firing 2/2 (every 1 sec)
    01-16 10:17:23 Command is enqueued to be sent at 10:17:23. Out channel 1405
    01-16 10:17:23 Sending trigger commands completed
    01-16 10:17:23 Command is sent. Out channel 1405
    01-16 10:17:23 Trigger "F14\CCuMix AccWt Sync" fires
    01-16 10:17:23 Command is enqueued to be sent at 10:17:23. Out channel 9001
    01-16 10:17:23 Sending trigger commands completed
    01-16 10:17:23 Command is sent. Out channel 9001
    01-16 10:17:32 Trigger state changed "F14\F14 Stop: PH3": Waiting
    01-16 10:17:32 Trigger state changed "F14\F14 Stop: Flow1": Waiting
    01-16 10:17:32 Trigger state changed "F14\F14 Stop: Temp1": Waiting
    01-16 10:17:33 Trigger state changed "F14\F14 Stop: Temp2": Waiting
    01-16 10:17:33 Trigger "F14\F14 MCnt Sync" fires
    01-16 10:17:33 Command is enqueued to be sent at 10:17:33. Out channel 1402
    01-16 10:17:33 Sending trigger commands completed
    01-16 10:17:33 Command is sent. Out channel 1402
    01-16 10:17:37 Trigger "F14\F14 MCnt Sync" fires
    01-16 10:17:37 Command is enqueued to be sent at 10:17:37. Out channel 1402
    01-16 10:17:37 Sending trigger commands completed
    01-16 10:17:37 Command is sent. Out channel 1402
    01-16 10:17:39 Trigger "F14\F14 MCnt Sync" fires
    01-16 10:17:39 Command is enqueued to be sent at 10:17:39. Out channel 1402
    01-16 10:17:39 Sending trigger commands completed
    01-16 10:17:39 Command is sent. Out channel 1402
    01-16 10:17:42 Trigger "F14\F14 MCnt Sync" fires
    01-16 10:17:42 Command is enqueued to be sent at 10:17:42. Out channel 1402
    01-16 10:17:42 Sending trigger commands completed
    01-16 10:17:42 Command is sent. Out channel 1402
    01-16 10:17:44 Trigger "F14\F14 MCnt Sync" fires
    01-16 10:17:44 Command is enqueued to be sent at 10:17:44. Out channel 1402
    01-16 10:17:44 Sending trigger commands completed
    01-16 10:17:44 Command is sent. Out channel 1402
    01-16 10:17:47 Trigger "F14\F14 MCnt Sync" fires
    01-16 10:17:47 Command is enqueued to be sent at 10:17:47. Out channel 1402
    01-16 10:17:47 Sending trigger commands completed
    01-16 10:17:47 Command is sent. Out channel 1402
    01-16 10:17:49 Trigger "F14\F14 MCnt Sync" fires
    01-16 10:17:49 Command is enqueued to be sent at 10:17:49. Out channel 1402
    01-16 10:17:49 Sending trigger commands completed
    01-16 10:17:49 Command is sent. Out channel 1402
    01-16 10:17:51 Module ModAutoControl is stopped
    --------------------------------------------------------------------------------
    
    
    CommSvc Log
    
    (Delay before trying again)
    

    ==== Restart after ModAutoControl is stopped

    
    --------------------------------------------------------------------------------
    01-16 10:17:54 <***C2><SYSTEM><ACT> ScadaServerService 5.1.4.0 is started
    01-16 10:17:54 <***C2><SYSTEM><ACT> Check the existence of the data directories is completed successfully
    01-16 10:17:54 <***C2><SYSTEM><ACT> Check the existence of the configuration database files is completed successfully
    01-16 10:17:54 <***C2><SYSTEM><ACT> Input channels are read from the configuration database. Active channel count: 129
    01-16 10:17:54 <***C2><SYSTEM><ACT> Ouput channels are read from the configuration database
    01-16 10:17:54 <***C2><SYSTEM><ACT> Users are read from the configuration database
    01-16 10:17:54 <***C2><SYSTEM><ACT> Formulas are read from the configuration database
    01-16 10:17:54 <***C2><SYSTEM><ACT> The formulas source code has been compiled
    01-16 10:17:54 <***C2><SYSTEM><ACT> Connection listener is started
    01-16 10:17:54 <***C2><SYSTEM><ACT> Start server
    01-16 10:17:54 <***C2><SYSTEM><ACT> Current data are loaded
    01-16 10:18:01 <***C2><SYSTEM><ACT> Connect to client 127.0.0.1
    01-16 10:18:01 <***C2><SYSTEM><ACT> The user ScadaComm is successfully authenticated
    01-16 10:18:17 <***C2><SYSTEM><ACT> Connection listener is stopped
    01-16 10:18:17 <***C2><SYSTEM><ACT> Server is stopped
    01-16 10:18:17 <***C2><SYSTEM><ACT> ScadaServerService is stopped
    --------------------------------------------------------------------------------
    
    
    CommSvc Log
    
    01-16 10:17:31 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:17:31 <***C2><SYSTEM><ACT> Start communication lines
    01-16 10:17:31 <***C2><SYSTEM><ACT> Start receiving commands
    01-16 10:17:33 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1
    01-16 10:17:37 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1
    01-16 10:17:39 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1
    01-16 10:17:43 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1
    01-16 10:17:44 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1
    01-16 10:17:48 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1
    01-16 10:17:49 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1
    01-16 10:17:51 <***C2><SYSTEM><EXC> Error requesting telecommand from SCADA-Server: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.
    01-16 10:17:51 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:17:51 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:17:52 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000
    01-16 10:17:52 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:17:52 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    01-16 10:17:53 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    .(repeat)
    01-16 10:18:01 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:18:17 <***C2><SYSTEM><EXC> Error requesting telecommand from SCADA-Server: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.
    01-16 10:18:17 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:18:17 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:18:18 <***C2><SYSTEM><EXC> Error connecting to SCADA-Server: No connection could be made because the target machine actively refused it 127.0.0.1:10000
    01-16 10:18:18 <***C2><SYSTEM><ACT> Disconnect from SCADA-Server
    01-16 10:18:18 <***C2><SYSTEM><ERR> Unable to connect to SCADA-Server. Try again.
    .(repeat)
    

    ==== Restart after ModAutoControl is turned back on. Back to Normal.

    
    --------------------------------------------------------------------------------
    01-16 10:18:19 <***C2><SYSTEM><ACT> ScadaServerService 5.1.4.0 is started
    01-16 10:18:20 <***C2><SYSTEM><ACT> Module is loaded from the file C:\ProgramData\RapidSCADA\581\ScadaServer\Mod\ModAutoControl.dll
    01-16 10:18:20 <***C2><SYSTEM><ACT> Check the existence of the data directories is completed successfully
    01-16 10:18:20 <***C2><SYSTEM><ACT> Check the existence of the configuration database files is completed successfully
    01-16 10:18:20 <***C2><SYSTEM><ACT> Input channels are read from the configuration database. Active channel count: 129
    01-16 10:18:20 <***C2><SYSTEM><ACT> Ouput channels are read from the configuration database
    01-16 10:18:20 <***C2><SYSTEM><ACT> Users are read from the configuration database
    01-16 10:18:20 <***C2><SYSTEM><ACT> Formulas are read from the configuration database
    01-16 10:18:20 <***C2><SYSTEM><ACT> The formulas source code has been compiled
    01-16 10:18:20 <***C2><SYSTEM><ACT> Connection listener is started
    01-16 10:18:20 <***C2><SYSTEM><ACT> Start server
    01-16 10:18:20 <***C2><SYSTEM><ACT> Current data are loaded
    01-16 10:18:20 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0
    01-16 10:18:27 <***C2><SYSTEM><ACT> Connect to client 127.0.0.1
    01-16 10:18:27 <***C2><SYSTEM><ACT> The user ScadaComm is successfully authenticated
    01-16 10:18:31 <***C2><SYSTEM><ACT> Command: out channel = 1405, user ID = 0
    01-16 10:18:32 <***C2><SYSTEM><ACT> Command: out channel = 1405, user ID = 0
    01-16 10:18:34 <***C2><SYSTEM><ACT> Command: out channel = 1402, user ID = 0
    01-16 10:18:34 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0
    01-16 10:18:34 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0
    01-16 10:19:00 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0
    01-16 10:19:04 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0
    01-16 10:19:30 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0
    01-16 10:19:34 <***C2><SYSTEM><ACT> Command: out channel = 9001, user ID = 0
    01-16 10:19:46 <***C2><SYSTEM><ACT> Connect to client 127.0.0.1
    01-16 10:19:46 <***C2><SYSTEM><ACT> The user ScadaWeb is successfully authenticated
    01-16 10:19:47 <***C2><SYSTEM><ACT> Check username and password for admin. Result: passed
    
    
    --------------------------------------------------------------------------------
    01-16 10:18:20 Start ModAutoControl module
    01-16 10:18:20 Check "Automatic control module" registration:
    Registration key is valid
    01-16 10:18:20 Module state loaded from file
    01-16 10:18:20 Variables in the commands are not used
    01-16 10:18:20 Trigger "F14\CCuMix AccWt Sync" fires
    01-16 10:18:20 Command is enqueued to be sent at 10:18:20. Out channel 9001
    01-16 10:18:20 Sending trigger commands completed
    01-16 10:18:20 Command is sent. Out channel 9001
    01-16 10:18:27 Trigger state changed "F14\F14 AutoStop": Waiting
    01-16 10:18:27 No trigger commands to send
    01-16 10:18:29 Trigger state changed "F14\F14 Stop: Flow1": Firing
    01-16 10:18:29 Sending commands skipped
    01-16 10:18:29 Trigger state changed "F14\F14 Stop: Temp1": Waiting
    01-16 10:18:30 Trigger state changed "F14\F14 Stop: Temp2": Firing
    01-16 10:18:30 Sending commands skipped
    01-16 10:18:31 Trigger state changed "F14\F14 ": Firing
    01-16 10:18:31 Command is enqueued to be sent at 10:18:31. Out channel 1405
    01-16 10:18:31 Sending trigger commands completed
    01-16 10:18:31 Command is sent. Out channel 1405
    01-16 10:18:32 Trigger state changed "F14\F14 ": Repeat firing 2/2 (every 1 sec)
    01-16 10:18:32 Command is enqueued to be sent at 10:18:32. Out channel 1405
    01-16 10:18:32 Sending trigger commands completed
    01-16 10:18:32 Command is sent. Out channel 1405
    01-16 10:18:32 Trigger state changed "F14\F14 Stop: PH3": Waiting
    01-16 10:18:34 Trigger "F14\F14 MCnt Sync" fires
    01-16 10:18:34 Command is enqueued to be sent at 10:18:34. Out channel 1402
    01-16 10:18:34 Sending trigger commands completed
    01-16 10:18:34 Command is sent. Out channel 1402
    01-16 10:18:34 Trigger "F14\CCuMix AccWt Sync" fires
    01-16 10:18:34 Command is enqueued to be sent at 10:18:34. Out channel 9001
    01-16 10:18:34 Sending trigger commands completed
    
    
    CommSvc Log
    
    01-16 10:18:27 <***C2><SYSTEM><ACT> Connect to SCADA-Server "localhost"
    01-16 10:18:31 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=4
    01-16 10:18:32 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=4
    01-16 10:18:34 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=114, number=1
    01-16 10:18:34 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=190, number=1
    01-16 10:18:34 <***C2><SYSTEM><ACT> The command is received from SCADA-Server: type=Standard, device=190, number=1
    .
    .
    .
    

    Could you privde a produre to troubleshoot such problem in general ?

    It just happend to me that I might try disableing AutomaticControl first.
    That’s pure luck though.

    • This topic was modified 3 years, 2 months ago by Mikhail.
    #8093
    Mikhail
    Moderator

    Hi,

    How often does the problem occur?
    What is your scenario of using Auto Control Module?

    It just happend to me that I might try disableing AutomaticControl first.

    Agree. This is a possible cause.

    #8094
    Mikhail
    Moderator

    Also provide a screenshot of the trigger settings that sends the last command:
    01-16 05:15:47 <***C2> Command: out channel = 1402, user ID = 0

    #8119
    zzz
    Participant

    > How often does the problem occur?
    Can’t say, sometimes it run fine for days w/o problem. Maybe could continue if not restarted? Sometimes it dies silently hours after restart.

    Yesterday moring, I found it stopped working around 01/19/23:30, about 6 hours after a restart. I can’t find the log anymore, it’s been overwritten. But everytime the server is restart after the silent death, it always give the ThreadAbortException message like shown in the above logs.

    I suspect some of the values in the channels got AutoControl crazy.
    Every time I restart the ServerSrv w/ autocontrol disabled, I have to wait for the communicator to fully work. Or the next restsart with ServerSrv w/ autocontrol enabled, will usually fail. Pity, I just thought about this, can’t find snapshot of those values anymore. The archive is at 30 secs interval.
    Mybe this could be an option for some troubleshoot feature to save a snapshot of current values.

    Could you shad some light, how could autocontrol module block the server thread processing communicator communiations?

    > What is your scenario of using Auto Control Module?
    – Sync data from sensor bus line to PLC bus lines.
    – Unified control for process metrics.
    – Execution of control message from production system.

    AutoControl Rules

    
    <TrGroup active="true" name="F14">
        <Trigger id="9" active="true" name="F14 WtReached" type="CnlDataTrigger">
            <CnlDataTriggerProps>
                <CnlNum>11410</CnlNum>
                <ChkCnlVal>true</ChkCnlVal>
                <CnlValCond>
                    <CompareOperator1>GreaterThan</CompareOperator1>
                    <CompareArgument1>905</CompareArgument1>
                    <LogicalOperator>None</LogicalOperator>
                    <CompareOperator2>LessThan</CompareOperator2>
                    <CompareArgument2>0</CompareArgument2>
                </CnlValCond>
                <Deadband>0</Deadband>
                <ChkCnlStat>false</ChkCnlStat>
                <CnlStat />
                <Delay>0</Delay>
                <Repeat>1</Repeat>
                <RepeatCnt>2</RepeatCnt>
            </CnlDataTriggerProps>
            <Commands>
                <Command cmdTypeID="0">
                    <KPNum>0</KPNum>
                    <CmdNum>4</CmdNum>
                    <CtrlCnlNum>1405</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>false</CopyCmdVal>
                    <CmdVal>1</CmdVal>
                </Command>
            </Commands>
        </Trigger>
        <Trigger id="3" active="true" name="Reset MCounter & MES CtrlBits On PreStart" type="EvTrigger">
            <EvTriggerProps>
                <ChkObjNum>false</ChkObjNum>
                <ObjNum>2</ObjNum>
                <ChkKPNum>false</ChkKPNum>
                <KPNum>114</KPNum>
                <ChkParamID>false</ChkParamID>
                <ParamID>0</ParamID>
                <ChkCnlNum>true</ChkCnlNum>
                <CnlNum>11404</CnlNum>
                <ChkNewCnlVal>true</ChkNewCnlVal>
                <NewCnlValCond>
                    <CompareOperator1>Equal</CompareOperator1>
                    <CompareArgument1>1</CompareArgument1>
                    <LogicalOperator>None</LogicalOperator>
                    <CompareOperator2>LessThan</CompareOperator2>
                    <CompareArgument2>0</CompareArgument2>
                </NewCnlValCond>
                <ChkCnlStat>false</ChkCnlStat>
                <NewCnlStat>0</NewCnlStat>
                <DescrCompareOper>None</DescrCompareOper>
                <Descr />
                <DataCompareOper>None</DataCompareOper>
                <Data>1</Data>
            </EvTriggerProps>
            <Commands>
                <Command cmdTypeID="0">
                    <KPNum>0</KPNum>
                    <CmdNum>1</CmdNum>
                    <CtrlCnlNum>1401</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>false</CopyCmdVal>
                    <CmdVal>1</CmdVal>
                </Command>
                <Command cmdTypeID="0">
                    <KPNum>0</KPNum>
                    <CmdNum>1</CmdNum>
                    <CtrlCnlNum>1402</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>false</CopyCmdVal>
                    <CmdVal>0</CmdVal>
                </Command>
                <Command cmdTypeID="0">
                    <KPNum>0</KPNum>
                    <CmdNum>1</CmdNum>
                    <CtrlCnlNum>1403</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>false</CopyCmdVal>
                    <CmdVal>0</CmdVal>
                </Command>
            </Commands>
        </Trigger>
        <Trigger id="16" active="true" name="F14 Flow" type="CnlDataTrigger">
            <CnlDataTriggerProps>
                <CnlNum>1403</CnlNum>
                <ChkCnlVal>true</ChkCnlVal>
                <CnlValCond>
                    <CompareOperator1>LessThan</CompareOperator1>
                    <CompareArgument1>*00</CompareArgument1>
                    <LogicalOperator>None</LogicalOperator>
                    <CompareOperator2>LessThan</CompareOperator2>
                    <CompareArgument2>0</CompareArgument2>
                </CnlValCond>
                <Deadband>1</Deadband>
                <ChkCnlStat>false</ChkCnlStat>
                <CnlStat />
                <Delay>60</Delay>
                <Repeat>0</Repeat>
                <RepeatCnt>0</RepeatCnt>
            </CnlDataTriggerProps>
            <Commands>
                <Command cmdTypeID="0">
                    <KPNum>0</KPNum>
                    <CmdNum>1</CmdNum>
                    <CtrlCnlNum>1403</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>false</CopyCmdVal>
                    <CmdVal>9</CmdVal>
                </Command>
                <Command cmdTypeID="1">
                    <KPNum>0</KPNum>
                    <CmdNum>1</CmdNum>
                    <CtrlCnlNum>1404</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>false</CopyCmdVal>
                    <EditMode>Str</EditMode>
                    <CmdDataStr>Flow overlimit</CmdDataStr>
                </Command>
            </Commands>
        </Trigger>
        <Trigger id="18" active="true" name="F14 Temp" type="CnlDataTrigger">
            <CnlDataTriggerProps>
                <CnlNum>1407</CnlNum>
                <ChkCnlVal>true</ChkCnlVal>
                <CnlValCond>
                    <CompareOperator1>LessThan</CompareOperator1>
                    <CompareArgument1>*00</CompareArgument1>
                    <LogicalOperator>Or</LogicalOperator>
                    <CompareOperator2>GreaterThan</CompareOperator2>
                    <CompareArgument2>*00</CompareArgument2>
                </CnlValCond>
                <Deadband>1</Deadband>
                <ChkCnlStat>false</ChkCnlStat>
                <CnlStat />
                <Delay>60</Delay>
                <Repeat>0</Repeat>
                <RepeatCnt>0</RepeatCnt>
            </CnlDataTriggerProps>
            <Commands>
                <Command cmdTypeID="0">
                    <KPNum>0</KPNum>
                    <CmdNum>1</CmdNum>
                    <CtrlCnlNum>1403</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>false</CopyCmdVal>
                    <CmdVal>9</CmdVal>
                </Command>
                <Command cmdTypeID="1">
                    <KPNum>0</KPNum>
                    <CmdNum>1</CmdNum>
                    <CtrlCnlNum>1404</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>false</CopyCmdVal>
                    <EditMode>Str</EditMode>
                    <CmdDataStr>Temp over limit</CmdDataStr>
                </Command>
            </Commands>
        </Trigger>
        <Trigger id="19" active="true" name="F14 Temp2" type="CnlDataTrigger">
            <CnlDataTriggerProps>
                <CnlNum>1406</CnlNum>
                <ChkCnlVal>true</ChkCnlVal>
                <CnlValCond>
                    <CompareOperator1>LessThan</CompareOperator1>
                    <CompareArgument1>*0</CompareArgument1>
                    <LogicalOperator>Or</LogicalOperator>
                    <CompareOperator2>GreaterThan</CompareOperator2>
                    <CompareArgument2>*0</CompareArgument2>
                </CnlValCond>
                <Deadband>1</Deadband>
                <ChkCnlStat>false</ChkCnlStat>
                <CnlStat />
                <Delay>60</Delay>
                <Repeat>0</Repeat>
                <RepeatCnt>0</RepeatCnt>
            </CnlDataTriggerProps>
            <Commands>
                <Command cmdTypeID="0">
                    <KPNum>0</KPNum>
                    <CmdNum>1</CmdNum>
                    <CtrlCnlNum>1403</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>false</CopyCmdVal>
                    <CmdVal>9</CmdVal>
                </Command>
                <Command cmdTypeID="1">
                    <KPNum>0</KPNum>
                    <CmdNum>1</CmdNum>
                    <CtrlCnlNum>1404</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>false</CopyCmdVal>
                    <EditMode>Str</EditMode>
                    <CmdDataStr>Temp2 over limit</CmdDataStr>
                </Command>
            </Commands>
        </Trigger>
        <Trigger id="21" active="true" name="F14 PH3" type="CnlDataTrigger">
            <CnlDataTriggerProps>
                <CnlNum>1402</CnlNum>
                <ChkCnlVal>true</ChkCnlVal>
                <CnlValCond>
                    <CompareOperator1>LessThan</CompareOperator1>
                    <CompareArgument1>6</CompareArgument1>
                    <LogicalOperator>None</LogicalOperator>
                    <CompareOperator2>LessThan</CompareOperator2>
                    <CompareArgument2>0</CompareArgument2>
                </CnlValCond>
                <Deadband>0</Deadband>
                <ChkCnlStat>false</ChkCnlStat>
                <CnlStat />
                <Delay>90</Delay>
                <Repeat>0</Repeat>
                <RepeatCnt>0</RepeatCnt>
            </CnlDataTriggerProps>
            <Commands>
                <Command cmdTypeID="0">
                    <KPNum>0</KPNum>
                    <CmdNum>1</CmdNum>
                    <CtrlCnlNum>1403</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>false</CopyCmdVal>
                    <CmdVal>9</CmdVal>
                </Command>
                <Command cmdTypeID="1">
                    <KPNum>0</KPNum>
                    <CmdNum>1</CmdNum>
                    <CtrlCnlNum>1404</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>false</CopyCmdVal>
                    <EditMode>Str</EditMode>
                    <CmdDataStr>PH3 over limit</CmdDataStr>
                </Command>
            </Commands>
        </Trigger>
        <Trigger id="5" active="true" name="F14 MCnt Sync" type="CnlDataChangeTrigger">
            <CnlDataChangeTriggerProps>
                <CnlNum>1410</CnlNum>
            </CnlDataChangeTriggerProps>
            <Commands>
                <Command cmdTypeID="0">
                    <KPNum>0</KPNum>
                    <CmdNum>1</CmdNum>
                    <CtrlCnlNum>1402</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>true</CopyCmdVal>
                    <CmdVal>0</CmdVal>
                </Command>
            </Commands>
        </Trigger>
        <Trigger id="14" active="true" name="CCuMix AccWt Sync" type="CnlDataChangeTrigger">
            <CnlDataChangeTriggerProps>
                <CnlNum>19001</CnlNum>
            </CnlDataChangeTriggerProps>
            <Commands>
                <Command cmdTypeID="0">
                    <KPNum>0</KPNum>
                    <CmdNum>1</CmdNum>
                    <CtrlCnlNum>9001</CtrlCnlNum>
                    <UserID>0</UserID>
                    <Delay>0</Delay>
                    <SendOnFiring>true</SendOnFiring>
                    <SendOnNorm>false</SendOnNorm>
                    <CopyCmdVal>true</CopyCmdVal>
                    <CmdVal>0</CmdVal>
                </Command>
            </Commands>
        </Trigger>
    </TrGroup>
    
    #8124
    JW
    Participant

    I had experienced similar things before when:

    Using Auto Control Module to send commands to one external modbus device.

    The cause I suspect:
    1 trigger send multiple commands;
    or several triggers send commands at the same time;

    It happens quite frequently when doing stress test, e.g. toggling the trigger every a few second.

    as I remember in detail log, it’s similar to the issue I mentioned in another post. (Modbus Comm Bug)
    it looks like scada mis-align the reply from the modbus slave software.
    https://drive.google.com/file/d/11y8e6zyzzKWXJovF0uj8TM22hCG9TJkH/view?usp=sharing

    I am going to find my records, as see if I can make a minimum project that reproduce this problem.

    #8131
    Mikhail
    Moderator

    I was asking for a screenshot, not XML. I can check the settings if you provide it.

    #8135
    zzz
    Participant

    Not that I don’t want to provide, but all file hosting service here require registration with cellphone which is now required by law, almost all foreign free img hosting are blocked.

    #8137
    manjey73
    Participant

    Use a VPN in your browser, which should help you get access to image hosting services

    #8138
    zzz
    Participant

    @MikHail: I have sent you the img via email. Unless someone has some not-yet-blocked less-known image hosting to recommend. I’v nowhere public to upload img.


    @123abc56
    : You may be right about the racing condition. But anyhow, I guess we need that current data table at that moment to inspect the problem. Too bad I didn’t snapshot that file.

    #8141
    Mikhail
    Moderator

    Use a VPN in your browser, which should help you get access to image hosting services

    It may be complicated in China. If somebody has experience with that, please share.

    #8142
    Mikhail
    Moderator

    As a workaround of the hanging problem, I suggest to set delay of the command to 1 instead of 0. In this case command is send using a queue that is more reliable. Let us know about the result.

    #8147
    JW
    Participant

    Hi, I have a few findings.

    1. when the server crashed, there is no log.
    the following part of the error message shows up when manually stop/restart the server.

    System.Threading.ThreadAbortException: Thread was being aborted.
       at System.Threading.Monitor.ReliableEnter(Object obj, Boolean& lockTaken)
       at Scada.Server.Engine.MainLogic.RaiseOnCurDataProcessing(Srez receivedSrez)
       at Scada.Server.Engine.MainLogic.ProcCurData(Srez receivedSrez)
    01-16 10:05:02 <***C2><SYSTEM><EXC> Error receiving and processing data from the client 127.0.0.1: Thread was being aborted.
    01-16 10:05:02 <***C2><SYSTEM><EXC> Error communicating with the client 127.0.0.1: Thread was being aborted.
    01-16 10:05:12 <***C2><SYSTEM><EXC> Error executing actions on current data calculated in module ModAutoControl: Thread was being aborted.
    01-16 10:05:12 <***C2><SYSTEM><ACT> Server is aborted
    01-16 10:05:12 <***C2><SYSTEM><ACT> ScadaServerService is stopped

    2. the server will crash even the device it’s sending command to is not online, so it may not be the communicator problem I mentioned.

    3. Adding 1s delay will reduce the probability / frequency of the server crash, but can not complete solve it.

    #8149
    JW
    Participant

    Hi,

    I made a test project that will trigger this crash issue consistently within 1~2min. It can be downloaded in the link below.

    Test project
    https://drive.google.com/file/d/1l5Z4y8vRrVlked83ZzkT9AtmZpPZ61LI/view?usp=sharing

    Project settings
    The data is generated from simulator driver of scada, with 1s update interval.
    The alarm generated from simulator will trigger commands that send to a modbus relay device. Commands are with 1s delay already.
    It will crash no matter the relay is connected or not. (You can connect a modbus slave/server simulator with 16 DO at 127.0.0.1:502 to see the command in action.)

    Web UI
    https://drive.google.com/file/d/17proisKbU5Gf3xLQ0bRhxrgphJ8h9Sr7/view?usp=sharing
    Administrator and server Log
    https://drive.google.com/file/d/1RcClfPdw3ozifkd5IMGiUuf_xvVgoSeb/view?usp=sharing
    Auto Control settings
    https://drive.google.com/file/d/1S5FPVlr9OMcmyV1tDK0NPWtcWE_2ftz-/view?usp=sharing

    Other notes
    If turn off auto control module, it will not crash on my 1.5 hour test.
    If add 1s more delay after request cycle to simulator, or set the delay of simulator device from 1s to 2s, the crash will happen less frequently. (but I have no license of auto control module on my this computer, I can only test for 10mins per round. I tested 4-5 rounds, only 1 round crashed at about 3 minutes.)

    #8151
    Mikhail
    Moderator

    Hi,

    Thanks a lot. I will try to reproduce the problem using you project and write here.

    Adding 1s delay will reduce the probability / frequency of the server crash, but can not complete solve it.

    I mean adding a delay to a command in the module settings, not to request. If delay is added, commands are added to a queue instead of being sent directly. The queue should protect the server app from hanging.

    but I have no license of auto control module on my this computer

    Get a trial key https://rapidscada.net/trial/TrialKeyGen.aspx

    #8154
    JW
    Participant

    I mean adding a delay to a command in the module settings, not to request. If delay is added, commands are added to a queue instead of being sent directly. The queue should protect the server app from hanging.

    Yes, 1s delay to commands in the auto module are added already in the above test project, which do reduce the crash probability / frequency. But it will still crash on this setting.

    Then I also try to test the effect of different request delay.

    seems they both have effect on this issue.

Viewing 15 posts - 1 through 15 (of 22 total)
  • You must be logged in to reply to this topic.