Forum Home › Forums › Development and Integration › Several days/channels graphs are generated slooooowly
- This topic has 21 replies, 2 voices, and was last updated 7 years, 10 months ago by Mikhail.
-
AuthorPosts
-
May 13, 2016 at 4:14 pm #1157inpelsaParticipant
Hi!
Generating a 30 day graph for the following amount of channels takes too long:
1 channel= 45 seconds
5 channels= 82 seconds
10 channels=130 secondsGraph is drawn on day-steps.
I monitored resources on the server + client and data processed/transfered is not very big. CPU+RAM+HDD are bored, on both, server and client.
Using proccess monitor on the server I see each single data is queried at once, not bulk, that can explain it.
I’m concerned about this because Reports with the ChartPro and ChartList components having several graphs/channels/days are less useful taking so long to be drawn.
I see data being saved in .DAT files, maybe in binary format but not on a relational database format. Wouldn’t be a good idea to store it on any standard good-perfomance database? Or improve performance/method access to .DAT files…
Spoiler to new users: To be able to have multi-channel/multi-day/multi-graph on the same webpage you must buy ChartPro and ChartList components (not very expensive through)
May 13, 2016 at 4:30 pm #1158inpelsaParticipantHaving a standard database system would allow to fix wrong data in the past.
If I get one day of wrong data (ie: ulong instead of signed long) data can appear as 4 thousand millions instead of -1 and graphs are useless by the huge automatic scale used.
Using SQL statements would be easy to fix/delete wrong data on a database.
thanks!
May 13, 2016 at 4:48 pm #1159inpelsaParticipantIn fact… for multichanel graphs the bottleneck “could be” client’s CPU speed:
Graph/webbrowser runs on a single-core CPU (no complain!) and I have 2 plot-points per minute = 2880 points/channel/data/day… my computer is a Intel core i3-2100 at 3.10 Ghz
on complex graphs it will not query next day data until CPU finishes drawing each day, so…
– why it takes so long to draw 2880 points on a 3 Ghz CPU?
– Could we configure it to 1 plot-point each minute instead of 2?May 14, 2016 at 8:17 am #1163MikhailModeratorHi,
Thank you for the useful research.
The approach of using binary data format has several advantages:
– It is optimized and can be improved to provide very fast access.
– Doesn’t depend in 3rd party software => easy to install and backupSo, I have no plans of moving to RDBMS.
Or improve performance/method access to .DAT files…
I agree. This work is permanent. For example, the new web app has modern cache to allow multiple users access data faster.
30-days 10-chnnels chart contains 24*60*30*10 = 432000 points. It is not a few.
May be the bottle neck is drawing these points.Could you generate similar Excel report and compare the duration?
It would be interesting to compare with other JavaScript charts such as http://www.highcharts.com in conjunction with MySQL or PostgreSQL (like here).If you could continue the research, it would be wonderfull 🙂
May 14, 2016 at 8:21 am #1164MikhailModerator– Could we configure it to 1 plot-point each minute instead of 2?
Please, explain what do you mean?
May 15, 2016 at 6:34 pm #1176inpelsaParticipantMy graphs from 1 month ago have 2 data points every minute, i.e. one at 00:00:00 and another one at 00:00:30
Seems right now every data point is each minute.
Which parameter controls this?
May 15, 2016 at 6:45 pm #1178inpelsaParticipantI’m sorry to say that I don’t agree with your decision on keeping with binary data but… OK.
I’m not sure which part on the client side makes it slow. I will take a look…
May 16, 2016 at 6:22 am #1181MikhailModeratorWhich parameter controls this?
SCADA-Server -> Minute data -> writing period
May 16, 2016 at 6:26 am #1182MikhailModeratorI’m sorry to say that I don’t agree with your decision on keeping with binary data but… OK.
May be the source code should be refactored to allow switching the storage.
May 16, 2016 at 11:39 am #1183inpelsaParticipant> I’m sorry to say that I don’t agree with your decision on keeping with binary data but… OK.
> May be the source code should be refactored to allow switching the storage.If I’m not wrong RapidScada should have relatively easy/simple writes and reads to the data, it should be ‘easy’ to switch to SQL insertions/queryes. And in those days where data should be free/open/customer owned…
Using SQLite should be ok (check license).
– If you are worried about using it with huge amounts of data, check PDF files on this work:
How good can databases deal with Netflow data?
– It could use just a single data file (or one file pro day) it is straight-forward to do backups
– And a non-install .DLL file to implement should make it easy to install.I will not insist more on this topic 🙂
=======================================================================
> 30-days 10-chnnels chart contains 24*60*30*10 = 432000 points.
> It is not a few.
432.000 points takes 130 seconds to draw = 3.300 points/second
3.300 pro second… well… not bad but I hope we can improve it.>Could you generate similar Excel report and compare the duration?
I have a 50.000 points Excel graph, it takes 21 seconds to draw it
But anyway… they are (and will be) graphs/objects so different that you can not get any reliable conclusion.> May be the bottle neck is drawing these points.
It could be, graph is generated linearly, it doesn’t get slower as more data comes in.>It would be interesting to compare with other JavaScript charts such as http://www.highcharts.com in conjunction with MySQL or PostgreSQL (like here).
Databases will give you that amount of data in a few seconds that’s not a problem. I will give a try to highcharts alone…
May 19, 2016 at 3:34 pm #1207inpelsaParticipantMmm… HighCharts is more complex/versalite it currently looks.
I can’t spend a lot of time on this but see what I hae found:Seems it can be done with milions of data, but maybe it requires you to provide data to the graph in a different way you currently do:
Millions of data points on HighCharts
What do you think?
May 24, 2016 at 5:57 am #1232MikhailModeratorNothing is impossible 🙂
May 24, 2016 at 6:01 am #1233MikhailModeratorThank you for the research. It would be useful to optimize the chart.
May 24, 2016 at 11:35 am #1241inpelsaParticipantbtw…
> May be the bottle neck is drawing these points.
After having the whole data, if you zoom and reset zoom, all 30 day points are drawn inmediatelly so it’s not about drawing.It could be some math (search for max, min, graph resize).
I do not expect to have that full featured graph, but a Report with some graphs, some channels and only 4 days of data… takes some time.
This is a basic performance feature it should already have: Maybe not for the 1 day 1 channel free graph version, but for the multi-day multi-channel PAID version it should.
Please, take a look to it.
Thanks,
May 24, 2016 at 1:17 pm #1244MikhailModeratorThe issue is enqueued.
-
AuthorPosts
- You must be logged in to reply to this topic.