Several days/channels graphs are generated slooooowly

Forum Home Forums Development and Integration Several days/channels graphs are generated slooooowly

Viewing 15 posts - 1 through 15 (of 22 total)
  • Author
    Posts
  • #1157
    inpelsainpelsa
    Participant

    Hi!

    Generating a 30 day graph for the following amount of channels takes too long:
    1 channel= 45 seconds
    5 channels= 82 seconds
    10 channels=130 seconds

    Graph is drawn on day-steps.

    I monitored resources on the server + client and data processed/transfered is not very big. CPU+RAM+HDD are bored, on both, server and client.

    Using proccess monitor on the server I see each single data is queried at once, not bulk, that can explain it.

    I’m concerned about this because Reports with the ChartPro and ChartList components having several graphs/channels/days are less useful taking so long to be drawn.

    I see data being saved in .DAT files, maybe in binary format but not on a relational database format. Wouldn’t be a good idea to store it on any standard good-perfomance database? Or improve performance/method access to .DAT files…

    Spoiler to new users: To be able to have multi-channel/multi-day/multi-graph on the same webpage you must buy ChartPro and ChartList components (not very expensive through)

    #1158
    inpelsainpelsa
    Participant

    Having a standard database system would allow to fix wrong data in the past.

    If I get one day of wrong data (ie: ulong instead of signed long) data can appear as 4 thousand millions instead of -1 and graphs are useless by the huge automatic scale used.

    Using SQL statements would be easy to fix/delete wrong data on a database.

    thanks!

    #1159
    inpelsainpelsa
    Participant

    In fact… for multichanel graphs the bottleneck “could be” client’s CPU speed:

    Graph/webbrowser runs on a single-core CPU (no complain!) and I have 2 plot-points per minute = 2880 points/channel/data/day… my computer is a Intel core i3-2100 at 3.10 Ghz

    on complex graphs it will not query next day data until CPU finishes drawing each day, so…

    – why it takes so long to draw 2880 points on a 3 Ghz CPU?
    – Could we configure it to 1 plot-point each minute instead of 2?

    #1163
    MikhailMikhail
    Moderator

    Hi,

    Thank you for the useful research.
    The approach of using binary data format has several advantages:
    – It is optimized and can be improved to provide very fast access.
    – Doesn’t depend in 3rd party software => easy to install and backup

    So, I have no plans of moving to RDBMS.

    Or improve performance/method access to .DAT files…

    I agree. This work is permanent. For example, the new web app has modern cache to allow multiple users access data faster.

    30-days 10-chnnels chart contains 24*60*30*10 = 432000 points. It is not a few.
    May be the bottle neck is drawing these points.

    Could you generate similar Excel report and compare the duration?
    It would be interesting to compare with other JavaScript charts such as http://www.highcharts.com in conjunction with MySQL or PostgreSQL (like here).

    If you could continue the research, it would be wonderfull 🙂

    #1164
    MikhailMikhail
    Moderator

    – Could we configure it to 1 plot-point each minute instead of 2?

    Please, explain what do you mean?

    #1176
    inpelsainpelsa
    Participant

    My graphs from 1 month ago have 2 data points every minute, i.e. one at 00:00:00 and another one at 00:00:30

    Seems right now every data point is each minute.

    Which parameter controls this?

    #1178
    inpelsainpelsa
    Participant

    I’m sorry to say that I don’t agree with your decision on keeping with binary data but… OK.

    I’m not sure which part on the client side makes it slow. I will take a look…

    #1181
    MikhailMikhail
    Moderator

    Which parameter controls this?

    SCADA-Server -> Minute data -> writing period

    #1182
    MikhailMikhail
    Moderator

    I’m sorry to say that I don’t agree with your decision on keeping with binary data but… OK.

    May be the source code should be refactored to allow switching the storage.

    #1183
    inpelsainpelsa
    Participant

    > I’m sorry to say that I don’t agree with your decision on keeping with binary data but… OK.
    > May be the source code should be refactored to allow switching the storage.

    If I’m not wrong RapidScada should have relatively easy/simple writes and reads to the data, it should be ‘easy’ to switch to SQL insertions/queryes. And in those days where data should be free/open/customer owned…

    Using SQLite should be ok (check license).
    – If you are worried about using it with huge amounts of data, check PDF files on this work:
    How good can databases deal with Netflow data?
    – It could use just a single data file (or one file pro day) it is straight-forward to do backups
    – And a non-install .DLL file to implement should make it easy to install.

    I will not insist more on this topic 🙂

    =======================================================================

    > 30-days 10-chnnels chart contains 24*60*30*10 = 432000 points.
    > It is not a few.
    432.000 points takes 130 seconds to draw = 3.300 points/second
    3.300 pro second… well… not bad but I hope we can improve it.

    >Could you generate similar Excel report and compare the duration?
    I have a 50.000 points Excel graph, it takes 21 seconds to draw it
    But anyway… they are (and will be) graphs/objects so different that you can not get any reliable conclusion.

    > May be the bottle neck is drawing these points.
    It could be, graph is generated linearly, it doesn’t get slower as more data comes in.

    >It would be interesting to compare with other JavaScript charts such as http://www.highcharts.com in conjunction with MySQL or PostgreSQL (like here).

    Databases will give you that amount of data in a few seconds that’s not a problem. I will give a try to highcharts alone…

    #1207
    inpelsainpelsa
    Participant

    Mmm… HighCharts is more complex/versalite it currently looks.
    I can’t spend a lot of time on this but see what I hae found:

    Seems it can be done with milions of data, but maybe it requires you to provide data to the graph in a different way you currently do:

    Millions of data points on HighCharts

    What do you think?

    #1232
    MikhailMikhail
    Moderator

    Nothing is impossible 🙂

    #1233
    MikhailMikhail
    Moderator

    Thank you for the research. It would be useful to optimize the chart.

    #1241
    inpelsainpelsa
    Participant

    btw…

    > May be the bottle neck is drawing these points.
    After having the whole data, if you zoom and reset zoom, all 30 day points are drawn inmediatelly so it’s not about drawing.

    It could be some math (search for max, min, graph resize).

    I do not expect to have that full featured graph, but a Report with some graphs, some channels and only 4 days of data… takes some time.

    This is a basic performance feature it should already have: Maybe not for the 1 day 1 channel free graph version, but for the multi-day multi-channel PAID version it should.

    Please, take a look to it.

    Thanks,

    #1244
    MikhailMikhail
    Moderator

    The issue is enqueued.

Viewing 15 posts - 1 through 15 (of 22 total)
  • You must be logged in to reply to this topic.