Memory Leak - TANGO Controls

# 8 years ago
Manu	Hi, Valgrind reports many "definitely lost" bytes which could be seen at a leak. BUT I don't think it is. If you read carefully where Valgrind reports about lost bytes, you will see that it is memory allocated in the class init_device() method. If you do not use the device Init commnamd, this code is executed once at device start-up. Reading/Writing attributes or executing commands do not trigger this init_device() method execution an dthis is why I would not really consider this as a real leak. Anyway, what is sure is that some memory freeing code is missing in the delete_device() method of this class. I have fixed this in my local copy of TangoTest and I will soon commit this bug fix. BUT as I explained above, this does not explain a permanent memory increase except if you repeat the Init command again and again. Is it what you are doing? Another interesting point: Do you poll some TangoTest device attribute with a modified polling buffer depth (much bigger than the default depth is 10)?

# 8 years ago

Hi,

Valgrind reports many "definitely lost" bytes which could be seen at a leak. BUT I don't think it is.
If you read carefully where Valgrind reports about lost bytes, you will see that it is memory allocated in the
class init_device() method. If you do not use the device Init commnamd, this code is executed once at device start-up.
Reading/Writing attributes or executing commands do not trigger this init_device() method execution an dthis is why I would
not really consider this as a real leak.

Anyway, what is sure is that some memory freeing code is missing in the delete_device() method of this class. I have fixed this
in my local copy of TangoTest and I will soon commit this bug fix.

BUT as I explained above, this does not explain a permanent memory increase except if you repeat the Init command again and again.
Is it what you are doing?

Another interesting point: Do you poll some TangoTest device attribute with a modified polling buffer depth (much bigger than the default depth is 10)?

# 8 years ago
Manu	Re, Well I have just looked into your second valgrind report you just send. In this one there is a real leak of 2976 bytes in 62 blocks. This leak is known and fixed if you apply the patches provided in this site. Go to the Download/Patches page and get and apply the 4 patches available for Tango 8.1.2 Regards Emmanuel

# 8 years ago
TCS_GMRT	Hello, I have applied patches. Here are the results with cat proc/ pid/status after applying patches Step 1: I ran the Tango Test program. Attaching Proc_0.txt. Step 2: After 10 min. once again I took readings into proc_1.txt proc Vmsize reported no increment. Step 3: Started the client - ATK panel from jive (1 instance) And noted the results in Proc_2.txt I noticed that proc Vmsize increased by around 66000 Kb. Step 4: Allowed to run it for 15 min. more. And noted the results in Proc_3.txt Again I noticed that proc Vmsize increased further by around 45000 Kb. I have also attached valgrind output log file. Can you provide the solution as how do we go ahead about fixing this issue? Thanks & Regards, TGMRT Regards, TCS_GMRT Edited 8 years ago Attachments: 202Kb

# 8 years ago

TCS_GMRT

Hello,
I have applied patches.
Here are the results with cat proc/ pid/status after applying patches

Step 1: I ran the Tango Test program. Attaching Proc_0.txt.

Step 2: After 10 min. once again I took readings into proc_1.txt

proc Vmsize reported no increment.

Step 3: Started the client - ATK panel from jive (1 instance) And noted the results in Proc_2.txt

I noticed that proc Vmsize increased by around 66000 Kb.

Step 4: Allowed to run it for 15 min. more. And noted the results in Proc_3.txt

Again I noticed that proc Vmsize increased further by around 45000 Kb.
I have also attached valgrind output log file.

Can you provide the solution as how do we go ahead about fixing this issue?

Thanks & Regards,
TGMRT

Regards,
TCS_GMRT

Edited 8 years ago

# 8 years ago
TCS_GMRT	Hello We are working with TANGO installed in Fadora 19. I am just asking whether anyone has tried TANGO on linux system instead of Virtual box? Thanks & Regards, TGMRT Regards, TCS_GMRT

# 8 years ago
Manu	Hello, Yes, Tango is used on many Linux hosts (Red Hat, Debian and other). For instance at the ESRF for the accelerators control system, we have more than 160 hosts running Linux! On top of that, you have to add the Linux crates used on our beam-lines… Back to your problem. I am aware of one more possible case which could explain your memory behavior but this one means that your event system is wrongly configured. I see two ways to look into this: 1 - Take Tango 9 (with patch). This case of memory increase due to wrongly configured event system is handled in Tango 9 but note that this does not solve why the event system is wrongly configured but it should solve the memory increase. 2 - Stay with Tango 8 and try to understand what happens in your event system. To do so, run the TangoTest device server with these environment variables: - export ORBtraceLevel=40 - export ORBtraceFile=<a file> Start your client (ATKPanel) and let the client run during 2 mins. Then send us the generated log file Regards Emmanuel

# 8 years ago

Manu

Hello,

Yes, Tango is used on many Linux hosts (Red Hat, Debian and other). For instance at the ESRF for the accelerators control system, we have more than 160 hosts running Linux! On top of that, you have to add the Linux crates used on our beam-lines…

Back to your problem.
I am aware of one more possible case which could explain your memory behavior but this one means that your event system is
wrongly configured. I see two ways to look into this:
1 - Take Tango 9 (with patch). This case of memory increase due to wrongly configured event system is handled in Tango 9 but note
that this does not solve why the event system is wrongly configured but it should solve the memory increase.
2 - Stay with Tango 8 and try to understand what happens in your event system. To do so, run the TangoTest device server with
these environment variables:

- export ORBtraceLevel=40
- export ORBtraceFile=<a file>

Start your client (ATKPanel) and let the client run during 2 mins. Then send us the generated log file

Regards

Emmanuel

# 8 years ago
Andy	Hi, just to add to Emmanuel's comments. TANGO is used on tens of sites around the world most (all?) of them running various flavours of Linux. AFAIK all of the are running TANGO 8 in production. Only the ESRF is running TANGO 9 in production on a large system. The virtual box is used only as a demonstrator for tutorials. I am not sure where your memory leak is coming from but as mentioned by Emmanuel there is an issue with the event system when the events are badly configured. But are you sure you are using events? TangoTest is not configured to send events by default. You can check with atkpanel to see if it uses events. I would strongly recommend you move to TANGO 9.1.0 patched unless you have some constraints which prevent this. If you need PyTango then you can mix TANGO 9 and TANGO 8. TANGO 9 could be your base installation and TANGO 8 only for PyTango. Can you check to make sure you have all the patches applied? Compare your patched version with the patched version you can download here: http://sourceforge.net/projects/tango-cs/files/Previous_Releases/tango-8.1.2c-patched.tar.gz/download If you send me your email and phone number (offline) we can try to contact you directly to understand your problem better and help you more efficiently. You can send it to andy dot gotz at esrf dot fr. Kind regards Andy

# 8 years ago

Andy

Hi,

just to add to Emmanuel's comments. TANGO is used on tens of sites around the world most (all?) of them running various flavours of Linux. AFAIK all of the are running TANGO 8 in production. Only the ESRF is running TANGO 9 in production on a large system. The virtual box is used only as a demonstrator for tutorials. I am not sure where your memory leak is coming from but as mentioned by Emmanuel there is an issue with the event system when the events are badly configured. But are you sure you are using events? TangoTest is not configured to send events by default. You can check with atkpanel to see if it uses events.

I would strongly recommend you move to TANGO 9.1.0 patched unless you have some constraints which prevent this. If you need PyTango then you can mix TANGO 9 and TANGO 8. TANGO 9 could be your base installation and TANGO 8 only for PyTango.

Can you check to make sure you have all the patches applied? Compare your patched version with the patched version you can download here:

http://sourceforge.net/projects/tango-cs/files/Previous_Releases/tango-8.1.2c-patched.tar.gz/download

If you send me your email and phone number (offline) we can try to contact you directly to understand your problem better and help you more efficiently. You can send it to andy dot gotz at esrf dot fr.

Kind regards

Andy

# 8 years ago
Manu	Hello, What are the news using Tango 9? Did you had some time to download and compile it? Do you still notice a memory increase with TangoTest using it? About the log file you sent to Andy, I confirm that it seems to have a problem in your event configuration. If I take the example of the TangoTest device double_scalar attribute, I see in your log file several call to register on attr_conf event on this attribute from ATKPanel (for instance at lines 927588, 927886, 938606, 950101, 961545,….) This is very suspicious. In order to also progress on that point, could you send us: 1 - The TANGO_HOST environment variable used on the server side 2 - The TANGO_HOST environment variable value used on ATKPanel side 3 - The result of the database device command DbGetCSDbServerList. To do, in Jive open DS "Databaseds" until you find its device (should be sys/database/2 for a default installation), right click and select "Test device". In the Commands tab, select the command DbGetCSDbServerList and click "Execute" Regards Emmanuel

# 8 years ago

Manu

Hello,

What are the news using Tango 9? Did you had some time to download and compile it? Do you still notice a memory increase with TangoTest using it?

About the log file you sent to Andy, I confirm that it seems to have a problem in your event configuration.
If I take the example of the TangoTest device double_scalar attribute, I see in your log file several call to register
on attr_conf event on this attribute from ATKPanel (for instance at lines 927588, 927886, 938606, 950101, 961545,….)
This is very suspicious. In order to also progress on that point, could you send us:

1 - The TANGO_HOST environment variable used on the server side
2 - The TANGO_HOST environment variable value used on ATKPanel side
3 - The result of the database device command DbGetCSDbServerList. To do, in Jive open DS "Databaseds" until you find its
device (should be sys/database/2 for a default installation), right click and select "Test device". In the Commands tab, select the command
DbGetCSDbServerList and click "Execute"

Regards

Emmanuel

# 8 years ago
TCS_GMRT	Hi Manu, Manu What are the news using Tango 9? Did you had some time to download and compile it? Do you still notice a memory increase with TangoTest using it? Yes. We downloaded and compiled TANGO 9(with patch). And there is still memory increase for TangoTest DS. Manu 1 - The TANGO_HOST environment variable used on the server side 2 - The TANGO_HOST environment variable value used on ATKPanel side We are running server and client on the same machine so the TANGO_HOST entry is same for both which is: TANGO_HOST = <static IP>:10000 Manu 3 - The result of the database device command DbGetCSDbServerList. To do, in Jive open DS "Databaseds" until you find its device (should be sys/database/2 for a default installation), right click and select "Test device". In the Commands tab, select the command DbGetCSDbServerList and click "Execute" Result of above steps is: —————————————————- Command: sys/database/2/DbGetCSDbServerList Duration: 20 msec Output argument(s) : array length: 1 [0] <host-name>:10000 Note: our host name is combination of 10 alphanumeric characters in lower case. As per your suggestion we also tried the same scenarios with 'TANGO_HOST = myhost:10000' and 'myhost' is defined in '/etc/hosts' to the IP of the system. After making above settings we are still observing memory increase of TangoTest DS. Thanks and Regards, TGMRT Regards, TCS_GMRT

# 8 years ago

TCS_GMRT

Hi Manu,

Manu
What are the news using Tango 9? Did you had some time to download and compile it? Do you still notice a memory increase with TangoTest using it?

Yes. We downloaded and compiled TANGO 9(with patch). And there is still memory increase for TangoTest DS.

Manu
1 - The TANGO_HOST environment variable used on the server side
2 - The TANGO_HOST environment variable value used on ATKPanel side

We are running server and client on the same machine so the TANGO_HOST entry is same for both which is:
TANGO_HOST = <static IP>:10000

Manu
3 - The result of the database device command DbGetCSDbServerList. To do, in Jive open DS "Databaseds" until you find its
device (should be sys/database/2 for a default installation), right click and select "Test device". In the Commands tab, select the command
DbGetCSDbServerList and click "Execute"

Result of above steps is:
—————————————————-
Command: sys/database/2/DbGetCSDbServerList
Duration: 20 msec
Output argument(s) :
array length: 1
[0] <host-name>:10000

Note: our host name is combination of 10 alphanumeric characters in lower case.

As per your suggestion we also tried the same scenarios with 'TANGO_HOST = myhost:10000' and 'myhost' is defined in '/etc/hosts' to the IP of the system.
After making above settings we are still observing memory increase of TangoTest DS.

Thanks and Regards,
TGMRT

Regards,
TCS_GMRT

# 8 years ago
Manu	Hi, Could you try to restart your database server with a TANGO_HOST environment variable set to the SAME value than the one used for ATKPanel and your TangoTest device server? Regards Emmanuel

# 8 years ago
TCS_GMRT	Hi Manu, Thanks for your help. As you suggested we made the changes of host-name. And restarted the Tango Database. Now we can still observe memory increase but its not as rapid as it was before. Same behavior is observed with TANGO 9 installation. Thanks and Regards, TGMRT Regards, TCS_GMRT