Event subscription breaks when target device restarted

# 8 years ago
johfor	Hi, I'm experiencing a strange error that sometimes happens when I have set up a change event subscription to an attribute (in this case, State, which is polled). If the target device's server (i.e. the one being subscribed to) is restarted, sometimes the subscription goes bad. Either I start receiving a lot of error events of the type "Event channel is not responding anymore", even though I also get normal events. Or, sometimes, it seems I stop getting events altogether. It's a bit random; sometimes restarting does not cause this problem. Also, I've only seen it where the subscriber itself is a TANGO device, but I have not done tests to see if this matters. The only way I've found to get rid of the issue is to restart the subscribing device. Unsubscribing and re-subscribing to the event does not have any noticable effect. I have not been able to reliably reproduce this, but it's happning quite frequently in our control system. I'm wondering if it's something anyone else has seen? Could it be related to some event issue in TANGO or ZMQ? I found a bugfix in tango 9 that looks like it might be related: #638 Missing events Versions: CentOS 6 PyTango 8.1.6 Tango 8.1.2 ZMQ 3.2.5

# 8 years ago

Hi,

I'm experiencing a strange error that sometimes happens when I have set up a change event subscription to an attribute (in this case, State, which is polled). If the target device's server (i.e. the one being subscribed to) is restarted, sometimes the subscription goes bad. Either I start receiving a lot of error events of the type "Event channel is not responding anymore", even though I also get normal events. Or, sometimes, it seems I stop getting events altogether. It's a bit random; sometimes restarting does not cause this problem. Also, I've only seen it where the subscriber itself is a TANGO device, but I have not done tests to see if this matters.

The only way I've found to get rid of the issue is to restart the subscribing device. Unsubscribing and re-subscribing to the event does not have any noticable effect.

I have not been able to reliably reproduce this, but it's happning quite frequently in our control system. I'm wondering if it's something anyone else has seen? Could it be related to some event issue in TANGO or ZMQ?

I found a bugfix in tango 9 that looks like it might be related:
#638 Missing events

Versions:
CentOS 6
PyTango 8.1.6
Tango 8.1.2
ZMQ 3.2.5

# 8 years ago
Andy	Hi Johfor, quick questions : (1) do you have the all the patches for TANGO 8 installed? Some of these fix a similar problem. (2) have you tried it with PyTango9 (pre-release) ? Andy

# 8 years ago
johfor	Hi Andy, 1. We're using TANGO 8.1.2 with patches 1-4. 2. No. I'll try that next. /Johan

# 8 years ago
Neilen	Hi, johfor Hi, The only way I've found to get rid of the issue is to restart the subscribing device. Unsubscribing and re-subscribing to the event does not have any noticable effect. Not sure if it is applicable, but are you catching and handling all exceptions in your client event callback? It seemed to me that having an unhandled exception in an event handler will prevent future events from being received. Cheers Neilen

# 8 years ago

Neilen

Hi,

johfor
Hi,

The only way I've found to get rid of the issue is to restart the subscribing device. Unsubscribing and re-subscribing to the event does not have any noticable effect.

Not sure if it is applicable, but are you catching and handling all exceptions in your client event callback? It seemed to me that having an unhandled exception in an event handler will prevent future events from being received.

Cheers
Neilen