Best practices for automation

Hej,

I am new to Tango but have some experience on other control systems in semiconductor automation. I am building a sort of plasma torch. I have played around with tango device servers and have been able to do what I would define as the low level IO for my machine, and in looking at Sardana, have been able to define macros to execute command series; you know, basically almost everything one would want out of a control system! :) however, one thing I have not figured out is what is the intended Tango way for enforcing automation. Let me explain what I mean:

Let’s take the case of my plasma torch. I have a low level safety plc designed to prevent certain personnel safety issues, like “hey, when hydrogen gas flowing, can’t open the oxygen valve” along with the codicil “hey, if the oxygen valve shows open, shut the hydrogen flow immediately”. These are both good things because quartz is frangible, and it makes sense that I implement these as fast safety plc hardware. But there are other things that I might want to do that are either beyond the scope of a plc or are not strictly safety related. A good example might be “hey, if reflected power in the rf source goes above $parameter, the plasma is probably out, try to relight automatically, and if after $parameter2 seconds the plasma isn’t relit, shut down the rf generator”.

For this case, what is the intended Tango way for something to sit and perform this work automatically?

The way I have approached this presently is to write a tango client Interlock-automation where it does deviceproxy to each specific device that needs to be pulled into the interlock and automation routine, then goes into an infinite loop with a delay that checks each of the interlock conditions every $parameter3 seconds and if a condition is met, does the correct behaviour. That said, this seems…hacky? I can’t call Sardana macros from this context so in the case of the “relight the plasma” I can’t just use my existing macro to do so and end up reimplementing the macro in native tango instead.

Further, because I am just naively evaluating based on time instead of on bit changes or events, it’s really inefficient in terms of overhead on tango. I know how I can refactor this to use events but it would still be cumbersome.

Finally, what really stinks about this approach is that my interlock behavior is now intrinsically hardcoded to devices that may or may not exist, and isn’t very hardened to stupid. For example, in this way, if the generator device server breaks, my interlock client will puke, which means one of the other interlocks it is checking is no longer being serviced.

The other way I considered to do this was to write the interlock behavior at the device server level but that seems backwards to have the device server trying to evaluate directly other device servers, even though it avoids some of the above issues.

Any thoughts on this?

Cheers,
Mark
Hi Mark,

Welcome to the Tango world! A sort of plasma torch? That sounds interesting!
In the Tango world, it is pretty common to have a device server being a client of other Tango devices and doing some evaluations, computations depending on the states/data coming from other Tango devices.
If I were you, I would use a dedicated Tango device to do this evaluation job and to trigger the appropriate commands/macros depending on the result of the evaluation.

The infinite loop you were describing could be implemented using the Tango polling of a dedicated attribute or command in this new Tango device (e.g.: a command Evaluate could be executed every x seconds). It could also be done in a dedicated thread.
This Tango device could use the Tango events too to be notified only when something relevant changes in the other devices.

After the evaluation, this new Tango device could send a command to a Tango macro server device to execute a specific macro or spawn a thread and execute the macro sequence in this thread.

This new Tango device could even be exported by the same device server (same process) as your other Tango devices if you wish to (a Tango device server can export several devices from several Tango classes).

I personally prefer to implement this kind of behaviour in a Tango device server than in a client because we have tools to monitor the device servers and to ensure they are correctly running. The Starter device server can also be configured to automatically restart a crashed device server under certain conditions (in our control system for instance device servers are automatically restarted if they crash and if they were running since more than 2 hours).

The inconvenient to implement the automation routine in a client is you need to ensure the uniqueness of this client. With a device server, it is easy, you cannot run twice the same device server instance. With a client, you could be into trouble if they are several of them running at the same time.
Where I work, we try as much as possible to put the logic at the device server level and to have clients as simple as possible.
Another advantage of using a Tango device to do this evaluation work is that you could be notified if there is something wrong with the underlying devices. This new device could have its state reflecting the state of the other sub-devices (ON = everything OK, ALARM = at least one sub-device is in ALARM, UNKNOWN=at least one sub-device is not responding, etc…). You can implement your own logic here.
This new device can then be used with a TANGO alarm system to trigger some alarms if its state is not ON for instance and you could be notified if there is something wrong with this evaluation process.

Hoping this helps a bit,
Kind regards,
Reynald
Rosenberg's Law: Software is easy to make, except when you want it to do something new.
Corollary: The only software that's worth making is software that does something new.
Hi Mark,

as Reynald says your applications sounds very interesting! I agree with Reynald on all the points he made concerning preferring implementing the automation loop in a device server rather than the client. The automation device server is also a client of other device servers so the main difference is that it is encapsulated in a device server, runs only once, can be monitored and managed like other device servers.

An example of how at the ESRF we have used this approach is the procedure for topping up the e-beam every few minutes. The sequences to top up the e-beam are written in Python and executed via the Sardana Macro device server.

You mention that the client is heavy or exposed to errors from devices. This should not be an issue. If you use Tango events to receive updates from your lower level devices then you are informed immediately when something changes and your automation code can take action. If one or more devices are in error you need to take this into account in your code to either ignore it and/or protect the other devices. Another way of reading multiple devices in parallel is to use asynchronous calls. These are fired off to the device server(s) while the client continues immediately. Once all devices have been contacted the client code can synchronise with the answers and set a timeout to not wait longer than a fixed time. For example you could asynchronously read all devices and then set a timeout of N milliseconds before continuing. The devices that have not answered yet are marked as not responding and need to be treated accordingly in your automation algorithm.

Which programming language are you planning on writing your code in?

There is an example of doing of using asynchronous commands in PyTango on this page:

https://tango-controls.readthedocs.io/en/latest/tutorials-and-howtos/how-tos/how-to-pytango.html

You can see an example of a python device server with a worker thread here :

https://github.com/andygotz/nap

Thanks to Tiago for the recipe!

Cheers

Andy

Meh, it's just an ICP source with some novelty on the process and robot side. I've worked on ion implanters in a past life and wanted to start with a control system that can do some of the things that I'm used to.

Thank you for the advice! I've ported my client code over to a device server and used the idea of attributes to set the device names which is a really clever way to do it. I'm also going to try to move everything over to events instead of just polling heavily like I am now. In reality my device load is so low that it probably doesn't matter, but might as well do things right, eh?

Everything is in python as I want to be able to bring persons onto the project as it grows and to be honest, the majority of the work will be process work, thus non computer scientists/software engineers working on it. All the physicists I know prefer python, so python it is.

One thing that I think that I didn't really grasp in Tango that I'm now starting to: a device server isn't just a controls endpoint like it is on an ethercat bus, or IOC like it is in EPICS, it's really any distributed controller that has the option of going to lower level IO. Neat to know!

Thanks guys,
Mark
 
Register or login to create to post a reply.