How to Fandango
Intended audience: developers, Programming language: python
by Sergi Rubio
fun4tango, functional programming for Tango
Warning
Fandango package is a refactored version of the old PyTango_utils package, which is now deprecated
Note
Fandango is now on github
Several recipes available at github
Description
Fandango (previously called PyTango_utils) is a Python module created to simplify the configuration of big control systems; implementing the behavior of Jive (configuration) and/or Astor (deployment) tools in methods that could be called from scripts using regexp and wildcards.
It has been later extended with methods commonly used in some of our python API’s (archiving, CCDB, alarms, vacca) or generic devices (composers, simulators, facades).
Downloading
Fandango module is now on github:
1$ git clone https://github.com/tango-controls/fandango
For Tango 9 you can still get sources from sourceforge:
1$ svn co https://tango-cs.svn.sourceforge.net/svnroot/tango-cs/share/fandango/trunk/fandango fandango
Features
Most of submodules provide some usage recipes, see github:
This library provides submodules with utilities for PyTango device servers and applications written in python:
functional: functional programming, data format conversions, caseless regular expressions,
tango: tango api helper methods, search/modify using regular expressions,
dynamic attributes/states/commands and online python code evaluation, see the github,
server: Astor-like python API,
device: some templates for Tango device servers, TangoEval for fast “tango code” evaluation,
interface: device server inheritance,
db: MySQL access,
dicts,arrays: advanced containers, sorted/caseless list/dictionaries, .csv parsing,
log: logging,
objects: object templates, singletones, structs,
threads: serialized hardware access, multiprocessing,
linos: some linux tricks,
web: html parsing,
qt: some custom Qt classes, including worker-like threads.
Main Classes
DynamicDS / DynamicAttributes,
ServersDict,
TangoEval,
ThreadDict / SingletoneWorker,
TangoInterfaces (FullTangoInheritance).
Where it us used
Several PyTango APIs and device servers use Fandango modules:
SplitterBoxDS,
PyStateComposer,
SimulatorDS / PySignalSimulator,
PANIC / PyAlarm,
CSVReader.
Requirements
It requires PyTango to access Tango,
It requires Taurus to use Tango Events,
Some submodules have its own dependencies (Qt,MySQL), so they are always imported within try, except clauses.
Recipes
Get devices or attributes matching a regular expression
Using fandango.tango.get_matching_devices or get_matching_attributes:
1from fandango import tango
2tango.get_matching_devices('sr[0-9]+/vc/(ipct|vgct)*')
3 ['SR01/VC/IPCT-01A08-01',
4 'SR01/VC/IPCT-01A08-02',
5 'SR01/VC/IPCT-02A01-01',
6 'SR01/VC/VGCT-01A08-01',
7 'SR01/VC/VGCT-02A01-01',
8 'SR02/VC/IPCT-02A02-01',
9 ]
Search for device attribute/properties matching a regular expression
1fandango.tango.get_matching_device_properties('s01/*/*ct*','serial*')
2{'S01/VC/IPCT-01': {'SerialLine': 'S01/VC/SERIAL-01'},
3 'S01/VC/IPCT-02': {'SerialLine': 'S01/VC/SERIAL-02'},
4 'S01/VC/VGCT-01': {'SerialLine': 'S01/VC/SERIAL-10'}}
Obtain all information from a device
In [59]: fandango.tango.get_device_info('sr/vc/gll')
Out[59]: fandango.Struct({
'name': sr/vc/gll,
'level': 4,
'started': 11th February 2013 at 13:07:37,
'PID': 11024,
'ior': ...,
'server': PyStateComposer/SR_VC,
'host': nanana01,
'stopped': 11th February 2013 at 12:49:49,
'exported': 1, })
servers.ServersDict: the Astor-like python API
fandango.ServersDict is a dictionary of TServer classes indexed by server/instance names and loaded using wildcard expressions.
Provides Jive/Astor functionality to a list of servers and allows to select/start/stop them by host, class or devices It’s purpose is to allow generic start/stop of lists of Tango DeviceServers. This methods of selection provide new ways of search apart of Jive-like selection.
1from fandango import Astor
2astor = Astor()
3astor.load_by_name('snap*')
4astor.keys()
5 ['snapmanager/1', 'snaparchiver/1', 'snapextractor/1']
6
7server = astor['snaparchiver/1']
8server.get_device_list()
9 ['dserver/snaparchiver/1', 'archiving/snaparchiver/1']
10
11astor.states()
12server.get_all_states()
13 dserver/snaparchiver/1: ON
14 archiving/snaparchiver/1: ON
15
16astor.get_device_host('archiving/snaparchiver/1')
17 palantir01
18
19astor.stop_servers('snaparchiver/1')
20astor.stop_all_servers()
21astor.start_servers('snaparchiver/1','palantir01',wait=1000)
22astor.set_server_level('snaparchiver/1','palantir01',4)
23
24#Setting the polling of a device:
25server = astor['PySignalSimulator/bl11']
26for dev_name in server.get_device_list():
27 dev = server.get_device(dev_name)
28 attrs = dev.get_attribute_list()
29 [dev.poll_attribute(attr,3000) for attr in attrs]
Start/Stop all device servers in a machine (like Astor -> Stop All)
Stopping
1import fandango
2fandango.Astor(hosts=['my.host']).stop_all_servers()
and the other way round …
1astor = fandango.Astor(hosts=['my.host'])
2astor.start_all_servers()
if you just want to see if things are effectively running or not:
1astor.states()
Implement full (attibutes+properties) inheritance between PyTango classes
Just inheriting from a Device Server does not automatically updates all properties and attributes from the parent. The fandango.interface module enables that functionality using FullTangoInheritance function.
To use it you have to add 3 lines in the “__main__” part of your python file (and at the end of the file, if you want to further continue inheriting between classes):
1#Replace <YourDevice> and <ParentDevice> with your Device classes names
2
3if __name__ == '__main__':
4 try:
5 py = PyTango.Util(sys.argv)
6
7 # Adding DeviceServer Inheritance, added here to be not overwritten by Pogo
8 from fandango.interface import FullTangoInheritance
9 from <ParentDevice> import <ParentDevice>,<ParentDevice>Class
10 <YourDevice>,<YourDevice>Class = \
11 FullTangoInheritance('<YourDevice>',<YourDevice>,<YourDevice>Class, \
12 <ParentDevice>,<ParentDevice>Class,ForceDevImpl=True)
13
14 py.add_TgClass(<YourDevice>Class,<YourDevice>,'<YourDevice>')
15 U = PyTango.Util.instance()
16 U.server_init()
17 U.server_run()
18
19 except PyTango.DevFailed,e:
20 print '-------> Received a DevFailed exception:',e
21 except Exception,e:
22 print '-------> An unforeseen exception occured....',e
23
24# Adding DeviceServer Inheritance (to be visible by subclasses)
25from fandango.interface import FullTangoInheritance
26from <ParentDevice> import <ParentDevice>,<ParentDevice>Class
27<YourDevice>,<YourDevice>Class = FullTangoInheritance('<YourDevice>', <YourDevice>, <YourDevice>Class, <ParentDevice>, <ParentDevice>Class, ForceDevImpl=True)
dynamic.DynamicDS: template for Dynamic Attributes
DynamicAttributes are using DynamicDS template.
Use TangoEval to evaluate strings containing Tango Attributes
TangoEval class provides PyAlarm-like evaluation of strings containing attribute names (replacing them by its values). It is part of fandango.device module. The result of each evaluation is stored in te.result.
In [14]: from fandango import TangoEval
In [15]: te = TangoEval('(s01/vc/gauge-01/pressure + s01/vc/gauge-01/pressure) / 2.')
Out[15]: TangoEval: result = 7.2e-10
Use CSVArray to turn a .csv into a dictionary
1cat tmp/tree_test.csv
2A B 2
3 C 3
In [16]: csv = fandango.arrays.CSVArray('tmp/tree_test.csv')
In [17]: csv.expandAll()
In [18]: csv.getAsTree(lastbranch=1)
Out[18]: {'A': {'B': ['2'], 'C': ['3']}}
Fast property update
1import fandango.functional as fun
2servers = fandango.Astor('PyAlarm/*')
38 : devs = [d for d in fun.chain(*[servers[s].get_device_list() for s,v in servers.states().items() if v is not None]) if not d.startswith('dserver')]
4for d in devs:
5 prop = servers.proxies[d].get_property(['AlarmReceivers'])['AlarmReceivers']
6 servers.proxies[d].put_property({'AlarmReceivers':[s.replace('%SRUBIO','%DFERNANDEZ') for s in prop]})
7for d in devs: servers.proxies[d].ReloadFromDB()
ReversibleDict
In [133]: ch = fandango.dicts.ReversibleDict()
In [134]: ch.update([(unichr(ord('a')+i),i,unichr(ord('A')+i)) for i in range(26)])
In [135]: ch
Out[135]:
(u'a', 0, u'A')
(u'b', 1, u'B')
(u'c', 2, u'C')
(u'd', 3, u'D')
...
In [136]: ch['a']
Out[136]: (0, u'A')
In [137]: ch['A']
Out[137]: (0, u'a')
In [138]: ch['a'].keys()
Out[138]: set([0])
In [139]: ch['A'].keys()
ThreadDict
from PyPLC
1def initThreadDict(self):
2 def read_method(args,comm=self.Regs,log=self.debug): #It takes a key with commas and splits it to have a list of arguments
3 try:
4 log('>'*20 + ' In ThreadDict.read_method(%s)' % args)
5 args = [int(s) for s in args.split(',')[:2]]
6 return comm(args,asynch=True)
7 except PyTango.DevFailed,e:
8 print 'Exception in ThreadDict.read_method!!!'
9 print str(e).replace('\n','')[:100]
10 except Exception,e:
11 print '#'*80
12 print 'Exception in ThreadDict.read_method!!!'
13 print traceback.format_exc()
14 print '#'*80
15 return [] ## Arrays must not be readable if communication doesn't work!!!!
16
17 self.threadDict = fandango.ThreadDict(
18 read_method = read_method,
19 trace=True)
20 self.threadDict.set_timewait(max(0.1,self.ModbusTimeWait/1000.))
21
22 self.info('Mapped Arrays are: %s' % self.MapDict)
23
24 for var,maps in self.MapDict.items():
25 regs = self.GetCommands4Map(maps)
26 for reg in regs:
27 vals = ','.join(str(r) for r in reg)
28 self.debug('Adding %s(%s) as ThreadDict[%s]' % (var,reg,vals))
29 self.threadDict.append(vals,[])#period=[]) #append(key,value='',period=3000)
30
31 self.threadDict.start()
32 self.info('out of PyPLC.initThreadDict()')
33
34Pa leeer .........
35
36 for reg in regs:
37 key = ','.join(str(r) for r in reg)
38 val = self.threadDict[key]
Piped, iPiped, zPiped interfaces
Fandango will have now a new set of operators to use regular-or operator (‘|’) like a linux pipe between operators (inspired by Maxim Krikun - Shell-like data processing ).**
1cat('filename') | grep('myname') | printlines
Using fandango:
1from fandango.functional import *
2
3v | iPiped(rd.get_attribute_values,start_date='2012-07-10',stop_date='2012-07-17') | iPiped(PyTangoArchiving.utils.decimate) | zPiped(time2str) | plist
4
5#equals to:
6
7[(time2str(v[0]),v[1]) for v in PyTangoArchiving.utils.decimate(rd.get_Attribute_values(v,start_date='2012-07-10',stop_date='2012-07-17'))]
Available interfaces are:
1class Piped:
2 """This class gives a "Pipeable" interface to a python method:
3 cat | Piped(method,args) | Piped(list)
4 list(method(args,cat))
5 """
6 ...
7
8class iPiped:
9 """ Used to pipe methods that already return iterators
10 e.g.: hdb.keys() | iPiped(filter,partial(fandango.inCl,'elotech')) | plist
11 """
12 ...
13
14class zPiped:
15 """
16 Returns a callable that applies elements of a list of tuples to a set of functions
17 e.g. [(1,2),(3,0)] | zPiped(str,bool) | plist => [('1',True),('3',False)]
18 """
19 ...
Available operators are:
1pgrep = lambda exp: iPiped(lambda input: (x for x in input if inCl(exp,x)))
2pmatch = lambda exp: iPiped(lambda input: (x for x in input if matchCl(exp,str(x))))
3pfilter = lambda meth=bool,*args: iPiped(filter,partial(meth,*args))
4ppass = Piped(lambda x:x)
5plist = iPiped(list)
6psorted = iPiped(sorted)
7pdict = iPiped(dict)
8ptuple = iPiped(tuple)
9pindex = lambda i: Piped(lambda x:x[i])
10pslice = lambda i,j: Piped(lambda x:x[i,j])
11penum = iPiped(lambda input: izip(count(),input) )
12pzip = iPiped(lambda i:izip(*i))
13ptext = iPiped(lambda input: '\n'.join(imap(str,input)))