PyTangoArchiving Recipes
Intended audience: developers, Programming language: python
by Sergi Rubio
PyTangoArchiving is the python API for Tango Archiving.
This package allows to:
Integrate Hdb and Snap archiving with other python/PyTango tools.
Start/Stop Archiving devices in the appropiated order.
Increase the capabilities of configuration and diagnostic.
Import/Export .csv and .xml files between the archiving and the database.
Don’t edit this wiki directly, the source for this documentation is available at PyTangoArchiving UserGuide
Installing PyTangoArchiving:
Repository is available on sourceforge:
1$ svn co https://svn.code.sf.net/p/tango-cs/code/archiving/tool/PyTangoArchiving/trunk
Dependencies:
Tango Java Archiving, ArchivingRoot from sourceforge,
Taurus (optional)
fandango:
1$ svn co https://svn.code.sf.net/p/tango-cs/code/share/fandango/trunk/fandango fandango
Setup:
Follow Tango Java Archiving installation document to setup Java Archivers and Extractors.
Some of the most common installation issues are solved in several topics in Tango forums (search for Tdb/Hdb/Snap Archivers):
Install PyTango and MySQL-python using their own setup.py scripts.
fandango, and PyTangoArchiving parent folders must be added to your PYTHONPATH environment variable.
Although Java Extractors may be used, it is recommended to configure direct MySQL access for PyTangoArchiving
Accessing MySQL:
Although not needed, I recommend you to create a new MySQL user for data querying:
1$ mysql -u hdbmanager -p hdb
2
3$ GRANT USAGE ON hdb.* TO 'user'@'localhost' IDENTIFIED BY '**********';
4$ GRANT USAGE ON hdb.* TO 'user'@'%' IDENTIFIED BY '**********';
5$ GRANT SELECT ON hdb.* TO 'user'@'localhost';
6$ GRANT SELECT ON hdb.* TO 'user'@'%';
7
8$ mysql -u tdbmanager -p tdb
9
10$ GRANT USAGE ON tdb.* TO 'user'@'localhost' IDENTIFIED BY '**********';
11$ GRANT USAGE ON tdb.* TO 'user'@'%' IDENTIFIED BY '**********';
12$ GRANT SELECT ON tdb.* TO 'user'@'localhost';
13$ GRANT SELECT ON tdb.* TO 'user'@'%';
Check in a python shell that your able to access the database:
1import PyTangoArchiving
2
3PyTangoArchiving.Reader(db='hdb',config='user:password@hostname')
Then configure the Hdb/Tdb Extractor class properties to use this user/password for querying:
1import PyTango
2
3PyTango.Database().put_class_property('HdbExtractor',{'DbConfig':'user:password@hostname'})
4
5PyTango.Database().put_class_property('TdbExtractor',{'DbConfig':'user:password@hostname'})
You can test now access from a Reader (see recipes below) object or from a taurustrend/ArchivingBrowser UI (Taurus required):
1python PyTangoArchiving/widget/ArchivingBrowser.py
Download
Download PyTangoArchiving from sourceforge:
1svn co https://svn.code.sf.net/p/tango-cs/code/archiving/tool/PyTangoArchiving/trunk
Submodules
api,
getting servers/devices/instances implied in the archiving system and allowing
historic,
configuration and reading of historic data
snap,
configuration and reading of snapshot data,
xml,
conversion between xml and csv files
scripts,
configuration scripts
reader,
providing the useful Reader and ReaderProcess objects to retrieve archived data
General usage
In all these examples you can use hdb or tdb just replacing one by the other
Get archived values for an attribute
The reader object provides a fast access to archived values
In [9]: import PyTangoArchiving
In [10]: rd = PyTangoArchiving.Reader('hdb')
In [11]: rd.get_attribute_values('expchan/eh_emet02_ctrl/3/value','2013-03-20 10:00','2013-03-20 11:00')
Out[11]:
[(1363770788.0, 5.79643e-14),
(1363770848.0, 5.72968e-14),
(1363770908.0, 5.7621e-14),
(1363770968.0, 6.46782e-14),
...
Start/Stop/Check attributes
You must create an Archiving api object and pass to it the list of attributes with its archiving config:
1import PyTangoArchiving
2hdb = PyTangoArchiving.ArchivingAPI('hdb')
3attrs = ['expchan/eh_emet03_ctrl/3/value','expchan/eh_emet03_ctrl/4/value']
4
5#Archive every 15 seconds if change> +/-1.0, else every 300 seconds
6modes = {'MODE_A': [15000.0, 1.0, 1.0], 'MODE_P': [300000.0]}
7
8#If you omit the modes argument then archiving will be every 60s
9hdb.start_archiving(attrs, modes)
10
11hdb.load_last_values(attrs)
12{'expchan/eh_emet02_ctrl/3/value': [[datetime.datetime(2013, 3, 20, 11, 38, 9),
13 7.27081e-14]],
14 'expchan/eh_emet02_ctrl/4/value': [[datetime.datetime(2013, 3, 20, 11, 39),
15 -3.78655e-08]]
16}
17
18hdb.stop_archiving(attrs)
Loading a .CSV file into Archiving
The .csv file must have a shape like this one (any row starting with ‘#’ is ignored):
1Host Device Attribute Type ArchivingMode Periode >15 MinRange MaxRange
2
3#This header lines are mandatory!!!
4@LABEL Unique ID
5@AUTHOR Who?
6@DATE When?
7@DESCRIPTION What?
8
9#host domain/family/member attribute HDB/TDB/STOP periodic/absolute/relative
10
11cdi0404 LI/DI/BPM-ACQ-01 @DEFAULT periodic 300
12 ADCChannelAPeak HDB absolute 15 1 1
13 TDB absolute 5 1 1
14 ADCChannelBPeak HDB absolute 15 1 1
15 TDB absolute 5 1 1
16 ADCChannelCPeak HDB absolute 15 1 1
17 TDB absolute 5 1 1
18 ADCChannelDPeak HDB absolute 15 1 1
19 TDB absolute 5 1 1
The command to insert it is:
1import PyTangoArchiving
2PyTangoArchiving.LoadArchivingConfiguration('/...fbecheri_20130319.csv','hdb',launch=True)
There are some arguments to modify Loading behavior.
launch:
if not explicitly True then archiving is not triggered, it just verifies that format of the file is Ok and attributes are available
force:
if False the loading will stop at first error, if True then it tries all attributes even if some failed
overwrite:
if False attributes already archived will be skipped.
Checking the status of the archiving
1hdb = PyTangoArchiving.ArchivingAPI('hdb')
2hdb.load_last_values()
3filter = "/" #Put here whatever you want to filter the attribute names
4lates = [a for a in hdb if filter in a and hdb[a].archiver and hdb[a].modes.get('MODE_P') and hdb[a].last_date<(time.time()-(3600+1e-3*hdb[a].modes['MODE_P'][0]))]
5
6#Get the list of attributes that cannot be read from the control system (ask system responsibles)
7unav = [a for a in lates if not fandango.device.check_attribute(a,timeout=6*3600)]
8#Get the list of attributes that are not being archived
9lates = sorted(l for l in lates if l not in unav)
10#Get the list of archivers not running properly
11bad_archs = [a for a,v in hdb.check_archivers().items() if not v]
12
13#Restarting the archivers/attributes that failed
14bads = [l for l in lates if hdb[l] not in bad_archs]
15astor = fandango.Astor()
16astor.load_from_devs_list(bad_archs)
17astor.restart_servers()
18hdb.restart_archiving(bads)
Restart of the whole archiving system
1admin@archiving:> archiving_service.py stop-all
2...
3admin@archiving:> archiving_service.py start-all
4...
5admin@archiving:> archiving_service.py status
6
7#see archiving_service.py help for other usages
Using the Python API
Start/Stop of an small (<10) list of attributes
1#Stopping ...
2api.stop_archiving(['bo/va/dac/input','bo/va/dac/settings'])
3
4#Starting with periodic=60s ; relative=15s if +/-1% change
5api.start_archiving(['bo/va/dac/input','bo/va/dac/settings'],{'MODE_P':[60000],'MODE_R':[15000,1,1]})
6
7#Restarting and keeping actual configuration
8
9attr_name = 'bo/va/dac/input'
10api.start_archiving([attr_name],api.attributes[attr_name].extractModeString())
Checking if a list of attributes is archived
In [16]: hdb = PyTangoArchiving.api('hdb')
In [17]: sorted([(a,hdb.load_last_values(a)) for a in hdb if a.startswith('bl04')])
Out[17]:
[('bl/va/elotech-01/output_1',
[[datetime.datetime(2010, 7, 2, 15, 53), 6.0]]),
('bl/va/elotech-01/output_2',
[[datetime.datetime(2010, 7, 2, 15, 53, 11), 0.0]]),
('bl/va/elotech-01/output_3',
[[datetime.datetime(2010, 7, 2, 15, 53, 23), 14.0]]),
('bl/va/elotech-01/output_4',
[[datetime.datetime(2010, 7, 2, 15, 52, 40), 20.0]]),
...
Getting information about attributes archived
Getting the total number of attributes:
1import PyTangoArchiving
2api = PyTangoArchiving.ArchivingAPI('hdb')
3len(api.attributes) #All the attributes in history
4len([a for a in api.attributes.values() if a.archiving_mode]) #Attributes configured
Getting the configuration of attribute(s):
1#Getting as string
2modes = api.attributes['rs/da/bpm-07/CompensateTune'].archiving_mode
3
4#Getting it as a dict
5api.attributes['sr/da/bpm-07/CompensateTune'].extractModeString()
6
7#OR
8PyTangoArchiving.utils.modes_to_dict(modes)
Getting the list of attributes not updated in the last hour:
1failed = sorted(api.get_attribute_failed(3600).keys())
Getting values for an attribute:
1import PyTangoArchiving,time
2
3reader = PyTangoArchiving.Reader() #An HDB Reader object using HdbExtractors
4#OR
5reader = PyTangoArchiving.Reader(db='hdb',config='pim:pam@pum') #An HDB reader accessing to MySQL
6
7attr = 'bo04/va/ipct-05/state'
8dates = time.time()-5*24*3600,time.time() #5days
9values = reader.get_attribute_values(attr,*dates) #it returns a list of (epoch,value) tuples
Exporting values from a list of attributes as a text (csv / ascii) file
1from PyTangoArchiving import Reader
2rd = Reader(db='hdb') #If HdbExtractor.DbConfig property is set one argument is enough
3attrs = [
4 'bl11-ncd/vc/eps-plc-01/pt100_1',
5 'bl11-ncd/vc/eps-plc-01/pt100_2',
6 ]
7
8#If you ignore text argument you will get lists of values, if text=True then you get a tabulated file.
9ascii_values = rd.get_attributes_values(attrs,
10 start_date='2010-10-22',stop_date='2010-10-23',
11 correlate=True,text=True)
12
13print ascii_values
14
15#Save it as .csv if you want ...
16open('myfile.csv','w').write(ascii_values)
Filtering State changes for a device
1import PyTangoArchiving as pta
2rd = pta.Reader('hdb','...:...@...')
3vals = rd.get_attribute_values('bo02/va/ipct-02/state','2010-05-01 00:00:00','2010-07-13 00:00:00')
4bads = []
5for i,v in enumerate(vals[1:]):
6 if v[1]!=vals[i-1][1]:
7 bads.append((v[0],vals[i-1][1],v[1]))
8report = [(time.ctime(v[0]),str(PyTango.DevState.values[int(v[1])] if v[1] is not None else 'None'),str(PyTango.DevState.values[int(v[2])] if v[2] is not None else 'None')) for v in bads]
9
10report =
11[('Sat May 1 00:07:03 2010', 'UNKNOWN', 'ON'),
12...
Getting a table with last values for all attributes of a same device
1hours = 1
2device = 'bo/va/ipct-05'
3attrs = [a for a in reader.get_attributes() if a.lower().startswith(device)]
4vars = dict([(attr,reader.get_attribute_values(attr,time.time()-hours*3600)) for attr in attrs])
5table = [[time.ctime(t0)]+
6 [([v for t,v in var if t<=t0] or [None])[-1] for attr,var in sorted(vars.items())]
7 for t0,v0 in vars.values()[0]]
8print('\n'.join(
9 ['\t'.join(['date','time']+[k.lower().replace(device,'') for k in sorted(vars.keys())])]+
10 ['\t'.join([str(s) for s in t]) for t in table]))
Using CSV files
Loading an HDB/TDB configuration file
Create dedicated archivers first
If you want to use this option it will require some RAM resources in the host machine (64MbRAM/250Attributes) and installing the ALBA-Archiving bliss package.
1from PyTangoArchiving.files import DedicateArchiversFromConfiguration
2DedicateArchiversFromConfiguration('LX_I_Archiving.csv','hdb',launch=True)
TDB Archiving works different as it shouldn’t be working on diskless machines, using instead a centralized host for all archiver devices.
1DedicateArchiversFromConfiguration('LX_I_Archiving.csv','tdb',centralized='archiving01',launch=True)
Loading the .csv files
All the needed code to do it is:
1import PyTangoArchiving
2
3#With launch=False this function will do a full check of the attributes and print the results
4PyTangoArchiving.LoadArchivingConfiguration('/data/Archiving//LX_I_Archiving_.csv','hdb',launch=False)
5
6#With launch=True configuration will be recorded and archiving started
7PyTangoArchiving.LoadArchivingConfiguration('/data/Archiving//LX_I_Archiving_.csv','hdb',launch=True)
8
9#To force archiving of all not-failed attributes
10PyTangoArchiving.LoadArchivingConfiguration('/data/Archiving//LX_I_Archiving_.csv','hdb',launch=True,force=True)
11
12#Starting archiving in TDB mode (kept 5 days only)
13PyTangoArchiving.LoadArchivingConfiguration('/data/Archiving//LX_I_Archiving_.csv','tdb',launch=True,force=True)
Note
You must take in account the following conditions:
Names of attributes must match the NAME, not the LABEL! (that’s a common mistake)
Devices providing the attributes must be running when you setup archiving.
Regular expressions are NOT ALLOWED (I know previous releases allowed it, but never worked really well)
filtering a list of CSV configurations / attributes to load
You can use GetConfigFiles and filters/exclude to select a predefined list of attributes
1import PyTangoArchiving as pta
2
3filters = {'name':".*"}
4exclude = {'name':"(s.*bpm.*)|(s10.*rf.*)|(s14.*rf.*)"}
5
6#TDB
7confs = pta.GetConfigFiles(mask='.*(RF|VC).*')
8for target in confs:
9 pta.LoadArchivingConfiguration(target,launch=True,force=True,overwrite=True,dedicated=False,schema='tdb',filters=filters,exclude=exclude)
10
11#HDB
12confs = pta.GetConfigFiles(mask='.*BO.*(RF|VC).*')
13for target in confs:
14 pta.LoadArchivingConfiguration(target,launch=True,force=True,overwrite=True,dedicated=True,schema='hdb',filters=filters,exclude=exclude)
Comparing a CSV file with the actual configuration
1import PyTangoArchiving
2api = PyTangoArchiving.ArchivingAPI('hdb')
3config = PyTangoArchiving.ParseCSV('Archiving_RF_.csv')
4
5for attr,conf in config.items():
6 if attr not in api.attributes or not api.attributes[attr].archiving_mode:
7 print '%s not archived!' % attr
8 elif PyTangoArchiving.utils.modes_to_string(api.check_modes(conf['modes']))!=api.attributes[attr].archiving_mode:
9 print '%s: %s != %s' %(attr,PyTangoArchiving.utils.modes_to_string(api.check_modes(conf['modes'])),api.attributes[attr].archiving_mode)
Checking and restarting a known system from a .csv
1import PyTangoArchiving.files as ptaf
2borf = '/data/Archiving/BO_20100603_v2.csv'
3config = ptaf.ParseCSV(borf)
4import PyTangoArchiving.utils as ptau
5hdb = PyTangoArchiving.ArchivingAPI('hdb')
6
7missing = [
8 'bo/ra/fim-01/remotealarm',
9 'bo/ra/fim-01/rfdet1',
10 'bo/ra/fim-01/rfdet2',
11 'bo/ra/fim-01/arcdet5',
12 'bo/ra/fim-01/rfdet3',
13 'bo/ra/fim-01/arcdet3',
14 'bo/ra/fim-01/arcdet2',
15 'bo/ra/fim-01/vacuum']
16
17ptau.check_attribute('bo/ra/fim-01/remotealarm')
18missing = 'bo/ra/fim-01/arcdet4|bo/ra/fim-01/remotealarm|bo/ra/fim-01/rfdet1|bo/ra/fim-01/rfdet2|bo/ra/fim-01/arcdet5|bo/ra/fim-01/rfdet3|bo/ra/fim-01/arcdet3|bo/ra/fim-01/arcdet2|bo/ra/fim-01/vacuum'
19
20ptaf.LoadArchivingConfiguration(borf,filters={'name':missing},launch=True)
21ptaf.LoadArchivingConfiguration(borf,filters={'name':'bo/ra/eps-plc.*'},stop=True,force=True)
22ptaf.LoadArchivingConfiguration(borf,filters={'name':'bo/ra/eps-plc.*'},launch=True,force=True)
23
24rfplc = ptaf.ParseCSV(borf,filters={'name':'bo/ra/eps-.*'})
25stats = ptaf.CheckArchivingConfiguration(borf,period=300)