extractor_prod.MetaData Class Reference

Inheritance diagram for extractor_prod.MetaData:

Public Member Functions
def	__init__ (self, inputfile)
def	extract_metadata_to_pipe (self)
def	get_job (self, proc)

Static Public Member Functions
def	wait_for_subprocess (jobinfo, q)
def	mdart_gen (jobtuple)
def	md_handle_application (md)

Public Attributes
	inputfile

Static Public Attributes
string	mdtext = ''
	mdtext = ''.join(line.replace(", ,", ",") for line in jobtuple[0].split(' ') if line[-3:-1] != ' ,') More...
	mdtop = json.JSONDecoder().decode(mdtext)
	file_name = list(mdtop.keys())[0]

Static Private Attributes
	__metaclass__ = abc.ABCMeta

Detailed Description

Base class to hold / interpret general metadata

Definition at line 33 of file extractor_prod.py.

Constructor & Destructor Documentation

def extractor_prod.MetaData.__init__	(		self,
			inputfile
	)

Definition at line 38 of file extractor_prod.py.

    def __init__(self, inputfile):
        self.inputfile = inputfile

Member Function Documentation

def extractor_prod.MetaData.extract_metadata_to_pipe

(

self

)

Extract metadata from inputfile into a pipe for further processing.

Definition at line 41 of file extractor_prod.py.

    def extract_metadata_to_pipe(self):
        """Extract metadata from inputfile into a pipe for further processing."""
        local = self.inputfile
        if len(local) > 0:
            proc = Popen(["sam_metadata_dumper", local], stdout=PIPE,
                         stderr=PIPE)
        else:
            url = inputfile
            proc = Popen(["sam_metadata_dumper", url], stdout=PIPE,
                         stderr=PIPE)
        if len(local) > 0 and local != self.inputfile:
            os.remove(local)
        return proc
    

def extractor_prod.MetaData.get_job	(		self,
			proc
	)

Run the proc in a 60-sec timeout queue, return stdout, stderr

Definition at line 55 of file extractor_prod.py.

    def get_job(self, proc):
        """Run the proc in a 60-sec timeout queue, return stdout, stderr"""
        q = queue.Queue()
        thread = threading.Thread(target=self.wait_for_subprocess, args=[proc, q])
        thread.start()
        thread.join(timeout=7200)
        if thread.is_alive():
            print('Terminating subprocess because of timeout.')
            proc.terminate()
            thread.join()
        rc = q.get()
        jobout = q.get()
        joberr = q.get()
        if rc != 0:
            raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
        return jobout, joberr
    

def extractor_prod.MetaData.md_handle_application (   md )

static

If there's no application key in md dict, create the key with a blank dictionary.
Then return md['application'], along with mdval

Definition at line 94 of file extractor_prod.py.

    def md_handle_application(md):
        """If there's no application key in md dict, create the key with a blank dictionary.
        Then return md['application'], along with mdval"""
        if 'application' not in md:
            md['application'] = {}
        return md['application']

def extractor_prod.MetaData.mdart_gen (   jobtuple )

static

Take Jobout and Joberr (in jobtuple) and return mdart object from that

Definition at line 82 of file extractor_prod.py.

    def mdart_gen(jobtuple):
        """Take Jobout and Joberr (in jobtuple) and return mdart object from that"""

def extractor_prod.MetaData.wait_for_subprocess (   jobinfo,   q  )

static

Run jobinfo, put the return code, stdout, and stderr into a queue

Definition at line 73 of file extractor_prod.py.

    def wait_for_subprocess(jobinfo, q):
        """Run jobinfo, put the return code, stdout, and stderr into a queue"""
        jobout, joberr = jobinfo.communicate()
        rc = jobinfo.poll()
        for item in (rc, jobout, joberr):
            q.put(item)
        return

Member Data Documentation

extractor_prod.MetaData.__metaclass__ = abc.ABCMeta

staticprivate

Definition at line 35 of file extractor_prod.py.

extractor_prod.MetaData.file_name = list(mdtop.keys())[0]

static

Definition at line 90 of file extractor_prod.py.

extractor_prod.MetaData.inputfile

Definition at line 39 of file extractor_prod.py.

string extractor_prod.MetaData.mdtext = ''

static

mdtext = ''.join(line.replace(", ,", ",") for line in jobtuple[0].split('
') if line[-3:-1] != ' ,')

Definition at line 85 of file extractor_prod.py.

extractor_prod.MetaData.mdtop = json.JSONDecoder().decode(mdtext)

static

Definition at line 86 of file extractor_prod.py.

---

The documentation for this class was generated from the following file:

• duneutil/scripts/extractor_prod.py