extractor_dict.MetaData Class Reference

Inheritance diagram for extractor_dict.MetaData:

Public Member Functions
def	__init__ (self, inputfile)
def	extract_metadata_to_pipe (self)
def	get_job (self, proc)

Static Public Member Functions
def	wait_for_subprocess (jobinfo, q)
def	mdart_gen (jobtuple)
def	md_handle_application (md)

Public Attributes
	inputfile

Static Private Attributes
	__metaclass__ = abc.ABCMeta

Detailed Description

Base class to hold / interpret general metadata

Definition at line 30 of file extractor_dict.py.

Constructor & Destructor Documentation

def extractor_dict.MetaData.__init__	(		self,
			inputfile
	)

Definition at line 35 of file extractor_dict.py.

    def __init__(self, inputfile):
        self.inputfile = inputfile

Member Function Documentation

def extractor_dict.MetaData.extract_metadata_to_pipe

(

self

)

Extract metadata from inputfile into a pipe for further processing.

Definition at line 38 of file extractor_dict.py.

    def extract_metadata_to_pipe(self):
        """Extract metadata from inputfile into a pipe for further processing."""
        local = project_utilities.path_to_local(self.inputfile)
        if len(local) > 0:
            proc = Popen(["sam_metadata_dumper", local], stdout=PIPE,
                         stderr=PIPE)
        else:
            url = project_utilities.path_to_url(inputfile)
            proc = Popen(["sam_metadata_dumper", url], stdout=PIPE,
                         stderr=PIPE)
        if len(local) > 0 and local != self.inputfile:
            os.remove(local)
        return proc
    

def extractor_dict.MetaData.get_job	(		self,
			proc
	)

Run the proc in a 60-sec timeout queue, return stdout, stderr

Definition at line 52 of file extractor_dict.py.

    def get_job(self, proc):
        """Run the proc in a 60-sec timeout queue, return stdout, stderr"""
        q = queue.Queue()
        thread = threading.Thread(target=self.wait_for_subprocess, args=[proc, q])
        thread.start()
        thread.join(timeout=60)
        if thread.is_alive():
            print('Terminating subprocess because of timeout.')
            proc.terminate()
            thread.join()
        rc = q.get()
        jobout = q.get()
        joberr = q.get()
        if rc != 0:
            raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
        return jobout, joberr
    

def extractor_dict.MetaData.md_handle_application (   md )

static

If there's no application key in md dict, create the key with a blank dictionary.
Then return md['application'], along with mdval

Definition at line 90 of file extractor_dict.py.

    def md_handle_application(md):
        """If there's no application key in md dict, create the key with a blank dictionary.
        Then return md['application'], along with mdval"""
        if 'application' not in md:
            md['application'] = {}
        return md['application']

def extractor_dict.MetaData.mdart_gen (   jobtuple )

static

Take Jobout and Joberr (in jobtuple) and return mdart object from that

Definition at line 79 of file extractor_dict.py.

    def mdart_gen(jobtuple):
        """Take Jobout and Joberr (in jobtuple) and return mdart object from that"""
        mdtext = ''.join(line.replace(", ,", ",") for line in jobtuple[0].split('\n') if line[-3:-1] != ' ,')
        mdtop = json.JSONDecoder().decode(mdtext)
        if len(mdtop.keys()) == 0:
            print('No top-level key in extracted metadata.')
            sys.exit(1)
        file_name = mdtop.keys()[0]
        return mdtop[file_name]

def extractor_dict.MetaData.wait_for_subprocess (   jobinfo,   q  )

static

Run jobinfo, put the return code, stdout, and stderr into a queue

Definition at line 70 of file extractor_dict.py.

    def wait_for_subprocess(jobinfo, q):
        """Run jobinfo, put the return code, stdout, and stderr into a queue"""
        jobout, joberr = jobinfo.communicate()
        rc = jobinfo.poll()
        for item in (rc, jobout, joberr):
            q.put(item)
        return

Member Data Documentation

extractor_dict.MetaData.__metaclass__ = abc.ABCMeta

staticprivate

Definition at line 32 of file extractor_dict.py.

extractor_dict.MetaData.inputfile

Definition at line 36 of file extractor_dict.py.

---

The documentation for this class was generated from the following file:

• duneutil/python/extractor_dict.py