python.extractor_dict.MetaData Class Reference

Inheritance diagram for python.extractor_dict.MetaData:

Public Member Functions
def	__init__ (self, inputfile)
def	extract_metadata_to_pipe (self)
def	get_job (self, proc)

Static Public Member Functions
def	wait_for_subprocess (jobinfo, q)
def	mdart_gen (jobtuple)
def	md_handle_application (md)

Public Attributes
	inputfile

Static Private Attributes
	__metaclass__ = abc.ABCMeta

Detailed Description

Base class to hold / interpret general metadata

Definition at line 33 of file extractor_dict.py.

Constructor & Destructor Documentation

def python.extractor_dict.MetaData.__init__	(		self,
			inputfile
	)

Definition at line 38 of file extractor_dict.py.

    def __init__(self, inputfile):
        self.inputfile = inputfile

Member Function Documentation

def python.extractor_dict.MetaData.extract_metadata_to_pipe

(

self

)

Extract metadata from inputfile into a pipe for further processing.

Definition at line 41 of file extractor_dict.py.

    def extract_metadata_to_pipe(self):
        """Extract metadata from inputfile into a pipe for further processing."""
        local = project_utilities.path_to_local(self.inputfile)
        if len(local) > 0:
            proc = Popen(["sam_metadata_dumper", local], stdout=PIPE,
                         stderr=PIPE)
        else:
            url = project_utilities.path_to_url(inputfile)
            proc = Popen(["sam_metadata_dumper", url], stdout=PIPE,
                         stderr=PIPE)
        if len(local) > 0 and local != self.inputfile:
            os.remove(local)
        return proc
    

def python.extractor_dict.MetaData.get_job	(		self,
			proc
	)

Run the proc in a 60-sec timeout queue, return stdout, stderr

Definition at line 55 of file extractor_dict.py.

    def get_job(self, proc):
        """Run the proc in a 60-sec timeout queue, return stdout, stderr"""
        q = queue.Queue()
        thread = threading.Thread(target=self.wait_for_subprocess, args=[proc, q])
        thread.start()
        thread.join(timeout=60)
        if thread.is_alive():
            print('Terminating subprocess because of timeout.')
            proc.terminate()
            thread.join()
        rc = q.get()
        jobout = convert_str(q.get())
        joberr = convert_str(q.get())
        if rc != 0:
            raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
        return jobout, joberr
    

def python.extractor_dict.MetaData.md_handle_application (   md )

static

If there's no application key in md dict, create the key with a blank dictionary.
Then return md['application'], along with mdval

Definition at line 95 of file extractor_dict.py.

    def md_handle_application(md):
        """If there's no application key in md dict, create the key with a blank dictionary.
        Then return md['application'], along with mdval"""
        if 'application' not in md:
            md['application'] = {}
        return md['application']

def python.extractor_dict.MetaData.mdart_gen (   jobtuple )

static

Take Jobout and Joberr (in jobtuple) and return mdart object from that

Definition at line 84 of file extractor_dict.py.

    def mdart_gen(jobtuple):
        """Take Jobout and Joberr (in jobtuple) and return mdart object from that"""
        mdtext = ''.join(line.replace(", ,", ",") for line in jobtuple[0].split('\n') if line[-3:-1] != ' ,')
        mdtop = json.JSONDecoder().decode(mdtext)
        if len(list(mdtop.keys())) == 0:
            print('No top-level key in extracted metadata.')
            sys.exit(1)
        file_name = list(mdtop.keys())[0]
        return mdtop[file_name]

def python.extractor_dict.MetaData.wait_for_subprocess (   jobinfo,   q  )

static

Run jobinfo, put the return code, stdout, and stderr into a queue

Definition at line 73 of file extractor_dict.py.

    def wait_for_subprocess(jobinfo, q):
        """Run jobinfo, put the return code, stdout, and stderr into a queue"""
        jobout, joberr = jobinfo.communicate()
        jobout = convert_str(jobout)
        joberr = convert_str(joberr)
        rc = jobinfo.poll()
        for item in (rc, jobout, joberr):
            q.put(item)
        return

Member Data Documentation

python.extractor_dict.MetaData.__metaclass__ = abc.ABCMeta

staticprivate

Definition at line 35 of file extractor_dict.py.

python.extractor_dict.MetaData.inputfile

Definition at line 39 of file extractor_dict.py.

---

The documentation for this class was generated from the following file:

• larbatch/python/extractor_dict.py