Job Provenance

Primary Storage -- Overview

The Job Provenance (JP) Primary Storage Service is responsible to keep the JP data (definition of submitted jobs, execution conditions and environment, and important points of the job life cycle) in a compact and economic form.

The JP Primary storage, as described in section 8.4 of the Architecture deliverable DJRA1.1 provides public interfaces for data storing, retrieval based on basic metadata, and registration of Index servers for incremental feed.

Command interface to JP is completely covered by the WS interface covered here. Bulk file transfers are done via specialised protocols, currently gsiftp only.

Primary Storage -- Operations

CVS revision: $Header: /cvs/jra1mw/org.glite.jp.ws-interface/src/JobProvenancePS.xml,v 1.6 2005/08/12 15:26:03 akrenek Exp $

CommitUpload

Confirm a successfully finished file apload.

Inputs:

xsd:stringdestination: Destination URL returned by StartUpload before.

Outputs: N/A

FeedIndex

Request for feeding a JP Index server (issued by this server).

Inputs:

xsd:stringdestination: Endpoint of the listening index server.
list of xsd:stringattributes: Which attributes of jobs is the index server interested in.
list of primaryQueryconditions: Which jobs is the server interested in.
xsd:booleanhistory: Data on jobs stored at PS in the past are required.
xsd:booleancontinuous: Data on jobs that will arrive in future are required.

Outputs:

xsd:stringfeedId: Unique ID of the created feed session.
xsd:dateTimefeedExpires: When the session expires.

FeedIndexRefresh

Refresh an existing feed session.

Inputs:

xsd:stringfeedId: Existing feed session ID to be refreshed.

Outputs:

xsd:dateTimefeedExpires: New session expiration time.

GetJobAttributes

Query concrete attributes of a given job.

Inputs:

xsd:stringjobid: The job.
list of xsd:stringattributes: Which attributes should be retrieved.

Outputs:

list of attrValueattrValues: Values of the queried attributes.

GetJobFiles

Return URL's of files for a given single job.

Inputs:

xsd:stringjobid: The job.

Outputs:

list of jppsFilefiles: List of the stored files.

RecordTag

Record an additional user tag.

Inputs:

xsd:stringjobid: Job to which the tag is added.
tagValuetag: Name and value of the tag.

Outputs: N/A

RegisterJob

Inputs:

xsd:stringjob: Jobid of the registered job.
xsd:stringowner: Owner of the job (DN of X509 certificate).

Outputs: N/A

StartUpload

Start uploading a file.

Inputs:

xsd:stringjob: Jobid to which this file is related.
xsd:stringclass: Type of the file (URI). The server must have a plugin handing this type.
xsd:stringname: Name of the file (used to distinguish among more files of the same type).
xsd:dateTimecommitBefore: The client promisses to finish the upload before this time.
xsd:stringcontentType: MIME type of the file.

Outputs:

xsd:stringdestination: URL where the client should upload the file.
xsd:dateTimecommitBefore: Server's view on when the upload must be finished.

Index Server -- Overview

The Job Provenance (JP) Index Server is a volatile counterpart to the permanent JP Primary Storage. Index servers are populated with subsets of data from Primary storage(s) and indexed according to particular user needs.

The interface to Index server contains three logical parts: administraive (control), system and user. The administrative part is used by run-time index server configuration tool, the system one allows Primary storage(s) to feed data into the Index server, and the user one is available to users for queries.

Index Server -- Operations

CVS revision: $Header: /cvs/jra1mw/org.glite.jp.ws-interface/src/JobProvenanceIS.xml,v 1.4 2005/10/15 14:01:43 akrenek Exp $

AddFeed

Called by JP index serve admin tool to ask new primary storage server to feed it. Updates information on PS in index server, according to what JPPS currently knows.

Inputs:

feedSessionfeed: New feed IS URL, filter and query type.

Outputs: N/A

DeleteFeed

Called by JP index serve admin tool to remove one feed session.

Inputs:

xsd:stringfeedId: ID of feed to be removed.

Outputs: N/A

GetFeedIDs

Called by JP index serve admin tool to find out IS open feeds

Inputs:

list of feedSessionfeeds: List of active feeds on IS.

Outputs: N/A

QueryJobs

User query to index server.

Inputs:

list of indexQueryconditions: Query conditions, similar to LB.
list of xsd:stringattributes: Set of attributes to be retrieved directly from index server (if any).

Outputs:

list of jobRecordjobs: List of jobs matching the query.

UpdateJobs

Called by JP primary storage as a response to FeedIndex request. Updates information on jobs in index server, according to what JPPS currently knows.

Inputs:

xsd:stringfeedId: Id of the feed, as returned by JPPS FeedIndex operation.
xsd:booleanfeedDone: Flag of completed batch feed.
list of jobRecordjobAttributes: Attributes per job.

Outputs: N/A

JP Common Types

CVS revision: $Header: /cvs/jra1mw/org.glite.jp.ws-interface/src/JobProvenanceTypes.xml,v 1.11 2005/10/13 15:48:08 akrenek Exp $

attrOrig

Specification of attribute origin.

Enumeration (restriction of xsd:string in WSDL), exactly one of the values must be specified.

Values:

SYSTEM: JP system value, e.g. job owner.
USER: Explicitely stored by the user via RecordTag operation.
FILE: Coming from uploaded file.

attrValue

Single value of an attribute.

Structure (sequence complex type in WSDL)

Fields: ( type name description )

xsd:string name: Name of the attribute, including namespace.
stringOrBlob value: (optional) String value.
xsd:dateTime timestamp: When this value was recorded.
attrOrig origin: Where this value came from.
xsd:string originDetail: (optional)

feedSession

One session between IS and PS (aka feed) charactetristics.

Structure (sequence complex type in WSDL)

Fields: ( type name description )

xsd:string primaryServer: URL of primary server.
list of primaryQuery condition: Filter conditions.
xsd:int history: Query type.
xsd:int continuous: Query type
xsd:string feedId: (optional) Unique ID of the feed session.

genericFault

Structure (sequence complex type in WSDL)

Fields: ( type name description )

xsd:string source
xsd:int code
xsd:string text
xsd:string description: (optional)
genericFault reason: (optional)

indexQuery

Single query condition on a job. Similarly to LB, these outer conditions are logically ANDed.

Structure (sequence complex type in WSDL)

Fields: ( type name description )

xsd:string attr: Which attribute the condition refers to.
attrOrig origin: (optional) Specific attribute origin (if we do care).
list of indexQueryRecord record: List of conditions on attribute attr. These conditions are logically ORed.

indexQueryRecord

Single condition on an attribute.

Structure (sequence complex type in WSDL)

Fields: ( type name description )

queryOp op: Query operation.
stringOrBlob value: (optional) Value to compare attribute with.
stringOrBlob value2: (optional) Value to compare attribute with.

jobRecord

Information on a single job. Used for both feeding JP index server from primary storage and to answer user queries on index server.

Structure (sequence complex type in WSDL)

Fields: ( type name description )

xsd:string jobid: ID of the job.
xsd:string owner: Job owner.
list of attrValue attributes: (optional) Attribute values, required by query/feed and available right now.
list of xsd:string primaryStorage: (optional) User query only: which primary storage(s) have data on this job.
xsd:boolean remove: (optional) UpdateJobs only: this job no longer belongs to the feed. Attribute values are those which caused the change.

jppsFile

JP primary storage file identification.

Structure (sequence complex type in WSDL)

Fields: ( type name description )

xsd:string class: Type of the file (as set on StartUpload).
xsd:string name: Name of the file (if there are more of the same type per job).
xsd:string url: Where the file is stored on JP primary storage.

primaryQuery

A single condition on job.

Structure (sequence complex type in WSDL)

Fields: ( type name description )

xsd:string attr: Attribute name to query.
queryOp op: Operation.
attrOrig origin: (optional) Where the attribute value came from.
stringOrBlob value: Value to compare the job attribute with.
stringOrBlob value2: (optional) Another value (for op = WITHIN).

queryOp

Operators used in queries. Most are self-explanatory.

Enumeration (restriction of xsd:string in WSDL), exactly one of the values must be specified.

Values:

EQUAL
UNEQUAL
LESS
GREATER
WITHIN: The attribute is between two specified values.
EXISTS: The attribute exists (even having a NULL value).

stringOrBlob

Union (choice complex type in WSDL)

Fields: ( type name description )

xsd:string string: String value.
xsd:base64Binary blob: Binary value.

tagValue

A single user-recorded value for a job attribute.

Structure (sequence complex type in WSDL)

Fields: ( type name description )

xsd:string name: Name of the attribute, including namespace.
stringOrBlob value: (optional) Value.