PerfectHadoop: MapReduce History Server

This project provides a Swift wrapper of MapReduce History Server REST API:

• MapReduceHistory(): access to all applications on map / reduce history server.

Connect to Hadoop Map Reduce History Server

To connect to your Hadoop Map / Reduce History server by Perfect, initialize a MapReduceHistory() object with sufficient parameters:

// this connection could possibly do some basic operations
let history = MapReduceHistory(host: "mapReduceHistory.somedomain.com", port: 19888)

or connect to Hadoop Map / Reduce History server with a valid user name:

// add user name if need
let history = MapReduceHistory(host: "mapReduceHistory.somedomain.com", port: 19888, user: "your user name")

Authentication

If using Kerberos to authenticate, please try codes below:

// set auth to kerberos
let history = MapReduceHistory(host: "mapReduceHistory.somedomain.com", port: 19888, user: "username", auth: .krb5)

Parameters of MapReduceHistory Object

Get General Information

Call checkInfo() to get the general information of a Hadoop MapReduce History Server in form of a MapReduceHistory.Info structure:

guard let inf = try history.checkInfo() else {
    // something goes wrong here
}
print(inf.startedOn)
print(inf.hadoopVersion)
print(inf.hadoopBuildVersion)
print(inf.hadoopVersionBuiltOn)

Members of MapReduceHistory.Info

Historical MapReduce Jobs

Call checkJobs() to return an array of Job structure. The jobs resource provides a list of the MapReduce jobs that have finished. It does not currently return a full list of parameters. The simplest form is checkJobs(), which will return all jobs available:

let jobs = try history.checkJobs()
// or you can narrow the query result by set some parameters:
// let jobs = try his.checkJobs(state: .SUCCEEDED, queue: "default", limit: 10)
jobs.forEach { j in
  print(j.id)
  print(j.name)
  print(j.queue)
  print(j.state)
}

Parameters of checkJobs()

Data Structure of Job

Elements of the acls object

Check a Specific Job

It is also possible to check a specific historical job by a job id:

let job = try history.checkJob(jobId: "job_1484231633049_0005")

See Data Structure of Job as above.

JobAttempt of Job

When you make a request for the list of job attempts, the information will be returned as an array of job attempt objects.

guard let attempts = try history.checkJobAttempts(jobId: "job_1484231633049_0005") else {
    // something goes wrong
}
attempts.forEach { attempt in
    print(attempt.id)
    print(attempt.containerId)
    print(attempt.nodeHttpAddress)
    print(attempt.nodeId)
    print(attempt.startTime)
}

JobAttempt Structure:

Counters of Job

With the job counters API, you can object a collection of resources that represent al the counters for that job.

guard let js = try his.checkJobCounters(jobId: "job_1484231633049_0005") else {
    // something wrong
}//end guard
js.counterGroup.forEach{ group in
  print(group.counterGroupName)
  group.counters.forEach { counter in
    print(counter.name)
    print(counter.mapCounterValue)
    print(counter.reduceCounterValue)
    print(counter.totalCounterValue)
  }//next counter
}//next group

JobCounter Object

CounterGroup Object

Counter Object

Check Configuration of Job

Use checkJobConfig() to check configuration of a specific job:

guard let config = try his.checkJobConfig(jobId: "job_1484231633049_0005") else {
    /// something wrong
}
// print the configuration path
print(config.path)
// check properties of configuration file
for p in config.property {
  print(p.name)
  print(p.value)
  print(p.source)
}

JobConfig Object

A job configuration resource contains information about the job configuration for this job.

Property Object

Tasks of a Job

Use checkJobTasks() to collect task information from a specific job:

// get all tasks information from a specific job
let tasks = try his.checkJobTasks(jobId: "job_1484231633049_0005")

// print properties of each task
for t in tasks {
  print(t.progress)
  print(t.elapsedTime)
  print(t.state)
  print(t.startTime)
  print(t.id)
  print(t.type)
  print(t.successfulAttempt)
  print(t.finishTime)
}//next t

Parameters of checkJobTasks()

Calling checkJobTasks() requires a jobId all the time, however, you can also specify an extra parameter to indicate what type of tasks shall be listed in the query results:

• jobId: a string represents the id of the job
• taskType: optional, could be .MAP or .REDUCE

JobTask Object

The checkJobTasks() will return an array of JobTask data structure, as described below:

Query a Specific JobTask

If both jobId and jobTaskId are available, you can use checkJobTask() to perform a query on a specific job task:

guard let task = try his.checkJobTask(jobId: "job_1484231633049_0005", taskId: "task_1326381300833_2_2_m_0") else {
    // something wrong
}
print(task.progress)

The return value of checkJobTask will be a JobTask object, as described above.

Task Counters of Job

Method checkJobTaskCounters() can return counters of a specific task, as the example below:

guard let js = try his.checkJobTaskCounters(jobId: "job_1484231633049_0005", taskId: "task_1326381300833_2_2_m_0") else {
    // something wrong
}
// print the id of the job task counters
print(js.id)
// check out each counter group
js.taskCounterGroup.forEach{ group in
  print(group.counterGroupName)
  // print counters in each group.
  group.counters.forEach { counter in
    print(counter.name)
    print(counter.value)
  }
}

JobTaskCounter Object

Task Attempts

Function checkJobTaskAttempts() can return attempts from a specific job task.

let jobTaskAttempts = try his.checkJobTaskAttempts(jobId: "job_1484231633049_0005", taskId: "task_1326381300833_2_2_m_0")
for attempt in jobTaskAttempts {
  print(attempt.id)
  print(attempt.diagnostics)
  print(attempt.assignedContainerId)
  print(attempt.rack)
  print(attempt.state)
  print(attempt.progress)
}//next

Once done, the checkJobTaskAttempts will return an array of TaskAttempt object, as described below:

TaskAttempt Object

Elements of the TaskAttempt object:

Besides, For reduce task attempts you also have the following fields:

Query a Specific TaskAttempt

If a taskAttempt id is available, you can also check the specific one by checkJobTaskAttempt(), as demo below:

guard let jobTaskAttempts = try his.checkJobTaskAttempt(jobId: "job_1484231633049_0005", taskId: "task_1326381300833_2_2_m_0", "attempt_1326381300833_2_2_m_0_0") else {
    // something wrong
    print("there is no such a task attempt")
    return
}//end guard
print(attempt.id)
print(attempt.diagnostics)
print(attempt.assignedContainerId)
print(attempt.rack)
print(attempt.state)
print(attempt.progress)

The checkJobTaskAttempt() will return such an TaskAttempt object as described above.

Attempt Counters of Task

Method checkJobTaskAttemptCounters() can return counters of a specific task attempt, as the example below:

guard let counters = try his.checkJobTaskAttemptCounters(jobId: "job_1484231633049_0005", taskId: "task_1326381300833_2_2_m_0", "attempt_1326381300833_2_2_m_0_0") else {
    // something wrong
    print("Task Attempt Counters Acquiring failed")
    return
}
// print the counters' id
print(counters.id)
// enumerate all groups
for group in counters.taskAttemptCounterGroup {
  // print group name
  print(group.counterGroupName)
  // enumerate all counters in this group
  for counter in group.counters {
    print(counter.name)
    print(counter.value)
  }//next counter
}//next group
}

JobTaskAttemptCounters Object