ssh - How to view the logs of a spark job after it has completed and the context is closed? -

-

i running pyspark, spark 1.3, standalone mode, client mode.

i trying investigate spark job looking @ jobs past , comparing them. want view logs, configuration settings under jobs submitted, etc. i'm running trouble viewing logs of jobs after context closed.

when submit job, of course open spark context. while job running, i'm able open spark web ui using ssh tunneling. and, can access forwarded port localhost:<port no>. can view jobs running, , ones completed, this:

spark web ui example

then, if wish see logs of particular job, can using ssh tunnel port forwarding see logs on particular port particular machine job.

then, job fails, context still open. when happens, still able see logs above method.

but, since don't want have of these contexts open @ once, when job fails, close context. when close context, job appears under "completed applications" in image above. now, when try view logs using ssh tunnel port forwarding, before (localhost:<port no>), gives me page not found.

how view logs of job after context closed? and, imply relationship between spark context , logs kept? thank you.

again, running pyspark, spark 1.3, standalone mode, client mode.

spark event log / history-server use case.

enable event log

if conf/spark-default.conf not exist

cp conf/spark-defaults.conf.template conf/spark-defaults.conf

add following configuration conf/spark-default.conf. 

# enabled event log spark.eventlog.enabled  true  // store event log spark.eventlog.dir file:///users/rockieyang/git/spark/spark-events  // tell history server event log spark.history.fs.logdirectory file:///users/rockieyang/git/spark/spark-events

history server

start history server

sbin/start-history-server.sh

check history, default port 18080

http://localhost:18080/


Comments

Post a Comment

Popular posts from this blog

javascript - Slick Slider width recalculation -

-
i'm trying add slide in panel on slick container css. carousel works except new width not being calculated when space has been added. recalibrate once advance next slide i'd happen each time panel opened or closed. i've tried adding both margin , padding, neither makes difference. ideas? https://jsfiddle.net/mhigley/dpf7mlpl/ html: <button type="button" id="button"><i class="fa fa-arrow-right"></i></button> <aside> <h2>slide in panel</h2> <ol> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> </ol> </aside> <main role="main"> <div class="sections"> <section> <article> <h2>first slide</h2> <p>lorem ipsum d...
Read more

jsf - PrimeFaces Datatable - What is f:facet actually doing? -

-
looking through primefaces code in app noticed following line: <f:facet name="header">#{trainsearch.traincount} trains</f:facet> it looks overriding header, makes sense, can explain me in little more detail? what happening line of code? complete code listed below: <p:datatable id = "results" value = "#{trainsearch.trains}" var = "train" rendered="#{not empty trainsearch.trains}" styleclass = "train-search-table horizontal-border"> <f:facet name="header">#{trainsearch.traincount} trains</f:facet> <p:column headertext = "train id"> <p:panelgrid columns="1" styleclass = "train-id-grid" layout = "grid"> <h:outputtext styleclass = "train-id-label" value="#{train.traini}"/> <h:outputtex...
Read more

angular2 services - Angular 2 RC 4 Http post not firing -

-
i trying fire post request app , it's not working, tried multiple ways. don't see request being fired under network tab in chrome. not sure problem is, appreciated. import { injectable } '@angular/core'; import { http, headers } '@angular/http'; @injectable() export class userservice{ private _url="users"; constructor(private _http: http){ } checkifusernameexists(username){ return this._http.get(this._url+"/"+username) .map(response => response.json()) } createuser(user){ let headers = new headers(); headers.append('content-type', 'application/json') console.log(json.stringify(user)); return this._http.post(this._url, json.stringify(user),{headers: headers}) .map(response => response.json()) } } first need capture error , solve error: this._http.post(this._url, json.stringify(user),{headers: headers}) .subscribe( ...
Read more