amazon web services - Passing variables between EC2 instances in multi-step AWS data pipeline -

-

i have pipeline setup wherein have 3 main stages:

1) take input zipped file, unzip file in s3. run basic verification on each file guarantee integrity, move step 2

2) kick off 2 simultaneous processing tasks on separate ec2 instances (parallelization of step saves lot of time, need efficiency sake). each ec2 instance run data processing steps on of files in s3 unzipped in step 1, files required different each instance.

3) after 2 simultaneous processes both done, spin ec2 instance final data processing. once done, run cleanup job remove unzipped files s3, leaving original zip file in place.

so, 1 of problems we're running have 4 ec2 instances run pipeline process, there global parameters each ec2 instance have access to. if running on single instance, of course use shell variables accomplish task, need separate instances efficiency. our best idea store flat file in s3 bucket has access these global variables , read them on initialization , write them if need change. gross , there seems there should better way, can't figure 1 out yet. saw there's way set parameters can accessed @ part of pipeline, looks can set on per pipeline level, not on granularity of each run of pipeline. have resources here? appreciated.

we able solve using dynamodb keep track of variables/state. pipeline doesn't have mechanism this, other parameter values, unfortunately work per pipeline, not per job. you'll need setup dynamodb instance , use pipeline job id keep track of state, connecting via cli tools or sdk.


Comments

Post a Comment

Popular posts from this blog

javascript - Slick Slider width recalculation -

-
i'm trying add slide in panel on slick container css. carousel works except new width not being calculated when space has been added. recalibrate once advance next slide i'd happen each time panel opened or closed. i've tried adding both margin , padding, neither makes difference. ideas? https://jsfiddle.net/mhigley/dpf7mlpl/ html: <button type="button" id="button"><i class="fa fa-arrow-right"></i></button> <aside> <h2>slide in panel</h2> <ol> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> </ol> </aside> <main role="main"> <div class="sections"> <section> <article> <h2>first slide</h2> <p>lorem ipsum d...
Read more

jsf - PrimeFaces Datatable - What is f:facet actually doing? -

-
looking through primefaces code in app noticed following line: <f:facet name="header">#{trainsearch.traincount} trains</f:facet> it looks overriding header, makes sense, can explain me in little more detail? what happening line of code? complete code listed below: <p:datatable id = "results" value = "#{trainsearch.trains}" var = "train" rendered="#{not empty trainsearch.trains}" styleclass = "train-search-table horizontal-border"> <f:facet name="header">#{trainsearch.traincount} trains</f:facet> <p:column headertext = "train id"> <p:panelgrid columns="1" styleclass = "train-id-grid" layout = "grid"> <h:outputtext styleclass = "train-id-label" value="#{train.traini}"/> <h:outputtex...
Read more

angular2 services - Angular 2 RC 4 Http post not firing -

-
i trying fire post request app , it's not working, tried multiple ways. don't see request being fired under network tab in chrome. not sure problem is, appreciated. import { injectable } '@angular/core'; import { http, headers } '@angular/http'; @injectable() export class userservice{ private _url="users"; constructor(private _http: http){ } checkifusernameexists(username){ return this._http.get(this._url+"/"+username) .map(response => response.json()) } createuser(user){ let headers = new headers(); headers.append('content-type', 'application/json') console.log(json.stringify(user)); return this._http.post(this._url, json.stringify(user),{headers: headers}) .map(response => response.json()) } } first need capture error , solve error: this._http.post(this._url, json.stringify(user),{headers: headers}) .subscribe( ...
Read more