How to import data from MySql into Hive using Apache Nifi? -

-

i trying import data mysql hive using querydatabasetable , puthiveql processors, error occurs.

i have questions:

  1. what output format of puthiveql?
  2. should output table created beforehand or processor that?
  3. where can find template mysql hive process?

here information questions:

  1. the flow files input puthiveql output after have been sent hive (or if send fails), output format (and contents) identical input format/contents.

  2. the output table should created beforehand, send puthiveql "create table if not exists" statement first , create table you.

  3. i'm not aware of existing template, basic approach following:

querydatabasetable -> convertavrotojson -> splitjson -> evaluatejsonpath -> updateattribute (optional) -> replacetext -> puthiveql

  • querydatabasetable incremental fetches of mysql table.

  • convertavrotojson records format can manipulate (there aren't many processors handle avro)

  • splitjson create flow file each of records/rows

  • evaluatejsonpath can extract values records , put them in flow file attributes

  • updateattribute add attributes containing type information. optional, used if using prepared statements puthiveql

  • replacetext builds hiveql statement (insert, e.g.) either parameters (if want prepared statements) or hard-coded values attributes

  • puthiveql executes statement(s) records hive

in nifi 1.0, there convertavrotoorc processor, more efficient way data hive (as query hive). approach convert results of querydatabasetable orc files, placed in hdfs (using puthdfs), , generates partial hive ddl statement create table (using type information avro records). pass statement (after filling in target location) puthiveql, , can start querying table.

there plans puthivestreaming processor takes avro records input, flow querydatabasetable -> puthivestreaming, insert records directly hive (and more efficient multiple insert statements).


Comments

Post a Comment

Popular posts from this blog

javascript - Slick Slider width recalculation -

-
i'm trying add slide in panel on slick container css. carousel works except new width not being calculated when space has been added. recalibrate once advance next slide i'd happen each time panel opened or closed. i've tried adding both margin , padding, neither makes difference. ideas? https://jsfiddle.net/mhigley/dpf7mlpl/ html: <button type="button" id="button"><i class="fa fa-arrow-right"></i></button> <aside> <h2>slide in panel</h2> <ol> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> </ol> </aside> <main role="main"> <div class="sections"> <section> <article> <h2>first slide</h2> <p>lorem ipsum d...
Read more

jsf - PrimeFaces Datatable - What is f:facet actually doing? -

-
looking through primefaces code in app noticed following line: <f:facet name="header">#{trainsearch.traincount} trains</f:facet> it looks overriding header, makes sense, can explain me in little more detail? what happening line of code? complete code listed below: <p:datatable id = "results" value = "#{trainsearch.trains}" var = "train" rendered="#{not empty trainsearch.trains}" styleclass = "train-search-table horizontal-border"> <f:facet name="header">#{trainsearch.traincount} trains</f:facet> <p:column headertext = "train id"> <p:panelgrid columns="1" styleclass = "train-id-grid" layout = "grid"> <h:outputtext styleclass = "train-id-label" value="#{train.traini}"/> <h:outputtex...
Read more

angular2 services - Angular 2 RC 4 Http post not firing -

-
i trying fire post request app , it's not working, tried multiple ways. don't see request being fired under network tab in chrome. not sure problem is, appreciated. import { injectable } '@angular/core'; import { http, headers } '@angular/http'; @injectable() export class userservice{ private _url="users"; constructor(private _http: http){ } checkifusernameexists(username){ return this._http.get(this._url+"/"+username) .map(response => response.json()) } createuser(user){ let headers = new headers(); headers.append('content-type', 'application/json') console.log(json.stringify(user)); return this._http.post(this._url, json.stringify(user),{headers: headers}) .map(response => response.json()) } } first need capture error , solve error: this._http.post(this._url, json.stringify(user),{headers: headers}) .subscribe( ...
Read more