cloudera cdh - Ensure that Impala query gets materialized -

-

it there reliable , efficient way ensure impala query results gets materialized without printing results console? example use inner join query.

the obvious way materialize query results create table select.

create table t3 stored parquet select t1.* t1 inner join t2 on t1.id=t2.id;

the problem writes disc therefore inefficient. i'm looking efficient way execute query , ensure results materialized.

as example, in spark can use .cache method followed .count ensure query materialized.

val t3 = t1.join(t2, "id") t3.cache t3.count

i try workaround sub-query.

select count(*) (select t1.* t1 inner join t2 on t1.id=t2.id) t3;

but still need ensure sub-query materialized, not obvious if query optimizer discovers i'm interested in total count. maybe there hints enforce or other tricks?

afaik can't impala, , never able to.
cloudera designed tool support bi tools such tableau, qlik, microstrategy etc. -- not support ad hoc etl scripts.

on other hand hive ships "hpl-sql" procedural language wrapper might fit needs. caveats:

  • requires hive 2.0+
  • requires running whole script inside hpl-sql interpreter, not base hive client (nor standard jdbc connection)

and hpl-sql tool claims supports impala queries never investigated claim. solve problem, kind of clumsy workaround.

references:
  hive-11055 (pl/hql tool contributed hive code base)
  hpl/sql website

speaking of workarounds, why not use spark, suggested yourself? might read impala/hive tables, either spark native parquet libraries, or custom jdbc connection impala daemon. in essence similar hpl/sql solution.


Comments

Post a Comment

Popular posts from this blog

javascript - Slick Slider width recalculation -

-
i'm trying add slide in panel on slick container css. carousel works except new width not being calculated when space has been added. recalibrate once advance next slide i'd happen each time panel opened or closed. i've tried adding both margin , padding, neither makes difference. ideas? https://jsfiddle.net/mhigley/dpf7mlpl/ html: <button type="button" id="button"><i class="fa fa-arrow-right"></i></button> <aside> <h2>slide in panel</h2> <ol> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> </ol> </aside> <main role="main"> <div class="sections"> <section> <article> <h2>first slide</h2> <p>lorem ipsum d...
Read more

jsf - PrimeFaces Datatable - What is f:facet actually doing? -

-
looking through primefaces code in app noticed following line: <f:facet name="header">#{trainsearch.traincount} trains</f:facet> it looks overriding header, makes sense, can explain me in little more detail? what happening line of code? complete code listed below: <p:datatable id = "results" value = "#{trainsearch.trains}" var = "train" rendered="#{not empty trainsearch.trains}" styleclass = "train-search-table horizontal-border"> <f:facet name="header">#{trainsearch.traincount} trains</f:facet> <p:column headertext = "train id"> <p:panelgrid columns="1" styleclass = "train-id-grid" layout = "grid"> <h:outputtext styleclass = "train-id-label" value="#{train.traini}"/> <h:outputtex...
Read more

angular2 services - Angular 2 RC 4 Http post not firing -

-
i trying fire post request app , it's not working, tried multiple ways. don't see request being fired under network tab in chrome. not sure problem is, appreciated. import { injectable } '@angular/core'; import { http, headers } '@angular/http'; @injectable() export class userservice{ private _url="users"; constructor(private _http: http){ } checkifusernameexists(username){ return this._http.get(this._url+"/"+username) .map(response => response.json()) } createuser(user){ let headers = new headers(); headers.append('content-type', 'application/json') console.log(json.stringify(user)); return this._http.post(this._url, json.stringify(user),{headers: headers}) .map(response => response.json()) } } first need capture error , solve error: this._http.post(this._url, json.stringify(user),{headers: headers}) .subscribe( ...
Read more