scala - Why is predicate pushdown not working? -

-

programm sketch

  • i create hivecontext hivecontext.
  • with context, create dataframe df jdbc relational table.
  • i register dataframe df via df.registertemptable("testtable").
  • i start hivethriftserver2 via hivethriftserver2.startwithcontext(hivecontext).

the testtable contains 1,000,000 entries, columns id (int) , name (varchar)

+-----+--------+ | id  |  name  | +-----+--------+ | 1   | hello  | | 2   | hello  | | 3   | hello  | | ... | ...    |

with beeline access sql endpoint (at port 10000) of hivethriftserver , perform query. e.g.

select * testtable id='3'

when inspect querylog of db sql statements executed see

/*sql #:1000000 t:657*/  select \"id\",\"name\" test;

so there happens no predicate pushdown , clause missing.

questions

this gives raise following questions:

  • why no predicate pushdown performed?
  • can changed not using registertemptable?
  • if so, how? or known restriction of hivethriftserver?

counterexample

if create dataframe df in spark sqlcontext , call 

df.filter( df("id") === 3).show()

i observe 

/*sql #:1*/select \"id\",\"name\" test id = 3;

as expected.


Comments

Post a Comment

Popular posts from this blog

javascript - Slick Slider width recalculation -

-
i'm trying add slide in panel on slick container css. carousel works except new width not being calculated when space has been added. recalibrate once advance next slide i'd happen each time panel opened or closed. i've tried adding both margin , padding, neither makes difference. ideas? https://jsfiddle.net/mhigley/dpf7mlpl/ html: <button type="button" id="button"><i class="fa fa-arrow-right"></i></button> <aside> <h2>slide in panel</h2> <ol> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> </ol> </aside> <main role="main"> <div class="sections"> <section> <article> <h2>first slide</h2> <p>lorem ipsum d...
Read more

jsf - PrimeFaces Datatable - What is f:facet actually doing? -

-
looking through primefaces code in app noticed following line: <f:facet name="header">#{trainsearch.traincount} trains</f:facet> it looks overriding header, makes sense, can explain me in little more detail? what happening line of code? complete code listed below: <p:datatable id = "results" value = "#{trainsearch.trains}" var = "train" rendered="#{not empty trainsearch.trains}" styleclass = "train-search-table horizontal-border"> <f:facet name="header">#{trainsearch.traincount} trains</f:facet> <p:column headertext = "train id"> <p:panelgrid columns="1" styleclass = "train-id-grid" layout = "grid"> <h:outputtext styleclass = "train-id-label" value="#{train.traini}"/> <h:outputtex...
Read more

javascript - Error handling and status codes with knex in koa -

-
i'm experimenting koa , knex build restful web service, , while got basics working i'm stuck @ handling errors. i'm using koa-knex-middleware wraps knex in generator. i have following code defines happens , post requests particular resource: var koa = require('koa'); var koabody = require('koa-body')(); var router = require('koa-router')(); var app = koa(); var knex = require('./koa-knex'); app.use(knex({ client: 'sqlite3', connection: { filename: "./src/server/devdb.sqlite" } })); router .get('/equipment', function *(next){ this.body = yield this.knex('equipment'); }) .post('/equipment', koabody, function *(next){ this.body = yield this.knex('equipment').insert(this.request.body) }); app.use(router.routes()).use(router.allowedmethods()); app.listen(4000); this works in general, can't manage change http status codes. examp...
Read more