sql - Spark - searching spatial data - partition pruning -

-

i have large quantity of geotagged rows - hundreds of millions - need query using spark sql doing distance calculations points. sql works ok using basic triginometry , haversine distance function. result set returned latitude between +/- meters latitude point, , same longitude; ordered distance desc, , top-n find closest points. far good. data global, storing points in memory inefficient.

my questions:

  1. how benefit realize using partitioning pruning partitioning data latitude ranges, longitude subranges? reduce search area 1-3 latitude partitions, , < 10 longitude subpartitions. lot less data; dont know spark sql optimizer can prune partitions , subpartitions. unclear if partition pruning on cached rdd particularly beneficial. there no join involved.

  2. i partition using parquet files, , subsequently read in parquet partitions needed, instead of data. there other file format should use has partitioning capability?

you'll benefit partition pruning when initial read. spark sql's optimizer pruning if data in columnar data format (like parquet) , you're not reading columns, other sql db do. if filtering of data prior caching, you'll querying , persisting smaller subset of data anyways. optimizer take queries pass in , best read minimal amount of data disk.

orc files file format might want into. orc smaller in size when sitting on hdfs, slower when comes reading data off of disk.

finally, when caching dataframe, spark-sql use columnar compression format persisting data, should able put more in memory think this. allows efficient querying, don't have read data won't show in results.


Comments

Post a Comment

Popular posts from this blog

javascript - Slick Slider width recalculation -

-
i'm trying add slide in panel on slick container css. carousel works except new width not being calculated when space has been added. recalibrate once advance next slide i'd happen each time panel opened or closed. i've tried adding both margin , padding, neither makes difference. ideas? https://jsfiddle.net/mhigley/dpf7mlpl/ html: <button type="button" id="button"><i class="fa fa-arrow-right"></i></button> <aside> <h2>slide in panel</h2> <ol> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> </ol> </aside> <main role="main"> <div class="sections"> <section> <article> <h2>first slide</h2> <p>lorem ipsum d...
Read more

jsf - PrimeFaces Datatable - What is f:facet actually doing? -

-
looking through primefaces code in app noticed following line: <f:facet name="header">#{trainsearch.traincount} trains</f:facet> it looks overriding header, makes sense, can explain me in little more detail? what happening line of code? complete code listed below: <p:datatable id = "results" value = "#{trainsearch.trains}" var = "train" rendered="#{not empty trainsearch.trains}" styleclass = "train-search-table horizontal-border"> <f:facet name="header">#{trainsearch.traincount} trains</f:facet> <p:column headertext = "train id"> <p:panelgrid columns="1" styleclass = "train-id-grid" layout = "grid"> <h:outputtext styleclass = "train-id-label" value="#{train.traini}"/> <h:outputtex...
Read more

angular2 services - Angular 2 RC 4 Http post not firing -

-
i trying fire post request app , it's not working, tried multiple ways. don't see request being fired under network tab in chrome. not sure problem is, appreciated. import { injectable } '@angular/core'; import { http, headers } '@angular/http'; @injectable() export class userservice{ private _url="users"; constructor(private _http: http){ } checkifusernameexists(username){ return this._http.get(this._url+"/"+username) .map(response => response.json()) } createuser(user){ let headers = new headers(); headers.append('content-type', 'application/json') console.log(json.stringify(user)); return this._http.post(this._url, json.stringify(user),{headers: headers}) .map(response => response.json()) } } first need capture error , solve error: this._http.post(this._url, json.stringify(user),{headers: headers}) .subscribe( ...
Read more