hive - Combining data based on column in spark -

-

i have data in following format in hive table.

user |  purchase | time_of_purchase

i want data in 

user | list of purchases ordered time

how do in pyspark or hiveql?

i have tried using collect_list in hive not retain order correctly timestamp.

edit : adding sample data asked kartikkannapur. here sample data

94438fef-c503-4326-9562-230e78796f16 | bread | jul 7 20:48 94438fef-c503-4326-9562-230e78796f16 | shaving cream | july 10 14:20 a0dcbb3b-d1dd-43aa-91d7-e92f48cee0ad | milk | july 7 3:48 a0dcbb3b-d1dd-43aa-91d7-e92f48cee0ad | bread | july 7 3:49 a0dcbb3b-d1dd-43aa-91d7-e92f48cee0ad | lotion | july 7 15:30

the output want 

94438fef-c503-4326-9562-230e78796f16 | bread , shaving cream a0dcbb3b-d1dd-43aa-91d7-e92f48cee0ad | milk , bread , lotion

one way of doing is

first create hive context , read table rdd.

from pyspark import hivecontext purchaselist = hivecontext(sc).sql('from purchaselist select *')

then process rdd

from datetime import datetime dt purchaselist = purchaselist.map(lambda x:(x[0],[x[1],dt.strptime(x[2],"%b %d %h:%m")])) purchasebyuser = purchaselist.groupbykey() purchasebyuser = purchasebyuser.map(lambda x:(x[0],[y[0] y in sorted(x[1], key=lambda z:z[1])])) print(purchasebyuser.take(2))

output

[('94438fef-c503-4326-9562-230e78796f16', ['bread', 'shaving cream']), ('a0dcbb3b-d1dd-43aa-91d7-e92f48cee0ad', ['milk', 'bread', 'lotion'])]

save rdd new hive table

schema_rdd = hivecontext(sc).inferschema(purchasebyuser) schema_rdd.saveastable('purchasebyuser')

for reading , writing hive table see stackoverflow question , spark docs


Comments

Post a Comment

Popular posts from this blog

javascript - Slick Slider width recalculation -

-
i'm trying add slide in panel on slick container css. carousel works except new width not being calculated when space has been added. recalibrate once advance next slide i'd happen each time panel opened or closed. i've tried adding both margin , padding, neither makes difference. ideas? https://jsfiddle.net/mhigley/dpf7mlpl/ html: <button type="button" id="button"><i class="fa fa-arrow-right"></i></button> <aside> <h2>slide in panel</h2> <ol> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> </ol> </aside> <main role="main"> <div class="sections"> <section> <article> <h2>first slide</h2> <p>lorem ipsum d...
Read more

jsf - PrimeFaces Datatable - What is f:facet actually doing? -

-
looking through primefaces code in app noticed following line: <f:facet name="header">#{trainsearch.traincount} trains</f:facet> it looks overriding header, makes sense, can explain me in little more detail? what happening line of code? complete code listed below: <p:datatable id = "results" value = "#{trainsearch.trains}" var = "train" rendered="#{not empty trainsearch.trains}" styleclass = "train-search-table horizontal-border"> <f:facet name="header">#{trainsearch.traincount} trains</f:facet> <p:column headertext = "train id"> <p:panelgrid columns="1" styleclass = "train-id-grid" layout = "grid"> <h:outputtext styleclass = "train-id-label" value="#{train.traini}"/> <h:outputtex...
Read more

angular2 services - Angular 2 RC 4 Http post not firing -

-
i trying fire post request app , it's not working, tried multiple ways. don't see request being fired under network tab in chrome. not sure problem is, appreciated. import { injectable } '@angular/core'; import { http, headers } '@angular/http'; @injectable() export class userservice{ private _url="users"; constructor(private _http: http){ } checkifusernameexists(username){ return this._http.get(this._url+"/"+username) .map(response => response.json()) } createuser(user){ let headers = new headers(); headers.append('content-type', 'application/json') console.log(json.stringify(user)); return this._http.post(this._url, json.stringify(user),{headers: headers}) .map(response => response.json()) } } first need capture error , solve error: this._http.post(this._url, json.stringify(user),{headers: headers}) .subscribe( ...
Read more