access amazon S3 bucket from hadoop specifying SecretAccessKey from command line -

-

i trying access amazon s3 bucket using hdfs command. here command run:

$ hadoop fs -ls s3n://<accesskeyid>:<secretaccesskey>@<bucket-name>/tpt_files/ -ls: invalid hostname in uri s3n://<accesskeyid>:<secretaccesskey>@<bucket-name>/tpt_files usage: hadoop fs [generic options] -ls [-d] [-h] [-r] [<path> ...]

my secretaccesskey includes “/”. cause of such behavior?

in same time have aws cli installed in server , can access bucket using aws cli without issues (accesskeyid , secretaccesskey configured in .aws/credentials):

aws s3 ls s3:// <bucket-name>/tpt_files/

if there way how access amazon s3 bucket using hadoop command without specifying keys in core-site.xml? i’d prefer specify keys in command line.

any suggestions helpful.

the best practice run hadoop on instance created ec2 instance profile role, , s3 access specified policy of assigned role. keys no longer needed when using instance profile.
http://docs.aws.amazon.com/java-sdk/latest/developer-guide/credentials.html

you can launch amis instance profile role , cli , sdks use it. if code uses defaultawscredentialsproviderchain class, credentials can obtained through environment variables, system properties, or credential profiles file (as ec2 instance profile role).


Comments

Post a Comment

Popular posts from this blog

javascript - Slick Slider width recalculation -

-
i'm trying add slide in panel on slick container css. carousel works except new width not being calculated when space has been added. recalibrate once advance next slide i'd happen each time panel opened or closed. i've tried adding both margin , padding, neither makes difference. ideas? https://jsfiddle.net/mhigley/dpf7mlpl/ html: <button type="button" id="button"><i class="fa fa-arrow-right"></i></button> <aside> <h2>slide in panel</h2> <ol> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> </ol> </aside> <main role="main"> <div class="sections"> <section> <article> <h2>first slide</h2> <p>lorem ipsum d...
Read more

jsf - PrimeFaces Datatable - What is f:facet actually doing? -

-
looking through primefaces code in app noticed following line: <f:facet name="header">#{trainsearch.traincount} trains</f:facet> it looks overriding header, makes sense, can explain me in little more detail? what happening line of code? complete code listed below: <p:datatable id = "results" value = "#{trainsearch.trains}" var = "train" rendered="#{not empty trainsearch.trains}" styleclass = "train-search-table horizontal-border"> <f:facet name="header">#{trainsearch.traincount} trains</f:facet> <p:column headertext = "train id"> <p:panelgrid columns="1" styleclass = "train-id-grid" layout = "grid"> <h:outputtext styleclass = "train-id-label" value="#{train.traini}"/> <h:outputtex...
Read more

angular2 services - Angular 2 RC 4 Http post not firing -

-
i trying fire post request app , it's not working, tried multiple ways. don't see request being fired under network tab in chrome. not sure problem is, appreciated. import { injectable } '@angular/core'; import { http, headers } '@angular/http'; @injectable() export class userservice{ private _url="users"; constructor(private _http: http){ } checkifusernameexists(username){ return this._http.get(this._url+"/"+username) .map(response => response.json()) } createuser(user){ let headers = new headers(); headers.append('content-type', 'application/json') console.log(json.stringify(user)); return this._http.post(this._url, json.stringify(user),{headers: headers}) .map(response => response.json()) } } first need capture error , solve error: this._http.post(this._url, json.stringify(user),{headers: headers}) .subscribe( ...
Read more