html - Python not progressing a list of links -

-

so, need more detailed data have dig bit deeper in html code of website. wrote script returns me list of specific links detail pages, can't bring python search each link of list me, stops @ first one. doing wrong?

 beautifulsoup import beautifulsoup  import urllib2  lxml import html  import requests   #open site  html_page = urllib2.urlopen("http://www.sitetoscrape.ch/somesite.aspx")  #inform beautifulsoup soup = beautifulsoup(html_page)  #search specific links link in soup.findall('a', href=re.compile('/d/part/of/thelink/ineed.aspx')):     #print found links     print link.get('href')     #complete links     complete_links = 'http://www.sitetoscrape.ch' + link.get('href')     #print complete links     print complete_links # #everything works fine point #  page = requests.get(complete_links) tree = html.fromstring(page.text)  #details name = tree.xpath('//dl[@class="services"]')  in name:     print i.text_content()

also: tutorial can recommend me learn how put output in file , clean up, give variable names, etc?

i think want list of links in complete_links instead of single link. @pynchia , @lemonhead said you're overwritting complete_links every iteration of first loop.

you need 2 changes:

  • append links list , use loop , scrap each link

    # [...] same code here  links_list = [] link in soup.findall('a', href=re.compile('/d/part/of/thelink/ineed.aspx')):     print link.get('href')     complete_links = 'http://www.sitetoscrape.ch' + link.get('href')     print complete_links     link_list.append(complete_links)  # append new link list

  • scrap each accumulated link in loop

    for link in link_list:     page = requests.get(link)     tree = html.fromstring(page.text)      #details     name = tree.xpath('//dl[@class="services"]')      in name:         print i.text_content()

ps: recommend scrapy framework tasks that.


Comments

Post a Comment

Popular posts from this blog

javascript - Slick Slider width recalculation -

-
i'm trying add slide in panel on slick container css. carousel works except new width not being calculated when space has been added. recalibrate once advance next slide i'd happen each time panel opened or closed. i've tried adding both margin , padding, neither makes difference. ideas? https://jsfiddle.net/mhigley/dpf7mlpl/ html: <button type="button" id="button"><i class="fa fa-arrow-right"></i></button> <aside> <h2>slide in panel</h2> <ol> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> </ol> </aside> <main role="main"> <div class="sections"> <section> <article> <h2>first slide</h2> <p>lorem ipsum d...
Read more

jsf - PrimeFaces Datatable - What is f:facet actually doing? -

-
looking through primefaces code in app noticed following line: <f:facet name="header">#{trainsearch.traincount} trains</f:facet> it looks overriding header, makes sense, can explain me in little more detail? what happening line of code? complete code listed below: <p:datatable id = "results" value = "#{trainsearch.trains}" var = "train" rendered="#{not empty trainsearch.trains}" styleclass = "train-search-table horizontal-border"> <f:facet name="header">#{trainsearch.traincount} trains</f:facet> <p:column headertext = "train id"> <p:panelgrid columns="1" styleclass = "train-id-grid" layout = "grid"> <h:outputtext styleclass = "train-id-label" value="#{train.traini}"/> <h:outputtex...
Read more

angular2 services - Angular 2 RC 4 Http post not firing -

-
i trying fire post request app , it's not working, tried multiple ways. don't see request being fired under network tab in chrome. not sure problem is, appreciated. import { injectable } '@angular/core'; import { http, headers } '@angular/http'; @injectable() export class userservice{ private _url="users"; constructor(private _http: http){ } checkifusernameexists(username){ return this._http.get(this._url+"/"+username) .map(response => response.json()) } createuser(user){ let headers = new headers(); headers.append('content-type', 'application/json') console.log(json.stringify(user)); return this._http.post(this._url, json.stringify(user),{headers: headers}) .map(response => response.json()) } } first need capture error , solve error: this._http.post(this._url, json.stringify(user),{headers: headers}) .subscribe( ...
Read more