python - multiprocessing module and distinct psycopg2 connections -

-

i puzzled behavior of multiprocessing code using psycopg2 make queries in parallel postgres db.

essentially, making same query (with different params) various partitions of larger table. using multiprocessing.pool fork off separate query.

my multiprocessing call looks this:

pool = pool(processes=num_procs) results=pool.map(run_sql, params_list)

my run_sql code looks this:

def run_sql(zip2):     conn = get_connection()     curs = conn.cursor()     print "conn: %s curs:%s pid=%s" % (id(conn), id(curs), os.getpid())     ...     curs.execute(qry)     records = curs.fetchall()  def get_connection()     ...     conn = psycopg2.connect(user=db_user, host=db_host,                           dbname=db_name, password=db_pwd)      return conn

so expectation each process separate db connection via call get_connection() , print id(conn) display distinct value. however, doesn't seem case , @ loss explain it. print id(curs) same. print os.getpid() shows difference. somehow use same connection each forked process ?

conn: 4614554592 curs:4605160432 pid=46802 conn: 4614554592 curs:4605160432 pid=46808 conn: 4614554592 curs:4605160432 pid=46810 conn: 4614554592 curs:4605160432 pid=46784 conn: 4614554592 curs:4605160432 pid=46811

i think i've figured out. answer lies in fact multiprocessing in python shared-nothing entire memory space copied, functions , all. hence each process, though pid different, memory spaces copies of each other , address of connection within memory space ends being same. same reason why declaring global connection pool did useless, each process ended own connection pool 1 connection active @ time.


Comments

Post a Comment

Popular posts from this blog

javascript - Slick Slider width recalculation -

-
i'm trying add slide in panel on slick container css. carousel works except new width not being calculated when space has been added. recalibrate once advance next slide i'd happen each time panel opened or closed. i've tried adding both margin , padding, neither makes difference. ideas? https://jsfiddle.net/mhigley/dpf7mlpl/ html: <button type="button" id="button"><i class="fa fa-arrow-right"></i></button> <aside> <h2>slide in panel</h2> <ol> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> <li>list item</li> </ol> </aside> <main role="main"> <div class="sections"> <section> <article> <h2>first slide</h2> <p>lorem ipsum d...
Read more

jsf - PrimeFaces Datatable - What is f:facet actually doing? -

-
looking through primefaces code in app noticed following line: <f:facet name="header">#{trainsearch.traincount} trains</f:facet> it looks overriding header, makes sense, can explain me in little more detail? what happening line of code? complete code listed below: <p:datatable id = "results" value = "#{trainsearch.trains}" var = "train" rendered="#{not empty trainsearch.trains}" styleclass = "train-search-table horizontal-border"> <f:facet name="header">#{trainsearch.traincount} trains</f:facet> <p:column headertext = "train id"> <p:panelgrid columns="1" styleclass = "train-id-grid" layout = "grid"> <h:outputtext styleclass = "train-id-label" value="#{train.traini}"/> <h:outputtex...
Read more

angular2 services - Angular 2 RC 4 Http post not firing -

-
i trying fire post request app , it's not working, tried multiple ways. don't see request being fired under network tab in chrome. not sure problem is, appreciated. import { injectable } '@angular/core'; import { http, headers } '@angular/http'; @injectable() export class userservice{ private _url="users"; constructor(private _http: http){ } checkifusernameexists(username){ return this._http.get(this._url+"/"+username) .map(response => response.json()) } createuser(user){ let headers = new headers(); headers.append('content-type', 'application/json') console.log(json.stringify(user)); return this._http.post(this._url, json.stringify(user),{headers: headers}) .map(response => response.json()) } } first need capture error , solve error: this._http.post(this._url, json.stringify(user),{headers: headers}) .subscribe( ...
Read more