pg_dump and slowness

I am trying to make a backup of a very big postgresql(around 165 gb) for two days now. and at last I find out my mistake. never take a dump at same disk. it eats a lot of IO and kill all services which depends to dumped database.

first try I am dumping database to same disk and after around 6 hours web server started to giving timeout and lovely sitescope mails 🙂 and I had to kill that process.

then I read a lot and started to dumping the database to another machine and it was smooth took around 4 hours to dump and no web server gave any timeout.

example commands

pg_dump -Fc dbname > db.backup

and I started restore like this

pg_restore -d dbname db.backup

before that I needed to recreate the db from psql. and this have not finished I must tell that pg_restore has -j parameter which gives more thread to read the dump file and you can give cpu number to -j which will work faster.

pgbouncer max problemi

postgre upgrade oldu 8.4.5 yaptık. önünede pgbouncer kondu. 4 web serverının ikisinde error connection limit falan dedi. pgbouncer.ini den max connection sayısı artırıldı reload edildi. sonuç 2 serverım hala bağlanamıo hibernate dende şuna benzer bi exception geliyor:

org.hibernate.jdbc.AbstractBatcher – exception clearing maxRows/queryTimeout

çözüm

pgbpuncer restart istiyor yoksa hala connection açılmaz oluyor benim web serverlarımda tepki veremez kalıyor şükür bitti 🙂

postgresql de 5 dakikadan uzun çalışan query leri durdurmanın yolu

geçen gün timu sağolsun postgrede çalışan query leri query den görmeyi öğrendim sonra düşünürken dedim ulan bunları bir bir cancel edioz iyi hoşta aslında bir kısım select o tabloya update yapılırken çok bekleyebiliyor. olay web olunca 5 dakikadan uzun süredir çalışan queryleri bulmalıyım ve selectse ver kill i gitsin demi dedim bi kaç kişide hata alsın 🙂 sonuç güzel oldu


select pg_cancel_backend(procpid) from (
SELECT * FROM pg_stat_activity
where current_query like '%select%'
and query_start < now() - interval '5 minute'
order by current_query

) a

pg_restore

I know its been a long time I have not write anything. because I have not learn or done any new things. I dont want to write regular meaningless blogs here. anyway today I tried to restore a pg_dump out file to another postgresql server.

that pg_restore looks nice but in first try it started print lines of the file that was strange. at the end I learned that if we give the name of the database it will restore data to there. otherwise it works like scanner for a dump file.

here is the command: pg_restore -d dbName backupFileName

after that it does not print anything to stdout just starts restoring the data. if anyone knows how to show which table is being restored in that time that would be nice. because now I am cleaning some useless data with pgadmin manually.

Postgresql group by day, week and month examples

at the end of any product there will be a reporting interfaces for counts. let say you build a advertisement site which gives people to publish their products on the site. they will want to see how many people visited their product in daily basis or weekly.

I used to do this in mysql like this

SELECT create_time as Date, count(id) as ‘Count’, FROM
views_of_product
group by date_format(create_time, ‘%d %m %Y’)
order by date_format(create_time, ‘%Y-%m-%d’) desc limit 7

this will nicely show last seven days views. but I needed to do same thing in postgresql. and like other days its not easily to find. I should check other report codes from project but no I allways research on google 🙂 anyway here is my code:

select to_char(created, ‘YYYY-MM-DD’) ,count(id)
from videos_capture
group by to_char(created, ‘YYYY-MM-DD’)
order by to_char(created, ‘YYYY-MM-DD’) desc limit 7

its not bad actually works like a charm but understanding this code is not that easy. after I found this I started to implement needs and see better and easy to understand solutions.

weekly count example. this will show new users weekly parts for last 3 months:

SELECT date_trunc(‘week’, created) AS “Week” , count(id) AS “New Users”
FROM users
WHERE created > now() – interval ‘3 months’
GROUP BY “Week”
ORDER BY “Week”

yearly example. this will show monthly 1 year users:

SELECT date_trunc(‘month’, created) AS “Month” , count(id) AS “New Users”
FROM users
WHERE created > now() – interval ‘1 year’
GROUP BY “Month”
ORDER BY “Month”

I must say understanding postgresql’s sql more easy then mysql.

psql ilk deneme

14 bin satır bir schema dump ı var elimde pgadmin ile windowsda aldım başka bir windows makinede postgresql e bunu koyup bi şekilde yeni table oluşturmam lazım. tabii pgadmini açtım o makinede ama klasik çalışmadı pg_restore falan denedim yemedi. efenim örnek:
"c:Program FilesPostgreSQL8.2binpsql.exe" -Upostgres azbuz < "c:Documents and SettingsopakdilDesktopdump.backup"

sevimli oluo bu open source db ler.

Postgresql 8.2 de xml

mascix:xml ile xsl kullancam db den xml gelmesi lazim bu postgre nazlandi bugün bi ayarmi lazim ona
selamtux bilmiyom ki
xml çalisabiliyomu onuda bilmiyom

mascix:
postgre yi bilmion
🙂 çalisioda bende çalismadi
bunun ayari nerden verilio
selamtux
http://www.postgresql.org/docs/8.2/static/datatype-xml.html

mascix:
onu bugün okudum geç
selamtux
http://www.throwingbeans.org/postgresql_and_xml.html

mascix:
bi ayar lazim sanirim select örnekleri çalismadi
selamtux
contrib/xml2 has a function called xml_is_well_formed() that can be used in a CHECK constraint to enforce that a field contains well-formed XML. It does not support validation against a specific XML schema. A server-side language with XML capabilities could be used to do schema-specific XML checks.
postgreSQL contrip paketi kurulumnu?
kurulu ise xml2 diye bisi olacak içinde onu bi aç bi içine bak
sonra onu sisteme kur
kurulum bilgisi vardir
genelde dosyayi çalistirman yeterli oluyo (sql içinde çalistiracan)
yada derlemen gerekebilir
demekki okumussun ama bos okumussun 😀

mascix:
he ölesine
😀
xml2 diye bisey nerde olucak
selamtux
/usr/share/pogtgresq altinda olabilir

mascix:
libxml falan hersey ok
aslinda xml çalismasi lazim
ama ayar yapilcak seyi bulamadim
normal conf dosyalarina baktim ama yok
selamtux
usr/share/postgresql/8.2/contrib/

mascix:
pgxml.sql(C:Program FilesPostgreSQL8.2sharecontrib)
böle bise var orda bunumu çalistirim
selamtux
evet

mascix:
baboli sana saygi duyuyom
🙂
SELECT
article_id, xpath_string(article_xml,’/beatles/@id’) AS beatle_id
FROM
t_articles
WHERE
xpath_bool(article_xml,’/beatles/beatle[@alive=”yes”]’);
bu query çalismiodu artik çalisir oldu
hemde sen unix anlation
ben windowsdan çözom olayi
ooohhh
😀
selamtux
😀
kolaymis ama demi?
postgresql güzel bisi

mascix:
aga postgre zor bisey
selamtux
bu arada o contrip içerisinde çok kullanisli seyler var

mascix:
ben bu contribin olayini bilmiodum
sabah aksama kadar okudum
selamtux
normalde posgtreSQL fulltext search desteklemiyo

mascix:
kullanicilara uygun sistemi kafada bitirdim
selamtux
ama mssela tsearch2.sql bunu kuruyon full text desteklemeye basliyo

mascix:
ama implementasyonu yapamadim bunda takildimdi
anladim bundan sonra burdan sql çalistircam 😀
aslinda süper bisey ha bu böle çat diye opsiyon algilio
hemde ariza çikarmio
selamtux
btree_gist.sql bu güzel bise bak

mascix:
bu süper bi özellik neden dersen php de bile bi sürü dll arizasi çikar bunda yok 😀
selamtux
earthdistance.sql
Create the user-defined type for N-dimensional boxes

mascix:
sexsi very sexsi
🙂
selamtux
😀
ok ben filme dönüyom
günün kahramani bir problemi daha çözmenin vermis oldugu rahatlikla film izlemektedir

mascix:
SELECT xmlelement(name foo, xmlattributes(current_date as bar), ‘cont’, ‘ent’);
bu çalismio bende
sende çalisiomu
yaw dur iki dakka saat 11
🙂
selamtux
xmlelement diye bir fonksiyon yok o dosyada

mascix:
nerde var o
selamtux
xpath_nodeset
bilmiyom ki
contrip içinde yok

mascix:
fark ettim
ama doc da var
http://developer.postgresql.org/pgdocs/postgres/functions-xml.html
selamtux
http://www.postgresql.org/docs/8.2/static/datatype-xml.html
PostgreSQL 8.3RC1 Documentation
sendeki 8.2
8.3’de çok deli seyler geliyo
buda onlardan biri 😀

mascix:
annesine atliyim
yapma yaw
selamtux
8.2 doclarini kurcala

mascix:
heyecanim kaldi içimde xml den xsl le bi sürü html yapaciydim ben
selamtux
http://archives.postgresql.org/pgsql-interfaces/2007-03/msg00012.php
o kurdugun dosyanin içine bak oradaki fonksiyonlari kullanabilirsin

mascix:
xpath_table
bu güzel duruo bi arýyým bakam
yok buda arama aleti
🙁 database update i zor iste