{"id":1307,"date":"2013-09-08T13:26:40","date_gmt":"2013-09-08T20:26:40","guid":{"rendered":"http:\/\/blog.light42.com\/wordpress\/?p=1307"},"modified":"2013-09-18T17:25:51","modified_gmt":"2013-09-19T00:25:51","slug":"data-characterization-and-the-live","status":"publish","type":"post","link":"http:\/\/blog.light42.com\/wordpress\/?p=1307","title":{"rendered":"Data Characterization and the Live"},"content":{"rendered":"

People may already know about the OSGeo Live project<\/strong><\/a>. Its a great base as a VM since a)<\/strong> it is stable and very well tested, and b)<\/strong> it has much software installed<\/a>, but in a way that is transparent through install scripts<\/a>, so customization is as straightforward as it gets..<\/p>\n

\"some_7_pkgs_image\"<\/p>\n

I was faced with a bit of “data mining” prep, so I pulled out the OSGeo Live<\/strong> and set up a VM. I use VirtualBox and its GUI on my desktop i5 quite a bit. I made a new VM, gave it two cores and 1G of RAM, bridged networking instead of NAT, and set the CD storage to point to a nice new Live .iso<\/a>.<\/p>\n

There are quite a few PostgreSQL 9.1 pl-language extensions as packages for Ubuntu<\/a>, but not much for 9.2 apparently, (precise, raring and the XUbuntu precise Live all use 9.1), and of course little for the 9.3 as of yet.<\/p>\n

Since a rockin’ R<\/strong> environment is already pre-built on the OSGeo Live<\/a>, I didnt have to think about that.. I installed this package<\/a> and used my formula from a previous post<\/a>. Dont forget to set Postgres to use more RAM <\/em>(this is the generic formula I use to set System V memory limits on Linux in \/etc\/sysctl.conf<\/code> )<\/p>\n


\n#!\/bin\/bash
\nmem_bytes=`awk '\/MemTotal:\/ { printf \"%0.f\",$2 * 1024}' \/proc\/meminfo`
\nmem_max=`expr $mem_bytes \/ 2`
\npage_size=`getconf PAGE_SIZE`
\nshmall=`expr $mem_bytes \/ $page_size`
\necho \\# Maximum shared segment size in bytes
\necho kernel.shmmax = $mem_max
\necho \\# Maximum number of shared memory segments in pages
\necho kernel.shmall = $shmall
\n<\/code><\/p>\n

\n


\n## \/etc\/postgresql\/9.1\/main\/postgresql.conf<\/p>\n

# - Connection Settings -
\n .....<\/p>\n

max_connections = 24 # (change requires restart)<\/p>\n

.....<\/p>\n

# - Memory -<\/p>\n

shared_buffers = 240MB # min 128kB
\n # (change requires restart)
\ntemp_buffers = 16MB # min 800kB<\/p>\n

work_mem = 160MB # min 64kB
\nmaintenance_work_mem = 128MB # min 1MB
\nmax_stack_depth = 3MB # min 100kB<\/p>\n

<\/code><\/p>\n

\n

But, after waiting 160 seconds <\/em>for some descriptive stats on a column of 900,000 floats, a dozen times.. I was searching around for better performance..<\/p>\n


\nSELECT max(a.res),avg(a.res),median(a.res),min(a.res),sd(a.res)
\nFROM
\n( SELECT (pop_hs_diploma_pct *100)::integer as res
\n  FROM my_excellent_adventure_table
\n  WHERE pop_hs_diploma_pct > 0.01 ) a;<\/p>\n

\r\n max |         avg         | median | min |        sd        \r\n-----+---------------------+--------+-----+------------------\r\n  63 | 12.5609578591567519 |     12 |   1 | 6.88020337480763\r\n<\/pre>\n
<\/code><\/p>\n
SnowMan<\/strong><\/a> pointed out a short rant from a few years ago<\/a>, but actually, pl-R is still slow!<\/p>\n
Then I found this post<\/a> and the corresponding project<\/a>. I compiled, built and it worked just fine. Thx InterWeb!<\/strong><\/p>\n

\n--
\nSELECT
\n  min(pop_age65_up::integer),
\n  quantile( pop_age65_up, ARRAY[0.25, 0.5, 0.75]),
\n  avg(pop_age65_up::integer),
\n  max(pop_age65_up::integer)
\nFROM
\n  my_excellent_adventure_table
\nWHERE pop_age65_up > 1;<\/p>\n
\r\n min |      quantile          |   avg  | max  \r\n-----+------------------------+--------+------\r\n   1 | {2.4556,4.7930,8.5264} | 8.6491 | 3878\r\n<\/pre>\n
<\/code><\/p>\n","protected":false},"excerpt":{"rendered":"
People may already know about the OSGeo Live project. Its a great base as a VM since a) it is stable and very well tested, and b) it has much software installed, but in a way that is transparent through install scripts, so customization is as straightforward as it gets.. I was faced with a […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11,12,5],"tags":[],"_links":{"self":[{"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1307"}],"collection":[{"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1307"}],"version-history":[{"count":28,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1307\/revisions"}],"predecessor-version":[{"id":1337,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1307\/revisions\/1337"}],"wp:attachment":[{"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1307"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1307"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1307"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}