{"id":3322,"date":"2017-03-20T08:08:29","date_gmt":"2017-03-20T15:08:29","guid":{"rendered":"http:\/\/blog.light42.com\/wordpress\/?page_id=3322"},"modified":"2017-03-20T08:08:29","modified_gmt":"2017-03-20T15:08:29","slug":"15feb-archive","status":"publish","type":"page","link":"http:\/\/blog.light42.com\/wordpress\/?page_id=3322","title":{"rendered":"15Feb Archive"},"content":{"rendered":"

15 Feb 17
\nAutomated Ingestion Setup — California AB 802 Support
\n========================================================<\/p>\n

Primary Components of Image Recognition<\/strong>
\n * Automated data loading from primary sources
\n * Automated Retrieval of specific content
\n * Engine(s) applied to content
\n * Review and Feedback<\/p>\n

  previously, we focused on automation of vector sources (auth<\/code> and osm<\/code>),
\nthis time, focus on new digital imagery content, automated\/scriptable viewer,
\nand first steps in recognition. Also note content in \/ECN<\/a><\/p>\n

DOQQ California 2014<\/strong><\/p>\n

What is a DOQQ ? -LINK-<\/a> CA 2014 – statewide, 1 meter resolution, 4-band (true color and color-infrared) GeoTIFF tiles; County-composites typically lack the Infrared (IR) band.<\/p>\n

2016 is 0.6 meter resolution. 11TB of hard disks delivered to:
\nBruce Nielsen<\/strong> CA State GIS Coordinator
\nBruce.Nielsen@ca.usda.gov<\/p>\n

\"\"<\/a>

2014 DOQQ added in Northern California<\/p><\/div>\n

\r\n## A prioritized queue for DOQQ Downloading\r\n...\r\nPriority 1 => commercial; 2 => MF; 3 => both; 0 => neither\r\n  subtract whole counties already done\r\n...\r\nauth_buildings=# select priority, count(*) from doqq_processing group by \r\npriority order by priority;\r\n  priority | count\r\n----------+-------\r\n         0 |  5683\r\n         1 |   957\r\n         2 |   415\r\n         3 |  4007\r\n\r\n\r\n doqqid | priority | fetched | processed \r\n--------+----------+---------+-----------\r\n   6684 |        0 | f       | f\r\n   1021 |        0 | f       | f\r\n   4944 |        3 | f       | f\r\n   6490 |        1 | f       | f\r\n   3467 |        3 | f       | f\r\n   4380 |        3 | t       | f\r\n   ...\r\n\r\nAs of 13Feb17, a little over a 1\/4 of the priority 3 DOQQs have been fetched and \r\nprocessed and about 22% of all the priority DOQQs. Fetching is the \r\nlimiting factor with 4,184 left to fetch.\r\n\r\nEst. 4,184 * 5 min \/ 60 min\/hr = 349 hrs, or 15 days to finish fetching.\r\nupdate 15Feb17: Year 2014 DOQQs processed, compressed and IR extracted:  1484\r\n<\/pre>\n
NAIP Imagery Misc<\/strong>
\n   http:\/\/ucanr.edu\/blogs\/blogcore\/
\n   http:\/\/www.atlas.ca.gov\/download.html<\/p>\n
 
\n2D Building Extraction<\/strong> other<\/em>
\n   GEM User Guide -LINK-<\/a> -HOME-<\/a>
\n   IEEE Explorer -LINK-<\/a>
\n   NYPL-Spacetime FOSS vectorizer -LINK-<\/a>
\n   NYPL-Spacetime FOSS building browser-LINK-<\/a>
\n   Facebook ML Object Recognition -LINK-<\/a><\/p>\n
\n
\r\n  osmb\/?zoom=18&lat=40.90375&lon=-124.08229&layers=0000BTFFFFFF\r\n\r\n    Too much detail in one scene:<\/em> \r\n  Thought Experiment: \r\n    If you were a computer program, which parts would you identify as \"buildings\" ?  \r\n    Is the color version better ?\r\n\r\n \"\"<\/a>\"\"<\/a>\r\n\r\n \r\n<\/pre>\n
\r\n##--- BIS Session record - 14Feb17\r\n\r\n##- confirm operation\r\ndbb@i7d:~\/CEC_i7d\/bis-2\/bisapi-hamlin\/bis-workd$ source \/home\/dbb\/CEC_i7d\/bis-2\/bisenv\/bin\/activate bisenv\r\n(bisenv) dbb@i7d:~\/CEC_i7d\/bis-2\/bisapi-hamlin\/bis-workd$ python segment.py --test\r\n\r\n##-- pick a sample image\r\n##       tiffinfo    generic TIFF details\r\n##       rio info    short description \r\n##\r\n(bisenv) dbb@i7d:~\/CEC_i7d\/bis-2\/bisapi-hamlin\/bis-workd$ tiffinfo  \/wd4m\/ca_naip_2014_quads\/final-ir\/m_4012408_sw_10_1_20140607.tif \r\n...\r\n  DocumentName: Arcata North SW 4012408\r\n  DateTime: 2014:09:18 10:08:51\r\n...\r\n\r\n##--\r\n(bisenv) dbb@i7d:~\/CEC_i7d\/bis-2\/bisapi-hamlin\/bis-workd$ rio info  \/wd4m\/ca_naip_2014_quads\/final-ir\/m_4012408_sw_10_1_20140607.tif \r\n{\r\n   \"res\" : [ 1.02267103930246e-05, 1.02267103930246e-05 ],\r\n   \"shape\" : [ 6764, 7021 ],\r\n   \"lnglat\" : [\r\n      -124.093772354795,\r\n      40.9062477860959\r\n   ],\r\n   \"dtype\" : \"uint8\",\r\n   \"driver\" : \"GTiff\",\r\n   \"blockysize\" : 256,\r\n   \"bounds\" : [ -124.129673, 40.87166105, -124.0578714, 40.94083452  ],\r\n   \"count\" : 2,\r\n   \"crs\" : \"EPSG:4326\",\r\n   \"width\" : 7021,\r\n   \"interleave\" : \"pixel\",\r\n   \"nodata\" : null,\r\n   \"height\" : 6764,\r\n   \"transform\" : [ 1.02267103930246e-05, 0, -124.12967322163, 0, -1.02267103930246e-05, 40.9408345206451 ],\r\n   \"tiled\" : true,\r\n   \"colorinterp\" : [ \"gray\", \"alpha\" ],\r\n   \"blockxsize\" : 256\r\n}\r\n\r\n##--\r\n## run file - Too Large, crash\r\n##  hand extract smaller sample\r\n## run two segmentations at very low threshold\r\nTFILE=\/wd4m\/Arcata_Ex0.png\r\n(bisenv) dbb@i7d:~\/CEC_i7d\/bis-2\/bisapi-hamlin\/bis-workd$ python segment.py ${TFILE} -t [20,50] --tile True --xt 10\r\n thresh 1\r\n ...\r\n\r\n##--\r\n## more samples\r\npython segment.py ${TFILE}   -t [16,32,64,96] --tile True --xt 10\r\npython segment.py ${TFILE}   -t [108,124,142] --tile True --xt 10\r\npython segment.py ${TFILE}   -t [150,180,210,240] --tile True --xt 10\r\n\r\n##-- End Session\r\n\r\n<\/pre>\n
\n
OSM Buildings Load Chain<\/strong><\/p>\n
\r\n    fetch-osm-ca-latest\r\n       adds OSM California into OSM hot folder\r\n\r\n    reload-osm-buildings\r\n       DROP osm_buildings database\r\n       C++ tool filters out 2D buildings by tag\r\n\r\nRELOAD SUMMARY:\r\nAnalyzing building data ...  Wed Feb 15 12:18:35 PST 2017\r\nselect key, count(*) as cnt from (select (each(tags)).* from areas) as foo group by key order by cnt desc;\r\n                  key                  |   cnt   \r\n---------------------------------------+---------\r\n building                              | 4442239\r\n lacounty:bld_id                       | 2924011\r\n lacounty:ain                          | 2922837\r\n height                                | 2898947\r\n ele                                   | 2890401\r\n start_date                            | 2795684\r\n building:units                        | 2680758\r\n ...\r\n\r\n$psql osm_buildings -f osm_bldg_pt_count0.sql \r\n\r\n zone |  cnt  | zone |  cnt  \r\n------+-------+------+-------\r\n PZ_1 | 17354 | PZ_2 | 12207\r\n(1 row)\r\n\r\n zone |  cnt   | zone |  cnt  \r\n------+--------+------+-------\r\n PZ_3 | 739953 | PZ_4 | 39836\r\n(1 row)\r\n\r\n zone |  cnt   | zone |  cnt   \r\n------+--------+------+--------\r\n PZ_5 | 303110 | PZ_6 | 204510\r\n(1 row)\r\n\r\n zone |   cnt   | zone |  cnt  \r\n------+---------+------+-------\r\n PZ_7 | 3085052 | PZ_8 | 39997\r\n\r\n<\/pre>\n
 
\nOSM Misc<\/strong>
\n  https:\/\/tile.openstreetmap.org\/cgi-bin\/debug
\n  https:\/\/taginfo.openstreetmap.org\/keys\/building#wiki<\/p>\n
01 Feb 17
\nAutomated Ingestion Setup — California AB 802 Support
\n========================================================<\/p>\n
Data Provenance<\/strong> v0.01 -LINK-<\/a><\/p>\n
Tripartite Solution <\/strong>
\nThree parts to iterative building definition: building outlines from authoritative sources auth_buildings<\/code>; building outlines from Openstreetmap osm_buildings<\/code>; machine-assisted recognition ma_buildings<\/code>. <\/p>\n
Code-driven workflow components work together to form the basis of this “human in-in-the-loop<\/em>” discovery and definition environment. Think of it as a “power assist” for a person, instead of a fully-autonomous discovery agent. Inference, association and record-keeping are provided as a service to the human operator. A key element is some rigor to a reproducible workflow. Two guiding examples are “Reproducible Scientific Workflows for Data Science<\/em>” -LINK-<\/a> and the German Spatial Data Infrastructure (GD-SDI<\/strong>) ingestion system -LINK-<\/a> -WIKIPEDIA-<\/a>. These are not “the most advanced systems” available, but rather a “good fit” to the nature of this project, the actual resources available in the required time, and a good fit between cutting-edge tools and tools that are well-understood and time-tested. Especially in high-tech, there is benefit to a stable tool chain.<\/p>\n
Let’s look at the first two legs of this three-legged data system today:<\/p>\n
OSM_Buildings<\/strong><\/p>\n
\r\n\r\n  bin\/fetch-osm-ca-latest\r\n\r\n 721M Jan 27 14:43    california-170128.osm.pbf\r\n\r\n  bin\/reload-osm-buildings\r\n  Usage: reload-osm-buildings -go [-f]\r\n    -go  go ahead and do it\r\n    -f   fetch new osm data\r\n\r\n#--\r\ndbb@i7d:~\/CEC_i7d\/Code_Misc_repo$ psql osm_buildings -f osm_bldg_pt_count0.sql\r\n\r\n zone |  cnt  | zone |  cnt  \r\n------+-------+------+-------\r\n PZ_1 | 16347 | PZ_2 | 12140\r\n\r\n zone |  cnt   | zone |  cnt  \r\n------+--------+------+-------\r\n PZ_3 | 668975 | PZ_4 | 39829\r\n\r\n zone |  cnt   | zone |  cnt   \r\n------+--------+------+--------\r\n PZ_5 | 302767 | PZ_6 | 204249\r\n\r\n zone |   cnt   | zone |  cnt  \r\n------+---------+------+-------\r\n PZ_7 | 2240489 | PZ_8 | 39908\r\n<\/pre>\n
\r\n        bin\/osm_building_stats tool; stats on osm_buildings; 31Jan2017\r\n\r\n pz_id |       pz_name       | fips |   county_name   | osm_bldgs_pts_count \r\n-------+---------------------+------+-----------------+---------------------\r\n     8 | san_diego           | 073  | San Diego       |               39908\r\n     7 | scag                | 025  | Imperial        |                 648\r\n     7 | scag                | 037  | Los Angeles     |             2154919\r\n     7 | scag                | 059  | Orange          |               36089\r\n     7 | scag                | 065  | Riverside       |               27398\r\n     7 | scag                | 071  | San Bernardino  |               18728\r\n     7 | scag                | 111  | Ventura         |                2707\r\n     6 | central_coast       | 053  | Monterey        |                6359\r\n     6 | central_coast       | 069  | San Benito      |               23822\r\n     6 | central_coast       | 079  | San Luis Obispo |              152462\r\n     6 | central_coast       | 083  | Santa Barbara   |               14433\r\n     6 | central_coast       | 087  | Santa Cruz      |                7173\r\n     5 | central_valley      | 019  | Fresno          |                9208\r\n     5 | central_valley      | 029  | Kern            |              189405\r\n     5 | central_valley      | 031  | Kings           |                 366\r\n     5 | central_valley      | 039  | Madera          |                 308\r\n     5 | central_valley      | 047  | Merced          |               23432\r\n     5 | central_valley      | 077  | San Joaquin     |               74145\r\n     5 | central_valley      | 099  | Stanislaus      |                3904\r\n     4 | sacog               | 017  | El Dorado       |                5407\r\n     4 | sacog               | 061  | Placer          |                5934\r\n     4 | sacog               | 067  | Sacramento      |               15399\r\n     4 | sacog               | 101  | Sutter          |                 109\r\n     4 | sacog               | 113  | Yolo            |               12802\r\n     4 | sacog               | 115  | Yuba            |                 178\r\n     3 | bay_area            | 001  | Alameda         |               65732\r\n     3 | bay_area            | 013  | Contra Costa    |               11140\r\n     3 | bay_area            | 041  | Marin           |                2942\r\n     3 | bay_area            | 055  | Napa            |                3493\r\n     3 | bay_area            | 075  | San Francisco   |              159723\r\n     3 | bay_area            | 081  | San Mateo       |              215838\r\n     3 | bay_area            | 085  | Santa Clara     |              204109\r\n     3 | bay_area            | 095  | Solano          |                2229\r\n     3 | bay_area            | 097  | Sonoma          |                3769\r\n     2 | sierra              | 003  | Alpine          |                1367\r\n     2 | sierra              | 005  | Amador          |                 329\r\n     2 | sierra              | 009  | Calaveras       |                 223\r\n     2 | sierra              | 027  | Inyo            |                 351\r\n     2 | sierra              | 043  | Mariposa        |                1763\r\n     2 | sierra              | 051  | Mono            |                9266\r\n     2 | sierra              | 057  | Nevada          |                 843\r\n     2 | sierra              | 091  | Sierra          |                 611\r\n     2 | sierra              | 107  | Tulare          |                1999\r\n     2 | sierra              | 109  | Tuolumne        |                 208\r\n     1 | northern_california | 007  | Butte           |                1328\r\n     1 | northern_california | 011  | Colusa          |                  43\r\n     1 | northern_california | 015  | Del Norte       |                  59\r\n     1 | northern_california | 021  | Glenn           |                 147\r\n     1 | northern_california | 023  | Humboldt        |                1599\r\n     1 | northern_california | 033  | Lake            |                5668\r\n     1 | northern_california | 035  | Lassen          |                 142\r\n     1 | northern_california | 045  | Mendocino       |                1612\r\n     1 | northern_california | 049  | Modoc           |                 175\r\n     1 | northern_california | 063  | Plumas          |                  67\r\n     1 | northern_california | 089  | Shasta          |                2348\r\n     1 | northern_california | 093  | Siskiyou        |                 264\r\n     1 | northern_california | 103  | Tehama          |                  63\r\n     1 | northern_california | 105  | Trinity         |                  11\r\n\r\n<\/pre>\n
Auth_Buildings<\/strong><\/p>\n
\r\n    bin\/create-auth-buildings\r\n    bin\/add-auth-building-layer\r\n\r\n--\r\npsql (9.6.1)\r\nauth_buildings=# \\dt *.*\r\n\r\n       Schema       |              Name              | Type  |  Owner   \r\n--------------------+--------------------------------+-------+----------\r\n ambag              | bldgs                          | table | dbb\r\n bkrsfld            | bldgs                          | table | dbb\r\n census_p           | pz_region_defs                 | table | dbb\r\n census_p           | tl_2016_06_bg                  | table | dbb\r\n census_p           | tl_2016_06_cousub              | table | dbb\r\n census_p           | tl_2016_uac10_ca               | table | dbb\r\n census_p           | tl_2016_us_county              | table | dbb\r\n la14               | la_bldgs14_del_pt              | table | dbb\r\n la14               | la_bldgs14_invalid             | table | dbb\r\n la14               | la_bldgs14_pt                  | table | dbb\r\n la14               | lariac2_buildings_deleted_2014 | table | dbb\r\n la14               | lariac4_buildings_2014         | table | dbb\r\n marin0             | bldgs                          | table | dbb\r\n newportb           | bldgs                          | table | dbb\r\n newportb           | bldgs_orig                     | table | dbb\r\n petaluma           | bldgs                          | table | dbb\r\n roseville          | bldgs                          | table | dbb\r\n sacramento0        | bldgs                          | table | dbb\r\n sangis             | bldgs                          | table | dbb\r\n santa_cruz         | bldgs                          | table | dbb\r\n santacruz0         | bldgs                          | table | dbb\r\n sf0                | bldgs                          | table | dbb\r\n solano             | bldgs                          | table | dbb\r\n solano0            | bldgs                          | table | dbb\r\n\r\n<\/pre>\n
Machine-Assisted Buildings<\/strong>
\n  sample session -LINK-<\/a>  -PAPER-<\/a><\/p>\n
\n
Mapserver<\/strong> 7.04 Install<\/p>\n
  – many details to install, not for beginners
\n  – overview of the software –LINK-<\/a>
\n  – database object IDs are visible at higher zoom levels
\n  – some interesting links:
\n    San Mateo County, Menlo Park  mapserv -LINK-<\/a><\/p>\n
Using the layers picker, on can find many differences between OSM buildings and others.
\nTour:  before changing any settings, look at what is presented. What do you notice ?
\nNext, click the blue ‘plus’ sign on the upper-right side, notice the contents.<\/p>\n
\n
* State of Jupyter Article  -LINK-<\/a>
\n  – Notebooks are central to this solution<\/p>\n
\n
Big News from Google Earth -LINK-<\/a>
\nbut, the rumor is that they are not going to release the client app<\/p>\n","protected":false},"excerpt":{"rendered":"
15 Feb 17 Automated Ingestion Setup — California AB 802 Support ======================================================== Primary Components of Image Recognition * Automated data loading from primary sources * Automated Retrieval of specific content * Engine(s) applied to content * Review and Feedback   previously, we focused on automation of vector sources (auth and osm), this time, focus on […]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":1166,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"_links":{"self":[{"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/pages\/3322"}],"collection":[{"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3322"}],"version-history":[{"count":1,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/pages\/3322\/revisions"}],"predecessor-version":[{"id":3323,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/pages\/3322\/revisions\/3323"}],"up":[{"embeddable":true,"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=\/wp\/v2\/pages\/1166"}],"wp:attachment":[{"href":"http:\/\/blog.light42.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3322"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}