* rebuilt tiling
\n – there was a bug in the tiling code that caused gaps, due to a floating point truncation; using %f format for the float, as input to the transform, was insufficient resolution and the tiles were slightly malformed
\n – after retiling Inglewood and Humboldt Arcata, re-run training and prediction
\n – take advantage of the change, and clip Humboldt to a water layer<\/p>\n
* Made a quick spatial join of Predicted set to Polys Humboldt Data Inglewood \/ Training Humboldt \/ Prediction MGRS Grid Reference<\/strong> MGRS 10t 10s 11s start<\/p><\/div><\/p>\n buildings.buildings final<\/p>\n 5563240 from all tables <\/p>\n Google Earth Enterprise on Github Developer See is using SEGNET based ML 24 Mar 17 Machine Learning — Tuning the Training Process<\/strong><\/p>\n The BIS2<\/strong> segmentation engine has three paramaters. For this series of tests In The 675 tables test tables: 675 = 9 * 5 * 5 * 3<\/p>\n In the next step, statistics are calculated on the previous results (code in Completion of this step is to choose the best parameter set of a search objective. In this experiment, we are matching shape-to-shape<\/strong>. Python Visual display of segmentation in a Training Area of Inglewood, California.<\/p><\/div>\n Relevance Tests<\/strong><\/p>\n pctoverlap<\/strong> => the area of the intersection between the target training poly and a segmented poly, divided by the total area of the segmented poly; meaning, the percentage of the segmented poly that is intersecting the target, converges to 1.0 on a segmented poly that is entirely intersecting the target<\/p>\n cov1<\/strong> => the area of a building that is not covered by the intersection with the segmentation poly plus<\/em> the absolute value of the difference between the segmented poly and the building poly; meaning, a measure of non-fit; converges to zero on a perfect fit (actually, we changed this again, see code)<\/em><\/p>\n cov2<\/strong> => the area of the segmented poly outside of the intersection with the target, divided by the area of the segmented poly; meaning, the proportion of the poly that is not part of the target fit; converges to 0 on a perfect fit<\/p>\n Process Steps<\/strong><\/p>\n Building Size<\/strong> Next, what is a reasonable cutoff for buildings between 90 m-sq and MAX_SIZE Let's start simple.<\/p>\n A quick look at the full training set shows that roughly 90,000 of the 250,000 bldgs are less than 150 m-sq, which means that 160,000 or 64% are >= 150 m-sq . My first reaction is ... look for >= 150 m-sq<\/p>\n --<\/p>\n 20Mar17 Next Up<\/strong><\/p>\n Segmentation is a general-purpose topic in imaging right now, so inevitably there will be other toolkits and advanced researchers that use it.. Some parts of the setup now would have to be generalized to handle other segmentation engines, but that is \"out of scope\" for this pass. For example, masking, poly splitting and poly merging, are well know in image segmentation. We have many options to explore, but must prioritize as we make the best use of the BIS2 tool we started with.. <\/p>\n We have successfully added a fifth band to the raster layer 'mode'='sobel'. revised relevance query:<\/p>\n 50-03-03<\/strong> dist2road<\/strong> NAIP 2016 60cm segmented; predicted and classified based on Inglewood_5b model, sklearn Gradient Boost Tree (GBT).<\/p><\/div>\n <\/p>\n Inglewood_5b training polygons, alongside classified segmented polygons<\/p><\/div>\n Visualized output per parameter set:<\/p>\n http:\/\/ct.light42.com\/\/osmb\/?zoom=19&lat=33.87306&lon=-118.37408&layers=B000000FFFFFTFFT&seg=evaltest_evaltest1_tc_70_03_07<\/p>\n http:\/\/ct.light42.com\/osmb\/?zoom=19&lat=33.87306&lon=-118.37408&layers=B000000FFFFFTFFT&seg=evaltest_evaltest1_tc_70_03_05<\/p>\n ECN OSM Import<\/strong> BIDS<\/strong> OBIA<\/strong> https:\/\/www.ioer.de\/segmentation-evaluation\/results.html <\/p>\n OSM<\/strong> Jochen Topf reports that the multi-polygon fixing effort is going well and provides an issue thread to track the evolution. > I think this is it? NGA<\/strong> Development Seed<\/strong> <\/p>\n 17 Mar 17 \n “object-based approaches to segmentation ... are becoming increasingly important as Andy Lyons, PhD Topics in Segmentation<\/strong> Measuring Success<\/strong> \nAlthough many image segmentation methods have been developed, excerpted from:<\/em> research with examples of Object-based Image Analysis (OBIA):<\/em><\/p>\n An object-oriented approach to urban land cover change detection. Doxani, G.; Siachalou, S.; Tsakiri-Strati, M.; Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci 2008, 37, 1655\u20131660. [Google Scholar<\/a>]<\/p>\n Monitoring urban changes based on scale-space filtering and object-oriented classification. Doxani, G.; Karantzalos, K.; Strati, M.T.; Int. J. Appl. Earth Obs. Geoinf 2012, 15, 38\u201348. [Google Scholar<\/a>]<\/p>\n Building Change Detection from Historical Aerial Photographs more research -LINK-<\/a><\/p>\n A Few Definitions: <\/p>\n Training buildings available in Inglewood, Calif.<\/p><\/div>Introduction:<\/em> Source imagery is fed to a segmentation engine using parameter combinations of threshold (t), shape count (s) and compactness\/smoothness (c). The results are saved. For every chosen entry in a table of known good training polygons, intersection is calculated and statistics are stored. Search is performed to find optimal \"goodness of fit\" by some criteria. The results parameters are then propagated to subsequent segmentation of new imagery, those results are stored, and searched for likely matches.<\/p>\n Known-good Building Polygons all have the following attributes available:<\/p>\n from<\/em> Segmentation Trials -LINEUP-<\/a> <\/p>\n 08 Mar 17 Project Big Picture<\/strong> ML Theory<\/strong> Project Details<\/strong> Building a Training Set: filter a set of 2D building polygons of interest from LA County ( Recognition: Segment a target search area using remote sensing layers NAIP 2016; use ML engine and training product to recognize likely matches; characterize the results.<\/p>\n Documentation & Handoff<\/strong> <\/p>\n * Training Set, continued:<\/p>\n - Garlynn's crosswalk JOIN table gives a LOT of category information on the buildings themselves. In effect, takes 300,000 or so non-residential buildings and puts a very fine grain classification on them.<\/p>\n First attempt at a JOIN with a VIEW.. did not work for QGis and had permissions problems within the database. Second attempt simply defines a TABLE in the First pass at a finely categorized non-residential building set in LA County as ML Training Set candidates.<\/p><\/div>\n Misc URLs<\/strong> .. and, Machine Learning problematic data <\/em><\/p>\n http:\/\/ct.light42.com\/osmb\/?zoom=18&lat=33.92182&lon=-118.35224&layers=000000BFFTFFTFF<\/a><\/p>\n Characterize Training Set Buildings<\/strong> Attributes and Sources For Training Set<\/strong> (gw): <\/p>\n Server Layout<\/strong><\/p>\n <\/p>\n California Department of Conservation<\/strong> also<\/em>: Some Breadth on (pre)Machine Learning Topics<\/strong> Chapter 5 - Model Estimation, page 116 Maximum Likelyhood Estimation -WIKIPEDIA-<\/a> <\/p>\n 01 Mar 17 * prep for segmentation<\/p>\n - web visible results -LINK-<\/a><\/p>\n <\/p>\nhtarg2<\/code>
\n – sent to garlynn for preliminary QC
\n – created a few QGis visualizations<\/p>\n
\n dist2road<\/p>\n
\n ctr_targ T\/F; ctr_seg T\/F; perc_overlap (0-1); <\/p>\n
\n coverage2 value (0-1);<\/p>\n
\n![\"\"](\"http:\/\/blog.light42.com\/wordpress\/wp-content\/uploads\/2013\/04\/mgrs_import_scrn.png\")
<\/a>
\n5457201 removing duplicates<\/p>\n
\nhttp:\/\/www.opengee.org\/<\/p>\n
\nhttps:\/\/github.com\/alexgkendall\/caffe-segnet
\nhttps:\/\/github.com\/developmentseed
\nhttps:\/\/www.developmentseed.org\/projects\/<\/p>\n
\nRecognition Trials — WA18
\n========================================================<\/p>\nevaltest<\/code>, the imagery zoom level is always “full image resolution” and only one NAIP DOQQ is segmented (a portion of Inglewood). Iterating through input parameters, segmentation is performed and the result polygons are saved. This is repeated many times, and for several ‘modes’<\/em> of the imagery. Valid imagery ‘mode’<\/em> starts with true color ‘tc’, false color ‘fc’ and infrared ‘ir’; new products such as a Sobel Filter result can be added. (20mar17 – first sobel-enhanced segmentation added)<\/p>\nevaltest<\/code>, 675 segmentation run combinations of (t,s,c,mode) at full resolution on NAIP 2016 imagery were executed and the results are stored as shapefiles on disk. A small python program (load-seg-data.py<\/code>) loads the shapefiles into a spatial table. <\/p>\n
\n 9 values for threshold (t) 10,20,30,40,50,60,70,80,90
\n 5 values for shape (a) 0.1,0.3,0.5,0.7,0.9
\n 5 values for compact (c) 0.1,0.3,0.5,0.7,0.9
\n 3 values for mode ir,fc,tc<\/p>\nprep-relevance.py<\/code>). These stats are saved in tables in the ma_buildings relevance<\/code> schema. Tables have parameter values embedded in their names in a structured way (see below). <\/p>\neval-stats.py<\/code> generates SQL which contains a regular expression which filters the 675 tables of run result stats by encoded name, performs summary calculations on all rows in those tables (connected in the output by UNION ALL) and sorts all of that by a desired stats output column to discover outcomes in the test runs. Built-in Postgres math functions min() max() avg() stddev()<\/strong> are executed on three columns from each table, and counts of two more booleans. A SQL UNION ALL construct lists the chosen stats for all considered tables, one line per table, which then can be sorted in any way that is convenient to pick winners. Using this generalized format, statistics for any model run can quickly be compared. On the testing hardware, 450 tables of a few thousand rows each, are summarized and sorted for comparison in about 1.5 seconds.<\/p>\n<\/a>\r\n-- Five Spatial Tests are Performed on Every Segment Intersecting a Training Building --\r\n\r\nCREATE TABLE relevance.evaltest_evaltest1_MODE_T_S_C\r\n(\r\n gid integer PRIMARY KEY,\r\n class integer, -- 0 not considered; 2 commercial bldg\r\n pctoverlap double precision,\r\n coverage1 double precision,\r\n coverage2 double precision,\r\n centr_seg boolean,\r\n centr_trg boolean\r\n)\r\n\r\n-- Structure of a model run stats table 23mar17\r\n--\r\nma_buildings=# select * from \"5bandt1_test2_5b_50_03_03\" limit 5;\r\n gid | class | pctoverlap | coverage1 | coverage2 | centr_seg | centr_trg \r\n-------+-------+-------------------+-------------------+------------------+-----------+-----------\r\n 64311 | 2 | 0.547329611192823 | 0.970785850687081 | 35.9228431691447 | t | f\r\n 64559 | 2 | 0.137682284191657 | 0.836077333010945 | 0.54216208047878 | f | f\r\n 64107 | 2 | 1 | 0.972281692288727 | 70.15447713594 | t | f\r\n 64698 | 2 | 0.728239585461471 | 0.924875366545267 | 17.6592649379458 | t | f\r\n 64721 | 2 | 0.923712040345278 | 0.943590470848252 | 30.8265007068801 | t | f\r\n(5 rows)\r\n\r\n\r\n\r\nRelevance Query:\r\n-- segmented poly (a) < = polys from a segmentation run w\/ given params \r\n-- target poly (b) <= training polys, what we are matching \r\n-- raw areas (c) <= all known bldg polys in the area\r\n--\r\n\r\n | insert into relevance.evaltest_evaltest1_fc_50_09_05 \r\n | select a.gid, \r\n | 1::integer, \r\n | st_area(st_intersection(a.geom,c.geom))\/st_area(a.geom), \r\n | st_area(c.geom) - st_area(st_intersection(a.geom,c.geom)) \r\n | + abs(st_area(a.geom) - st_area(c.geom)), \r\n | (st_area(c.geom) - st_area(st_intersection(a.geom,c.geom))) \r\n | \/ st_area(a.geom), \r\n | st_intersects(st_centroid(a.geom), c.geom), \r\n | st_intersects(st_centroid(c.geom), a.geom) \r\n | from seg_polys a \r\n | join raw.areas c on st_intersects(a.geom, c.geom) \r\n | left outer join \r\n relevance.evaltest_evaltest1_fc_50_09_05 b on a.gid=b.gid \r\n | where b.class is null and \r\n jobid='evaltest' and project='evaltest1' and \r\n mode='fc' and t=50.000000 and s=9.000000 and c=5.000000 \r\n on conflict do nothing \r\n<\/pre>\n
\r\neval-stats.py -m tc -f cov1 -o avg -g\r\n\r\n search all result tables in schema 'relevance' for test run summary tables matching\r\nimagery mode 'tc' True Color with any parameters; from those tables, \r\ncalc min() max() avg() and stddev() on all measures, but emit only the totals for the chosen column 'cov1', \r\none line per table; sort those results on column 'average' with the default sort order \r\nof smallest first. In this example, segmentation parameter 20-07-07 gave the lowest\r\naverage value of cov1 for all test runs, with 20-07-03 closely following. \r\n\r\n table class count minimum maximum average stddev centr_seg centr_trg\r\n----------------------------------------------------------------------------------------------------------\r\nevaltest_evaltest1_tc_20_07_07 1 7237 0.000324 0.801199 0.052418 0.083101 3420 1414\r\nevaltest_evaltest1_tc_20_07_03 1 6694 0.000476 0.800666 0.052979 0.083563 3158 1336\r\nevaltest_evaltest1_tc_30_05_09 1 5404 0.000354 0.800875 0.053037 0.084356 2406 1232\r\nevaltest_evaltest1_tc_30_03_09 1 6731 0.000324 0.800875 0.053139 0.082947 3202 1362\r\nevaltest_evaltest1_tc_40_01_07 1 5310 0.000468 0.899413 0.053150 0.084180 2394 1223\r\nevaltest_evaltest1_tc_30_05_03 1 5252 0.000320 0.801222 0.053332 0.082887 2362 1209\r\nevaltest_evaltest1_tc_30_01_01 1 8033 0.000331 0.801431 0.053378 0.082763 3932 1425\r\nevaltest_evaltest1_tc_30_01_05 1 8069 0.000331 0.800875 0.053403 0.081330 3936 1451\r\nevaltest_evaltest1_tc_30_01_07 1 8212 0.000394 0.801083 0.053454 0.082504 4053 1459\r\nevaltest_evaltest1_tc_20_05_03 1 9560 0.000320 0.801222 0.053536 0.081324 4769 1530\r\nevaltest_evaltest1_tc_30_05_05 1 5379 0.000531 0.800875 0.053862 0.083052 2403 1210\r\nevaltest_evaltest1_tc_20_05_05 1 9607 0.000476 0.801410 0.053940 0.083886 4795 1542\r\nevaltest_evaltest1_tc_30_03_05 1 6658 0.000320 0.800875 0.054063 0.083892 3125 1324\r\n<\/pre>\n
\r\n-- Classified Segments with a Given Band and Segmentation --\r\n\r\nma_buildings=# select class, count(*) from evaltest_evaltest1_ir_50_01_03 group by class;\r\n class | count \r\n-------+-------\r\n 1 | 4708\r\n 2 | 324\r\n(2 rows)\r\n<\/pre>\n
\nTopics in Training Target Selection<\/h4>\n
\n
\ndone -- 80 in Inglewood and 130 in Humboldt<\/em><\/li>\n
\npredict-segments.py and train-segments.py<\/em><\/li>\n
\n Very small buildings are more likely to be confused with other kinds of visible features, therefore remove training target buildings smaller than 90 m-sq<\/p>\n
\nThis is a stats distribution question.. We know the buildings of interest (commercial) look very much like other buildings. What is are identifying characteristics of commercial buildings that will help distinguish them from all buildings? Looking at distributions of features on all buildings is some curve, and the features of buildings we are looking for in the training set is a different curve.. Non-commercial buildings will have more occurrences in the low sq-m sizes, making false matches to commercial buildings more likely. <\/p>\n
\nUsing this five-band input, combined with segmentation trials, definitely
\nimproves the result. The toolchain did not have to change other than add the new layer.
\nDocumenting the details of this is better left at the code level.<\/p>\n\r\ninsert into relevance.\"inglewood_run1_5b_50_03_03\" \r\n 1::integer, \r\n -- pctoverlap \r\n st_area(st_intersection(a.geom,c.geom))\/st_area(a.geom), \r\n -- coverage1 \r\n (st_area(c.geom) + st_area(a.geom) - \r\n 2.0*st_area(st_intersection(a.geom,c.geom))) \/ \r\n (st_area(c.geom) + st_area(a.geom)), \r\n -- coverage2 \r\n (st_area(c.geom) - st_area(st_intersection(a.geom,c.geom))) \r\n \/ st_area(a.geom), \r\n -- centr_seg \r\n st_intersects(st_centroid(a.geom), c.geom), \r\n -- centr_trg \r\n st_intersects(st_centroid(c.geom), a.geom) \r\n from seg_polys_5b a -- (3) segment polygons \r\n join la14.lariac4_buildings_2014 c on st_intersects(a.geom, c.geom) -- (4) building polygons \r\n left outer join relevance.\"inglewood_run1_5b_50_03_03\" b on a.gid=b.gid \r\n where b.class is null and jobid='inglewood' and project='run1' and \r\n mode='5b' and t=50.000000 and s=3.000000 and c=3\r\n\r\n -- previous\r\n select a.gid, \r\n 1::integer, \r\n -- pctoverlap \r\n st_area(st_intersection(a.geom,c.geom))\/st_area(a.geom), \r\n -- coverage1 \r\n (st_area(c.geom) + st_area(a.geom) - \r\n 2.0*st_area(st_intersection(a.geom,c.geom))) \/ \r\n (st_area(c.geom) + st_area(a.geom)), \r\n -- coverage2 \r\n (st_area(c.geom) - st_area(st_intersection(a.geom,c.geom))) \r\n \/ st_area(a.geom), \r\n -- centr_seg \r\n st_intersects(st_centroid(a.geom), c.geom), \r\n -- centr_trg \r\n st_intersects(st_centroid(c.geom), a.geom) \r\n from seg_polys_5b a \r\n join raw.areas c on st_intersects(a.geom, c.geom) \r\n left outer join relevance.\"5bandt1_test2_5b_50_03_04\" b on a.gid=b.gid \r\n where \r\n b.class is null and jobid='5bandt1' and project='test2' and mode='5b' and \r\n t=50.000000 and s=3.000000 and c=4.000000 \r\n on conflict do nothing \r\n\r\n--\r\ntrain-segments.py\r\n from sklearn.neighbors import KNeighborsClassifier\r\n from sklearn.ensemble import GradientBoostingClassifier\r\n\r\n\r\n--\r\nma_buildings=# \\d inglewood_run1_5b_50_03_03\r\nTable \"relevance.inglewood_run1_5b_50_03_03\"\r\n Column | Type | Modifiers \r\n------------+------------------+-----------\r\n gid | integer | not null\r\n class | integer | \r\n pctoverlap | double precision | \r\n coverage1 | double precision | \r\n coverage2 | double precision | \r\n centr_seg | boolean | \r\n centr_trg | boolean | \r\nIndexes:\r\n \"inglewood_run1_5b_50_03_03_pkey\" PRIMARY KEY, btree (gid)\r\n\r\n<\/pre>\n
<\/a><\/p>\n
\n We selected this parameter set to use for the test area.. It is good at selecting a lot of area from a large, single roof. Complex roofs that have colored parts, using the training tests we have, do not get joined using these settings.<\/p>\n
\n This measure seems important, yet it is a substantial cost to calculate, at least the way we are doing it now.<\/p>\n\r\n 5415961.887 ms statement: \r\n update sd_data.seg_polys_5b a \r\n set dist2road=(select st_distance(a.geom, b.geom) from roads b \r\n order by b.geom < ->a.geom limit 1) \r\n where dist2road is null and a.gid between 400001 and 500001\r\n<\/pre>\n
\nCompleted Prediction Run<\/h4>\n
<\/a><\/a>\r\n## Complete Machine-Learning Pipeline pass\r\n## 23March17 -dbb\r\n\r\n# tile the cousub for inglewood\r\n.\/tile-area.py -o inglewood -f 0603791400 -s 2000 -m 5b\r\n\r\n# segment all the tiles in the inglewood directory\r\n.\/segment-dir.py -m 5b -t 50 -s 0.3 -c 0.3 inglewood\/\r\n\r\n# load all the segment polygons into the database, compute dist2road\r\n# with jobid=inglewood, project=run1\r\n# segments go into sd_data.seg_polys_5b\r\n.\/load-seg-data.py -p run1 -m 5b inglewood inglewood\/\r\n\r\n# compare the segment polygons to the training polygons\r\n# filtering any training polygons less than 150 sq-m in area\r\n# save the results in relevance.inglewood_run1_5b_50_03_03\r\n.\/prep-relevance.py -b inglewood_run1_5b_50_03_03 -j inglewood -p run1\r\n-m 5b -t 50 -s 0.3 -c 0.3 -z 150\r\n\r\n# Using scikit learn and Gradient Boost Tree ML module\r\n# train it using the segments in the database\r\n# and save the trained object in inglewood_run1_5b_50_03_03-r0.5.pkl.gz\r\n.\/train-segments.py -j inglewood -p run1 -m 5b -r 0.5 -b\r\ninglewood_run1_5b_50_03_03 inglewood_run1_5b_50_03_03-r0.5.pkl.gz\r\n\r\n# tile an area for Humboldt based on a bounding box\r\n# and place the tiles in directory humboldt\r\n.\/tile-area.py -o humboldt -b -124.17664,40.85411,-124.05923,41.00326 -s\r\n2000 -m 5b\r\n\r\n# segment all the tiles in the humboldt directory\r\n.\/segment-dir.py -m 5b -t 50 -s 0.3 -c 0.3 humboldt\/\r\n\r\n# load all the segment polygons into the database, compute dist2road\r\n# with jobid=humboldt, project=target\r\n# segments go into sd_data.seg_polys_5b\r\n.\/load-seg-data.py -p target -m 5b humboldt humboldt\/\r\n\r\n# run the trained classifier inglewood_run1_5b_50_03_03-r0.5.pkl.gz\r\n# against the humboldt segments and save the results to\r\n# predict.humboldt_target_5b_50_03_03_r005\r\n.\/predict-segments.py -b humboldt_target_5b_50_03_03_r005 -j humboldt -p\r\ntarget -m 5b inglewood_run1_5b_50_03_03-r0.5.pkl.gz\r\n\r\n<\/pre>\n
\n
\nOther Topics<\/h4>\n
\nhttps:\/\/github.com\/darkblue-b\/ECN_osm_import\/blob\/master\/load-osm-buildings.cpp<\/p>\n
\n https:\/\/bids.berkeley.edu\/news\/searchable-datasets-python-images-across-domains-experiments-algorithms-and-learning<\/p>\n
\nhttp:\/\/wiki.landscapetoolbox.org\/doku.php\/remote_sensing_methods:object-based_classification<\/p>\n
\nhttps:\/\/www.ioer.de\/segmentation-evaluation\/news.html<\/p>\n
\n https:\/\/wiki.openstreetmap.org\/wiki\/United_States_admin_level#cite_ref-35<\/p>\n
\n https:\/\/blog.jochentopf.com\/2017-03-09-multipolygon-fixing-effort.html
\n https:\/\/github.com\/osmlab\/fixing-polygons-in-osm\/issues\/15
\n-<\/p>\n<\/a>
\nOn Mar 22, 2017 9:59 AM, \"Brad Neuhauser\"
\n> http:\/\/wiki.openstreetmap.org\/wiki\/Available_Building_Footprints
\n>
\n> On Wed, Mar 22, 2017 at 11:54 AM, Rihards
\n>
\n>> On 2017.03.22. 18:37, Clifford Snow wrote:
\n>> > I am happy to announce that Microsoft has made available approximately
\n>> > 9.8 million building footprints including building heights in key
\n>> > metropolitan areas. These footprints are licensed ODbL to allow
\n>> > importing into OSM. These footprints where manually created using high
\n>> > resolution imagery. The data contains no funny field names such as
\n>> > tiger:cfcc or gnis:featureid or fcode=46003, just building height.
\n>> >
\n>> >
\n>> > Please remember to follow the import guidelines.
\n>> >
\n>> > The wiki [1] has more information on these footprints as well as links
\n>> > to download.
\n>>
\n>> the link seems to be a copypaste mistake \ud83d\ude42
\n>>
\n>> > [1] http:\/\/www.openstreetmap.org\/user\/dieterdreist\/diary\/40727
\n>> >
\n>> > Enjoy,
\n>> > Clifford Snow
\n>> >
\n>> >
\n>> > --
\n>> > @osm_seattle
\n>> > osm_seattle.snowandsnow.us
\n>> > OpenStreetMap: Maps with a human touch
\n>> >
\n>> ><\/p>\n
\nhttps:\/\/apps.nga.mil\/Home<\/p>\n
\nhttps:\/\/developmentseed.org\/blog\/2017\/01\/30\/machine-learning-learnings\/<\/p>\n
\n
\nRecognition Trials -- WA18
\n========================================================<\/p>\n
\n the spatial resolution of images gets smaller relative to the size of the objects of interest.<\/em> ”<\/p>\n
\n University of California, Division of Agriculture and Natural Resources
\n Informatics in GIS -LINK-<\/a>\n<\/p><\/blockquote>\n
\nTopics for Detecting Buildings in Aerial Imagery<\/strong>:
\n- Object-based Image Analysis versus Pixel-based Image Analysis redux<\/em>
\n- Change-detection using multi-temporal imagery
\n- Inherent Difficulty of Identifying Buildings in Imagery, even for a human
\n- LIDAR can increase detection dramatically, but is difficult to scale<\/p>\n
\n- Object-based Image Analysis relies on Segmentation
\n- removing roads before segmentation
\n- removing vegetation before segmentation using IR layer
\n- using height information and shadow-detection<\/p>\n
\n- positive match rate to ground truth
\n- miss rate to ground truth
\n- false positives
\n- false negatives <\/p>\n
\nthere is a strong need for new and more sophisticated segmentation
\nmethods to produce more accurate segmentation results for urban object
\nidentification on a fine scale (Li et al., 2011)<\/em>.
\nSegmentation can be especially problematic in areas with low contrast
\nor where different appearance does not imply different meaning. In this
\ncase the outcomes are represented as wrongly delineated image objects
\n(Kanjir et al., 2010)<\/em>.<\/p>\n
\n URBAN OBJECT EXTRACTION FROM DIGITAL SURFACE MODEL AND DIGITAL AERIAL IMAGES Grigillo, Kanjir -LINK-<\/a>\n<\/p><\/blockquote>\n
\n
\n Using Dense Image Matching and Object-Based Image Analysis -LINK-<\/a>
\nNebiker S., Lack N., Deuber M.; Institute of Geomatics Engineering, FHNW University of Applied Sciences and Arts Northwestern Switzerland, Gr\u00fcndenstrasse 40, 4132 Muttenz, CH<\/p>\n
\nGSD<\/strong> -- ground sampling distance; real distance between pixel centers in imagery
\nDSM<\/strong> -- Digital Surface Model; similar to Digital Elevation Model
\n eDSM -- extracted Digital Surface Model; post-process result, varying definitions
\n nDSM -- normalized Digital Surface Model; varying definitions
\nFBM<\/strong> -- Feature-based mapping; store only features, not empty spaces
\nSGM<\/strong> -- Semi-global matching; computationally efficient pixel analysis
\nmorphological filtering -- computationally efficient edge methods
\nautomated shadow detection -- in OBIA
\nHough Transform -LINK-<\/a>
\n
\nRepresentative Public Test Process:
\n ISPRS Commission III, WG III\/4
\n ISPRS Test Project on Urban Classification and 3D Building Reconstruction<\/p>\n
\n\"... you dont want to be fooling with a lot of parameters.. you want paramater-free functions whenever you can.. thats one thing I think is exciting about neural networks (and the like), is that you just throw some images in there, and out come some magic answers. I think we will be seeing more of that.\"<\/em>
\nscikit-image: Image Analysis in Python \/ Intermediate
\nSciPy 2016 Tutorial * Stefan van der Walt<\/strong><\/p>\n
\nMachine Learning Pipeline Components<\/strong>
\n<\/a>\r\nma_buildings=# \\d train_bldgs_08mar17\r\n Table \"sd_data.train_bldgs_08mar17\"\r\n Column | Type | Modifiers \r\n---------------------------+-----------------------------+-----------\r\n gid | integer | not null\r\n building_type_id | integer | \r\n building_type_name | text | \r\n situscity | text | \r\n shp_area_m | integer | \r\n geom | geometry(MultiPolygon,4326) | \r\n naip_meta_1 | integer | \r\n naip_meta_2 | text | \r\n urban_ldc | integer | \r\n compact_ldc | integer | \r\n standard_ldc | integer | \r\n intersection_density_sqmi | double precision | \r\n acres_grid_gf | double precision | \r\n acres_grid_con | double precision | \r\n use_du_dens | double precision | \r\n<\/pre>\n
\nTheory of Determination of Relevance<\/strong> -- Clinton_Scarborough_2010_PERS<\/p>\n<\/p>\nAccuracy Assessment Measures for Object-based Image Segmentation Goodness: Clinton, et al<\/code><\/p>\n
\nPipeline Tools to date:
\n
\n<\/p>\n\r\nprep-relevance.py >options<\r\n create a table associated with subset of seg_polys and assign\r\n to a class based on mutual percent overlap with a truth polygon\r\n \r\n options:\r\n -h|--help\r\n -i|--info - report available jobid, project, t, s, c parameters and exit\r\n -b|--table tablename - table to create with relevance and class code (required)\r\n -j|--jobid name -+\r\n -p|--project name |\r\n -m|--mode ir|tc|fc|4b |- filters for selecting specific records\r\n -t|--threshold n.n |\r\n -s|--shape n.n |\r\n -c|--compact n.n -+\r\n -v|--verbose\r\n\r\n<\/pre>\n
\n <\/p>\n\r\ntrain-segments.py\r\nUsage: train-segments.py >options< trained-object.pkl.gz\r\n -i|--info - report available jobid, project, t, s, c parameters\r\n -b|--table tablename - table with relevance and class code\r\n -j|--jobid name -+\r\n -p|--project name |\r\n -t|--threshold n.n |- filter dataset to be extracted\r\n -s|--shape n.n |\r\n -c|--compact n.n -+\r\n -r|--relevance n.n - min relevance value to be part of class, default: 0.5\r\n -v|--verbose\r\n -x|--test - Use 50% of data to train and 50% to predict and report\r\n -a|--algorithm - KNN - KNearestNeighbors|\r\n GBT - Gradient Boost Tree, default: GBT\r\n\r\n conn = psycopg2.connect(\"dbname=ma_buildings<\/strong>\")\r\n\r\npredict-segments >options< trained-object.pkl.gz\r\n -i|--info - report available jobid, project, t, s, c parameters\r\n -b|--table tablename - table to create with relevance and class code\r\n -j|--jobid name -+\r\n -p|--project name |\r\n -t|--threshold n.n |- filter dataset to be extracted\r\n -s|--shape n.n |\r\n -c|--compact n.n -+\r\n -x|--test - do the prediction but don't write to the database\r\n\r\n\r\neval-stats.py\r\n scan intermediate relevance tables and compute descriptive statistics\r\n\r\neval-segmentation.py\r\n evaluate multiple t, s, c parameters of the segmentation process\r\n and try to pick the that gives up the best results\r\n\r\n for a given training set:\r\n * take 40% of the polygons and use them for training\r\n * take 40% of the polygons and use them for evaluation\r\n * save 20% of the polygons as a hold back set for final evaluation\r\n\r\n there are currently two proceses planned to do this:\r\n 1. a short path:\r\n compute the relevance for all the polygons\r\n and try to increase the average mutual coverage\r\n and decrease the stddev on the average mutual coverage\r\n take the best 2-3 sets of parameters and run them through 2.\r\n\r\n 2. a long path: (NOT IMPLEMENTED YET)\r\n compute the relevance for all the polygons\r\n create a trained classifier using the training polygons\r\n run the test polygons through the trained classifier\r\n evaluate the goodness of the fit\r\n\r\n In either case, save the parameters that gave the best fit.\r\n\r\n Relevance tables will be generated based on >jobid< >project< t s c\r\n\r\n-------------------------------------------------------\r\nload-seg-data.py [-c] jobid dir\r\n -c - optionally drop and tables\r\n jobid - string to uniquely identify this data\r\n dir - segmentizer output directory to walk and load\r\n\r\nFor a given input file like project\/tcase\/test-fc.vrt\r\nthe segmentizer generates files in path project\/tcase\/:\r\n\r\ntest-fc.vrt_30_05_03.tif\r\ntest-fc.vrt_30_05_03.tif_line.tif\r\ntest-fc.vrt_30_05_03.tif_rgb.tif\r\ntest-fc.vrt_30_05_03.tif_stats.csv\r\ntest-fc.vrt_30_05_03.tif.shp\r\n\r\nWe can generate files given this matrix:\r\n\r\n mode | t | S | c | z |\r\n ------------------------------\r\n fc | 30 | 05 | 03 | 19 |\r\n ir | 50 | ... | 07 | ... |\r\n tc | ... | ... | ... | ... |\r\n ------------------------------\r\n\r\nWe can generate any number of values for t, s, c, and z.\r\nSegmented results are stored in table seg_polys,\r\n\r\ncreate table sd_data.seg_polys (\r\n gid serial not null primary key,\r\n project text, -- from file path\r\n tcase text, -- from file path\r\n ...\r\n\r\n--\r\nsegment-dir options dir\r\n Call the segmentation engine for each file in a directory, with the given settings\r\n Produce an HTML overview for human interpretation of results\r\n\r\n [-t|--threshold T] - T1[,T2,...] thresholds, default: 60\r\n [-s|--shape S] - S1[,S2,...] shape rate, default: 0.9\r\n [-c|--compact C] - C1[,C2,...] compact\/smoothness, default: 0.5\r\n [-v] - be verbose\r\n [--nostats] - don't generate stats.csv file\r\n [--nolineup] - don't generate the lineup image\r\n\r\n\r\n--\r\nThe following database table enables reference of any dataset from a mapfile:\r\n\r\ncreate table sd_data.seg_tileindex (\r\n gid serial not null primary key,\r\n project text, -- from file path\r\n tcase text, -- from file path\r\n ...\r\n\r\n<\/pre>\n
\nOther Imaging Filters Dept.
\n A Sobel Filter<\/a> from sckit-image may be worth adding to the chain
\nThe filter detects edges well (though they are not squared and various weights).
\nA target area could be passed through the filter and added as a segmentation clue.<\/p>\n
\nRecognition Trials -- WA18
\n========================================================<\/p>\n
\n \"tripartite solution\" -- Authoritative Sources of 2D building footprints, with varying attributes (dbname=auth_buildings<\/code>), updates and ingestion are semi-manual<\/em>; Openstreet Buildings with minimal attributes (dbname=osm_buildings<\/code>), updates are automated<\/em>; Machine-Learning generated building locations (dbname=ma_buildings<\/code>), updates are machine-generated with supplementary attributes.<\/p>\n
\n Unsupervised Classification with pixel-based analysis (see below)
\n Supervised Classification with Object-based analysis (what we are using, newer)<\/p>\n
\n Database ma_buildings<\/code> schemas now include: building_types, a classification system for buildings in all use_codes; US Census TIGER, road network, block groups, county subportions; Grid 150m Los Angeles County, vast, high-resolution landuse and urban metrics data; LARIAC4 Buildings 2014, every building in LA from SCAG; sd_data training_buildings, a table-driven source of ML training targets (see below), picked from 249,927 classified geometry records of commercial and MF5+ buildings in LA County.<\/p>\ntable=sd_data.train_bldgs<\/code>); run a segmentation on remote sensing data layers NAIP 2016, store the results (table=seg_polys<\/code>).<\/p>\n
\n discussion<\/em><\/p>\n
\n\r\n\"Each and Every Building in LA, provided by the County\"\r\nma_buildings la14.lariac_buildings_2014\r\n----------------------------------------\r\ncode, bld_id, height, elev,\r\nlariac_buildings_2014_area, source, date_, ain, shape_leng, status, old_bld_id,\r\n ...\r\n\r\n\"Building Categorization from Vision_California release\"\r\nbuilding_types\r\n--------------------------------------------------------\r\n building_type_category text,\r\n building_type_id integer NOT NULL,\r\n building_type_name text,\r\n building_height_floors integer,\r\n total_far integer\r\n ...\r\n\r\nUseCode_2_crosswalk_UF_Building_Types2.csv\r\n-------------------------------------------\r\nusecode_2,usetype,usedescription,bt_id,mf5_plus_flag,include_flag\r\n10,Commercial,Commercial,41,,1\r\n11,Commercial,Stores,41,,1\r\n12,Commercial,Store Combination,39,,1\r\n13,Commercial,Department Stores,39,,1\r\n14,Commercial,Supermarkets,41,,1\r\n15,Commercial,\"Shopping Centers (Neighborhood, community)\",41,,\r\n16,Commercial,Shopping Centers (Regional),39,,1\r\n17,Commercial,Office Buildings,32,,1\r\n18,Commercial,Hotel & Motels,38,,1\r\n19,Commercial,Professional Buildings,32,,1\r\n....\r\n<\/pre>\n
sd_data<\/code> schema. Success<\/strong> !<\/p>\n<\/a>
\n http:\/\/ct.light42.com\/osmb\/work\/lineup.html<\/a>
\n http:\/\/ct.light42.com\/osmb\/la14-50-60-09-05\/lineup.html<\/a>
\n http:\/\/ct.light42.com\/osmb\/la14-50-60-09-01_05_07\/lineup.html<\/a><\/p>\n
\n - residential multifamily five units and up - are included
\n - how many buildings ? what sizes ? no filter on \"small bldgs\" yet<\/p>\n
\n- Should record imagery attributes <\/strong>(direction of sun angle, year, etc). This will come from the NAIP metadata. WHY<\/em><\/strong>: It obviously matters from which direction the sun is shining, as shadows will be found on the opposite sides of objects from the sun. Time of year will tell about the presence or absence of leaves on deciduous vegetation. The year allows for time-series comparisons between different vintages of the same dataset.
\n- City<\/strong> (name) \u2014 this can come from a number of sources, including the LARIAC buildings shapes themselves (SitusCity). WHY<\/em><\/strong>: Different cities may have different attributes, citywide, in a meaningful manner.
\n- The area of the polygon<\/strong>, from LARIAC buildings training set (Shape_Area). WHY<\/em><\/strong>: We want to be able to filter by this attribute for only large buildings.
\n- urban_ldc<\/strong> from the UF base grid. WHY<\/em><\/strong>: These are super-urban areas, more than 150 walkable intersections\/sqmi, most buildings here are skyscrapers.
\n- compact_ldc <\/strong>from the UF base grid. WHY<\/em><\/strong>: These are compact development types, more than 150 walkable intersections\/sqmi, with a walkable street grid.
\n- standard_ldc<\/strong> from the UF base grid. WHY<\/em><\/strong>: These are suburban development types, less than 150 walkable intersections\/sqmi,
\n- intersection_density_sqmi<\/strong> from the UF base grid. WHY<\/em><\/strong>: Intersection density is a critical variable in urbanism. It could be that the machine finds a strong correlation between it and other things. Let\u2019s find out.
\n- acres_grid_gf<\/strong> from the UF base grid. WHY<\/em><\/strong>: Greenfield acres tell us how many non-urbanized acres are within the grid. If this is above some threshold, say 85% of the grid cell, we can probably skip that grid cell for ML analysis.
\n- acres_grid_con<\/strong> from the UF base grid. WHY<\/em><\/strong>: Constrained acres also tell us how many non-urbanized acres are within the grid. If this is above some threshold, say 85% of the grid cell, we can probably skip that grid cell for ML analysis.
\n- use_du_dens<\/strong> from the UF base grid. WHY<\/em><\/strong>: We may be able to do something clever, like filter out the residential records from the buildings dataset by only those grid cells from the UF base load with a DU use density above some threshold (I\u2019m thinking 10-20 or so), which should get rid of some of the noise in the data.
\n- Building Size -- Filter all records by our threshold size: 2,000? 10,000? ...of 2D footprint. Maybe don\u2019t try this during the first pass? Keep it in your back pocket for later? Not sure.<\/p>\n
\n\r\n \/var\/local\/osmb_ab\/\r\n bin # all automation, plus small utils\r\n add-auth-building-layer do-backup json-to-postgres osm_building_stats\r\n create-auth-buildings extract-image.py load-osm-buildings reload-osm-buildings\r\n createdb-dee1 fetch-osm-ca-latest load-roads wget-roads\r\n \r\n bis2 # the Image Segmentation engine\r\n cgi-bin # mapserver(tm) delivers the view of data\r\n data # links to large data stores, raw inputs, csv files\r\n auth_buildings dbdump misc naip sd_data tiger2016-roads\r\n html # mapserver site \r\n maps # mapserver layer definitions\r\n src # misc code\r\n code_dbb ECN_osm_import naip_fetch naip-fetch2 naip-process osm-buildings sql\r\n\r\n<\/pre>\n
\n
\nnew maps site: https:\/\/maps.conservation.ca.gov\/<\/a>
\n\"... For further information or suggestions regarding the data on this site, please contact the DOGGR Technical Services Unit at 801 K Street, MS 20-20, Sacramento, CA 95814 or email DOGGRWebmaster@conservation.ca.gov. \"<\/p>\n
\n http:\/\/statewidedatabase.org\/ < - referred from a wikipedia page, no other info\n\n \n\n\n
\n
\n (see gw\/rdir\/snippets_misc for more like this<\/em>)
\n
\nPixel Matching -- Notes on Maximum Likelyhood Classification Analysis
\n \"... is a statistical decision criterion to assist in the classification of overlapping signatures; pixels are assigned to the class of highest probability.\" -LINK-<\/a><\/a>
\n \"one approach is to use Bayes' Rule and maximize the agreement between a model and the observed data\"<\/p>\n
\n The Nature of Mathematical Modeling; Gershenfeld, Neil, Cambridge University Press 1999<\/code><\/p>\n
\n Bayes' Rule -WIKIPEDIA-<\/a><\/p>\n
\nRecognition Trials -- WA18
\n========================================================<\/p>\n\r\n$ extract-image.py -o \/sand480\/extractions_work\/t2.tif -z 2 40.87427 -124.08024\r\n\/sand480\/extractions_work\/t2.tif-tc.tif\r\n\/sand480\/extractions_work\/t2.tif-ir.tif\r\n<\/pre>\n
<\/a> <\/a><\/p>\n<\/a><\/p>\n
\n