Skip to content

jan17_RE2_archive

Berkeley ImageSegbis-2
python environment is virtualenv; found a recipe to run ipython
notebook from within

source activate ENVNAME
pip install ipykernel
python -m ipykernel install --user --name ENVNAME --display-name "Python (whatever you want to call it)"

ETL Openstreetmap Buildings Import
* libosmium tool scripted
  input is California-latest.pbf filtered by tag “building”
 
Code: ECN_osm_import Github -LINK-
Report: Summary Counts -CSV-
 

This screenshot shows OSM California ogc_fid object 1, which is Hanger One at NASA-Ames Airflied, and other OSM Buildings around Sunnyvale, CA; imported using libosmium 2.11.


 


 

Comparison of libosmium import versus ogr import in San Luis Obisbo; libosmium layer uniques are bright red.

 
Not all data is as expected; a LINESTRING marked “building” -LINK-

-----------------------------------------------------------------------
gid       | 5
id        | 13863133
tags      | "name"=>"Gymnasium","building"=>"yes","created_by"=>"Potlatch 0.5c"
version   | 2
changeset | 600831
timestamp | 2007-11-23T17:48:58Z
uid       | 2376
user      | Bman
geom      | 0102000020E6100000050000009EFB613A63465DC02D1D18E53E05414033EF82D261465DC0ACF188C047054140E1AB675254465DC04411F7FD400541404BB846BA55465DC0A21EEC5E380541409EFB613A63465DC02D1D18E53E054140
-[ RECORD 6 ]-------------------------------------------------------------------------------

 
Some interactive manipulation of post-import stats: -LINK-

Jupyter Notebook Tech Available: -LINK-

 


California Air Resources Board — 2030 Scoping Plan UPDATE -LINK-
 ref. page 95 Section IV Key Sectors
  … The Governor’s Green Buildings Executive Order B-18-12 for State buildings and the California Green Building Standards (CALGreen) Code 130 are key state initiatives supporting emission reductions associated with buildings, and some local governments are taking action by adopting “beyond code” green building standards…

  Section 3: Efforts to Reduce Greenhouse Gases
  …
  Ongoing and Proposed Measures – Electricity
   ref. page 103
   Per AB 802, adopt the forthcoming CEC regulations governing building energy
use data access, benchmarking, and public disclosure.

 
18 Jan 17
Automated Ingestion Setup — California AB 802 Support
========================================================

Backup

  * 50GB BluRay Dual-Layer disk capacity
    burn time ~120 minutes  for one disk
    ATAPI: PIONEER BD-RW   BDR-207D, 1.21
    Ridata BD-R DL Media  $3/disk for 25 at Frys

  * Disk-to-Disk
    75GB/hour for USB connection
    300GB/hour for eSATA

  geodatamine_f2 compressed backup 13GB
  NAIP compressed backup 2000GB
  TIGER compressed backup 20GB

Hardware
 Desktop: Core i7-2720QM @ 2.20GHz
 Server: Core i7-960 @ 3.20GHz

Software Toolchain
 Berkeley Image Seg BIS

 Openstreetmap Toolchain

Openstreetmap Buildings Coverage
SF Bay Area example
18 Jan 2017

Task 2 Workflow Sketch
18 Jan 2017

11 Jan 17
Automated Ingestion Setup — California AB 802 Support
========================================================

WA 18 / Task 2

WA18 Objectives
“… develop data collection processes to compile or create existing building footprints using GIS and assign unique identification numbers to these footprints”

New Buildings Acquisition Software Stack:
Evaluation copy of Berkeley Image Seg
 – Is this an acceptable component of the delivery ?
 – what software environment will be delivered ?
 – what documentation of the environment will be delivered ?

Example Python Worksheet from LA Buildings Import -LINK-

Example Data Browser — Berkeley Geo Library -LINK-

Example Crowd Source Project — MissingMaps -LINK-

Buildings Data Life Cycle

  • acquisition
  • storage
  • review stages
  • feedback from stakeholders

Review Stages:
 Bldg Needs Review
 Bldg Reviewed – OK
 Bldg Reviewed – Flag Serious Errors
 Bldg Reviewed – Flag needs improvement (re-review)

Coverage Assesment:
 CoStar coincident
 CoStar no match count

Team

Ten Weeks Left — Making the Most of the Time
* What are the blockers to getting Garlynn started ?
– NORESCO agreement / invoicing
– confidentiality
– other?

Server i7d
– postgresql / postgis network config
– geoserver config
– NAIP mapserver config
storage ! 2x1TB | 2x3TB | 2x4TB | RAID1 boot

dbb@i7d:
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0p1      218G  122G   85G  60% /
/dev/sdg1       2.7T  1.2T  1.5T  45% /mnt
/dev/sdf1       917G  167G  705G  20% /mnt
/dev/sdd        2.7T   73M  2.6T   1% /mnt
/dev/sdb1       3.6T  2.7T  765G  79% /mnt
/dev/sdc1       3.6T  942G  2.5T  28% /mnt
/dev/sdh1       917G  647G  224G  75% /mnt

01 Jan 17
Data Integration pre-Setup — California AB 802 Support
========================================================

* Rebuilt i7d Linux Server
– PostgreSQL 9.6
– PostGIS 2.3
– GEOS 3.6
– osm2pgsql 0.91(p)

new drives to replace four year old disks… lots of file management.. deferred librarianship

* First experiments with Berkeley ImageSEG

21 Dec 16
Data Integration pre-Setup — California AB 802 Support
========================================================

* Berkeley ImageSEG
– email license terms with James

* Similarity Measures JTS Interface
-LINK-

* Example Work Context
– EPA POINT in Davis, California -LINK-