![\"spark_logo_sm\"](\"http:\/\/blog.light42.com\/wordpress\/wp-content\/uploads\/2014\/11\/apache_spark_logo_sm.png\"){kind=logo}
<\/p>\n<\/p>\n
At a minimum, suffice it to say I participated online in roughly twelve hours of lecture and lab on Nov 20 and 21, 2014 at AmpCamp 5<\/a><\/strong> (I also attended one in Fall 2012). I put an emphasis on python, IPython Notebook<\/a>, and SQL.<\/p>\n Once again this year, the camp mechanics went very smoothly — readable and succinct online exercises<\/a>; Spark docs<\/a>; Spark python, called pyspark<\/a> is advancing, although some interfaces may not be available to python yet; Spark SQL<\/a> appears to be useable.<\/p>\n To setup on my own Linux box, I unzipped the following files: The resulting directories provided a pre-built Spark 1.1 The Lab exercises<\/a> are almost all available as both Scala and python. Tools to do the first labs: <\/p>\n and for extra practice <\/p>\n IPython Notebook<\/strong><\/p>\n An online teaching assistant (TA) suggested a command line to launch the Notebook – here are my notes: ##-- TA suggestion ##-- a server already setup with a Notebook, options ##-- COMBINE The IPython Notebook worked ! Lots of conveniences, interactivity and viz potential immediately available against the pyspark environment. I created several Notebooks in short order, to test and explore, for example SQL<\/a>.<\/p>\n
\nampcamp5-usb.zip ampcamp-pipelines.zip training-downloads.zip<\/code><\/p>\n
\nUsing Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_65)<\/code><\/p>\n$SPARK_HOME\/bin\/spark-shell $SPARK_HOME\/bin\/pyspark<\/pre>\n
$SPARK_HOME\/bin\/spark-submit $SPARK_HOME\/bin\/run-example<\/pre>\n
\n<\/p>\n
\nIPYTHON_OPTS=\"notebook --pylab inline\" .\/bin\/pyspark --master \"local[4]\"<\/p>\n
\n--matplotlib inline --ip=192.168.1.200 --no-browser --port=8888<\/p>\n
\nIPYTHON_OPTS=\"notebook --matplotlib inline --ip=192.168.1.200 --no-browser --port=8888\" $SPARK_HOME\/bin\/pyspark --master \"local[4]\"
\n<\/code><\/p>\n