{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"clearfix\" style=\"padding: 10px; padding-left: 0px; padding-top: 40px\">\n",
    "<img src=\"https://www.dropbox.com/s/y8dd1z3sl4uofep/lipd_logo.png?raw=1\" width=\"700px\" class=\"pull-right\" style=\"display: inline-block; margin: 0px;\">\n",
    "</div>\n",
    "\n",
    "## Welcome to the LiPD Quickstart Notebook!\n",
    "\n",
    "This Notebook was created to help you get familiar with the commands you can use in LiPD. Follow each step and experiment as much as you'd like! Each step will help prepare you for the next step. \n",
    "\n",
    "### Table of Contents\n",
    "  * [Getting started with LiPD](#lipd)\n",
    "  * [LiPD Functions](#lipdfns)\n",
    "  * [Excel Converter](#excel)\n",
    "  * [NOAA Converter](#noaa)\n",
    "  * [DOI Updater](#doi)\n",
    "  * [Load LiPDs](#loadlipds)\n",
    "  * [Viewing LiPD data](#lipddata)\n",
    "  * [Extract TimeSeries](#timeseries)\n",
    "  * [Pandas](#pandas)\n",
    "  * [Remove files](#removelipds)\n",
    "  * [Glossary](#glossary)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Getting started with LiPD <a id=\"lipd\"></a>\n",
    "\n",
    "Now that you've covered some of the basics, let's look how to get started with the LiPD package. For this tutorial we will be using the example files included in the default Notebooks folder.\n",
    "\n",
    "1. Import the LiPD package from your system\n",
    "2. Run the cell\n",
    "3. A browse dialog window will ask you where your files are stored\n",
    "4. Choose option \"2\" and browse to `/<your_username>/LiPD_Notebooks/examples`\n",
    "\n",
    "<div class=\"alert alert-warning\" role=\"alert\" style=\"margin: 10px\">\n",
    "<p>**NOTE**</p>\n",
    "<p> The Notebooks folder is created in the root of the user directory on your computer. You may find it at `/<your_username>/LiPD_Notebooks`.\n",
    "\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Import the LiPD package and locate your files\n",
    "from lipd.start import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  LiPD functions <a id=\"lipdfns\"></a>\n",
    "\n",
    "What happened? \n",
    "* All the LiPD commands are now available in Jupyter\n",
    "* A target folder containing files has been set\n",
    "\n",
    "There are a few paths you can take from here. It all depends on what you'd like to accomplish and what files you provided. Look at the sections below to see about the specific processes. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Excel Spreadsheet Converter <a id=\"excel\"></a>\n",
    "\n",
    "-----\n",
    "\n",
    "Microsoft Excel spreadsheets (`.xls`, `.xlsx`) must be converted to LiPD before any data analysis can be done. The converter is designed to parse data from the spreadsheet template provided. Please insert your data in this template format to ensure a complete and accurate conversion to LiPD. Use the example file as a reference for correct formatting.\n",
    "\n",
    "Template File: [Excel_Template.xlsx](https://dl.dropboxusercontent.com/s/r9fdxko5yvirho1/excel_template.xlsx?dl=1)<br>\n",
    "\n",
    "<div class=\"alert alert-warning\" role=\"alert\" style=\"margin: 10px\">\n",
    "<p>**NOTE**</p>\n",
    "<p> The Excel converter will run for every `.xls` and `.xlsx` file in the target folder previously chosen.\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "excel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  NOAA Converter <a id=\"noaa\"></a>\n",
    "\n",
    "-----\n",
    "\n",
    "National Oceanic and Atmospheric Administration (NOAA) text files (`.txt`) must be converted to LiPD before any data analysis can be done. The converter is designed to parse data from the NOAA template provided. Please insert your data in this template format to ensure a complete and accurate conversion to LiPD. Use the example file as a reference for correct formatting.\n",
    "\n",
    "Template File: [NOAA_Template.txt](https://dl.dropboxusercontent.com/s/h09at89774ajpi7/noaa-blank.txt?dl=1)<br>\n",
    "\n",
    "<div class=\"alert alert-warning\" role=\"alert\" style=\"margin: 10px\">\n",
    "<p>**NOTE**</p>\n",
    "<p> The NOAA converter will run for every `.txt` file in your working directory.\n",
    "\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "noaa()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  DOI Updater <a id=\"doi\"></a>\n",
    "\n",
    "-----\n",
    "\n",
    "The DOI updater will take your LiPD files and update them with the most recent information provided by [doi.org](doi.org). The updater will run once per LiPD, and will skip any LiPDS that have already been updated previously. \n",
    "\n",
    "<div class=\"alert alert-warning\" role=\"alert\" style=\"margin: 10px\">\n",
    "<p>**NOTE**</p>\n",
    "<p> Running `doi()` before running `loadLipds()` **will not** update your current workspace.  Please run `doi()` at the beginning of your workflow.\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "doi()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Loading LiPD files <a id=\"loadlipds\"></a>\n",
    "\n",
    "-----\n",
    "\n",
    "`loadLipds()`<br>\n",
    "* Import all the the LiPD files from your target folder into a collection called a LiPD Library. The LiPD library holds data for each LiPD file.\n",
    "<br>\n",
    "\n",
    "`showLipds()`<br>\n",
    "* Show the names of the LiPD files in the current LiPD Library.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "loadLipds()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Viewing LiPD Data  <a id=\"lipddata\"></a>\n",
    "\n",
    "----\n",
    "\n",
    "`getCsv(filename)`<br>\n",
    "* Returns: dictionary\n",
    "* Show the CSV data for the target LiPD file.\n",
    "<br>\n",
    "\n",
    "`getMetadata(filename)`<br>\n",
    "* Returns: dictionary\n",
    "* Show the metadata for the target LiPD file.\n",
    "<br>\n",
    "\n",
    "`showMap(filename)`<br>\n",
    "* Show a map with location markers for one, several, or all LiPDs. \n",
    "\n",
    "<div class=\"alert alert-warning\" role=\"alert\" style=\"margin: 10px\">\n",
    "<p>**NOTE**</p>\n",
    "    <p> showMap() can display one, multiple, or all locations.</p>\n",
    "    <ul>\n",
    "        <li>One location: `showMap(\"ODP1098B.lpd\")`<br>\n",
    "        <li>Multiple locations: `showMap(\"ODP1098B.lpd, SAm-LagunaChepicdeJong2013.lpd\")` * work in progress *\n",
    "        <li>All locations: `showMap()`\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "lipd_c = getCsv(\"ODP1098B.lpd\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Call the variable to show the raw csv data\n",
    "lipd_c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "lipd_m = getMetadata(\"ODP1098B.lpd\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Call the variable to show the raw metadata\n",
    "lipd_m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "showMap(\"ODP1098B.lpd\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "showMap(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extract TimeSeries  <a id=\"timeseries\"></a>\n",
    "\n",
    "----\n",
    "\n",
    "Extract TimeSeries is able to pull LiPD data from the LiPD Library and put it into a data structure that makes sense for analyzation and computation. Similar to our LiPD Library, this command will create a TimeSeries Library that holds many entries.\n",
    "\n",
    "`extractTimeSeries()`<br>\n",
    "* Returns: dictionary\n",
    "* Creates a TimeSeries from the data in the LiPD Library.\n",
    "<br>\n",
    "\n",
    "`showTsos()`<br>\n",
    "* Shows the names of the TimeSeries objects in the TimeSeries Library.\n",
    "\n",
    "`find(expression, time_series)`<br>\n",
    "* Returns: Names of matching records(array), Filtered TimeSeries (dictionary)\n",
    "* Find all TimeSeries objects that match a certain criteria. \n",
    "\n",
    "\n",
    "`exportTimeSeries()`\n",
    "* Returns: none\n",
    "* Exports the TimeSeries Library back into the LiPD Library. \n",
    "\n",
    "<div class=\"alert alert-warning\" role=\"alert\" style=\"margin: 10px\">\n",
    "<p>**WARNING**</p>\n",
    "    <p>`exportTimeSeries()` will overwrite the contents of the LiPD Library</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "time_series = extractTimeSeries()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "showTsos(time_series)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "names, new_time_series = find(\"geo_meanLat > -50\", time_series)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# This should match the SAm-LagunaChepicdeJong2013 time series objects\n",
    "names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "names, new_time_series = find(\"archiveType is marine sediment && geo_meanElev <= -1000 && geo_meanElev > -1100\", time_series)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# This should match the ODP1098B time series objects\n",
    "names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Transparent function. No output.\n",
    "collapseTimeSeries()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pandas  <a id=\"pandas\"></a>\n",
    "\n",
    "----\n",
    "\n",
    "`lipd_to_df(filename)`\n",
    "* Returns: data frame(s) (dictionary)\n",
    "* Creates a collection of pandas data frames from LiPD data\n",
    "\n",
    "`ts_to_df(time_series, filename)`\n",
    "* Returns: data frame(s) (dictionary)\n",
    "* Creates a collection of pandas data frames from a TimeSeries object. The CSV data frame will be plot with depth, age, and year columns when available.\n",
    "\n",
    "<div class=\"alert alert-warning\" role=\"alert\" style=\"margin: 10px\">\n",
    "<p>**NOTE**</p>\n",
    "    <p>After creating the data frames, calling a specific data frame variable will display the formatted data. </p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Construct data frames from data in the LiPD file\n",
    "dfs_lipd = lipd_to_df(\"P178-15P.Tierney-1.2015.lpd\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print the available data frames\n",
    "showDfs(dfs_lipd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Call the collection to show raw text\n",
    "dfs_lipd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Calling the data frame objects will show a formatted data frame\n",
    "dfs_lipd[\"metadata\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dfs_lipd[\"paleoData\"][\"MD982181.Khider.2014.data.PaleoData.csv\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dfs_lipd[\"chronData\"][\"MD982181.Khider.2014.Chron1.ChronMeasurementTable.csv\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\" role=\"alert\" style=\"margin: 10px\">\n",
    "<p>**NOTE**</p>\n",
    "    <p>TimeSeries objects are not able to use the autocomplete feature. Be sure to run `showTsos()` and copy/paste the TimeSeries object of interest</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Construct data frames from data in one time series object\n",
    "dfs_ts = ts_to_df(time_series, \"ODP1098B_data_SST\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print the available data frames\n",
    "showDfs(dfs_ts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Call the collection to show raw text\n",
    "dfs_ts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Calling the data frame objects will show a formatted data frame\n",
    "dfs_ts[\"metadata\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dfs_ts[\"paleoData\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dfs_ts[\"chronData\"][\"ODP1098B.Chron1.ChronMeasurementTable.csv\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Removing LiPDs <a id=\"removelipds\"></a>\n",
    "----\n",
    "\n",
    "`removeLipd(filename)`\n",
    "* Remove one LiPD entry from the LiPD Library\n",
    "\n",
    "`removeLipds()`\n",
    "* Remove all LiPD entries from the LiPD Library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "removeLipds()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Now try to showLipds() to see that the LiPD Library is empty\n",
    "showLipds()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Saving <a id=\"saving\"></a>\n",
    "----\n",
    "\n",
    "`saveLipds()`\n",
    "* Saves each LiPD in the LiPD Library back to your computer as a LiPD file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "saveLipds()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Glossary <a id=\"glossary\"></a>\n",
    "\n",
    "----\n",
    "\n",
    "<div style=\"background-color: #f2f2f2\">\n",
    "<dl style=\"padding-top=20px\">\n",
    "\n",
    "<dt style=\"margin-left: 0.5em\"> DOI </dt><br>\n",
    "<dd style= \"margin-left: 2em\"> A Digital Object Identifier is a unique alphanumeric string assigned by a registration agency to identify content and provide a persistent link to its location on the Internet. The publisher assigns a DOI when your article is published and made available electronically. </dd><br>\n",
    "\n",
    "<dt style=\"margin-left: 0.5em\"> Environment / Workspace </dt><br>\n",
    "<dd style= \"margin-left: 2em\"> The current state of the Notebook. Variables and modules in the Notebook are constantly changing, and all of these contribute to the state of the workspace. </dd><br>\n",
    "\n",
    "<dt style=\"margin-left: 0.5em\"> LiPD </dt><br>\n",
    "<dd style= \"margin-left: 2em\"> Refers to the LiPD package or LiPD files, depending on the context.  </dd><br>\n",
    "\n",
    "<dt style=\"margin-left: 0.5em\"> LiPD Library </dt><br>\n",
    "<dd style= \"margin-left: 2em\"> A collection of LiPD file data. </dd><br>\n",
    "\n",
    "<dt style=\"margin-left: 0.5em\"> Module </dt><br>\n",
    "<dd style= \"margin-left: 2em\"> A set of related functions that is imported for use in the Notebook.</dd><br>\n",
    "\n",
    "<dt style=\"margin-left: 0.5em\"> NOAA </dt><br>\n",
    "<dd style= \"margin-left: 2em\"> National Oceanic and Atmospheric Administration. This document uses NOAA to signify a specific text file format used by the organization. </dd><br>\n",
    "\n",
    "<dt style=\"margin-left: 0.5em\"> Notebook </dt><br>\n",
    "<dd style= \"margin-left: 2em\"> Jupyter uses Notebooks as a way to save a single session of workflow and scientific computations. This Quickstart Notebook is for learning and documentation, though you may later create your own Notebooks with graphs, functions, and various datasets. </dd><br>\n",
    "\n",
    "<dt style=\"margin-left: 0.5em\"> Magic Commands </dt><br>\n",
    "<dd style= \"margin-left: 2em\"> Special built-in Jupyter commands that provide common useful functions that \"magically\" work. </dd><br>\n",
    "\n",
    "<dt style=\"margin-left: 0.5em\"> TimeSeries Library </dt><br>\n",
    "<dd style= \"margin-left: 2em\"> A collection of TimeSeries data and objects. </dd><br>\n",
    "\n",
    "<dt style=\"margin-left: 0.5em\"> TimeSeries Object </dt><br>\n",
    "<dd style= \"margin-left: 2em\"> An extracted piece of the TimeSeries from a LiPD file. </dd><br>\n",
    "\n",
    "</dl>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
