- {
- “cells”: [
- {
“cell_type”: “raw”, “metadata”: {
“papermill”: {
- <<<<<<< Updated upstream
- “duration”: 0.013553, “end_time”: “2020-02-11T21:00:30.433971”, “exception”: false, “start_time”: “2020-02-11T21:00:30.420418”,
“exception”: false, “start_time”: “2020-02-11T23:09:46.832799”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “raw_mimetype”: “text/restructuredtext”, “tags”: []
}, “source”: [
“.. _preprocessing:”]
}, {
“cell_type”: “markdown”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.009143, “end_time”: “2020-02-11T21:00:30.453308”, “exception”: false, “start_time”: “2020-02-11T21:00:30.444165”,
“exception”: false, “start_time”: “2020-02-11T23:09:46.857003”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “tags”: []
}, “source”: [
“# Features preprocessing”]
}, {
“cell_type”: “code”, “execution_count”: 1, “metadata”: {
“nbsphinx”: “hidden”, “papermill”: { - <<<<<<< Updated upstream
- “duration”: 2.487577, “end_time”: “2020-02-11T21:00:32.948821”, “exception”: false, “start_time”: “2020-02-11T21:00:30.461244”,
“exception”: false, “start_time”: “2020-02-11T23:09:46.875302”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “tags”: []
}, “outputs”: [], “source”: [
“# Hidden notebook set-upn”, “n”, “import os, sysn”, “import numpy as npn”, “import pandas as pdn”, “import xarray as xrn”, “import matplotlib.pyplot as pltn”, “%matplotlib inlinen”, “sys.path.insert(0, os.path.abspath(‘/Users/gmaze/git/github/gmaze/pyxpcm’))n”, “n”, “import pyxpcmn”, “from pyxpcm.models import pcmn”, “n”, “import seaborn as snsn”, “import cartopy.crs as ccrsn”, “import cartopy.feature as cfeaturen”, “import matplotlib.ticker as mtickern”, “import matplotlib as mpln”, “n”, “# Load sample data:n”, “ds = pyxpcm.tutorial.open_dataset(‘isas_snapshot’).load()n”, “n”, “# Define vertical axis and features to use:n”, “z = ds[‘depth’].where(ds[‘depth’]>=-1200, drop=True)n”, “features_pcm = {‘TEMP’: z, ‘TEMP’: z}n”, “n”, “m = pcm(K=3, features=features_pcm)”]
}, {
“cell_type”: “raw”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.006653, “end_time”: “2020-02-11T21:00:32.963525”, “exception”: false, “start_time”: “2020-02-11T21:00:32.956872”,
“exception”: false, “start_time”: “2020-02-11T23:09:49.663360”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “raw_mimetype”: “text/restructuredtext”, “tags”: []
}, “source”: [
“The Profile Classification Model (PCM) requires data to be preprocessed in order to match the model vertical axis, to scale feature dimensions with each others and to reduce the dimensionality of the problem. Some of these steps are mandatory and they all can be user parameterised.n”, “n”, “The PCM preprocessing operations are organised into 4 steps:n”, “n”, “.. image:: _static/Preprocessing_pipeline_2lines.pngn”, ” :width: 100%n”, ” :align: center”]
}, {
“cell_type”: “markdown”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.006608, “end_time”: “2020-02-11T21:00:32.977136”, “exception”: false, “start_time”: “2020-02-11T21:00:32.970528”,
“exception”: false, “start_time”: “2020-02-11T23:09:49.677573”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “tags”: []
}, “source”: [
“## Stack”]
}, {
“cell_type”: “markdown”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.00741, “end_time”: “2020-02-11T21:00:32.991539”, “exception”: false, “start_time”: “2020-02-11T21:00:32.984129”,
“exception”: false, “start_time”: “2020-02-11T23:09:49.691422”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “tags”: []
}, “source”: [
“This step mask, extract, flatten and transform any ND-array set of feature variables (eg: temperature, salinity) into a plain 2D-array collection of vertical profiles usable for machine learning methods.”]
}, {
“cell_type”: “markdown”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.006983, “end_time”: “2020-02-11T21:00:33.006119”, “exception”: false, “start_time”: “2020-02-11T21:00:32.999136”,
“exception”: false, “start_time”: “2020-02-11T23:09:49.704866”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “tags”: [], “toc-hr-collapsed”: false
}, “source”: [
“### Mask”]
}, {
“cell_type”: “raw”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.00649, “end_time”: “2020-02-11T21:00:33.019490”, “exception”: false, “start_time”: “2020-02-11T21:00:33.013000”,
“exception”: false, “start_time”: “2020-02-11T23:09:49.719397”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “raw_mimetype”: “text/restructuredtext”, “tags”: []
}, “source”: [
“This step computes a mask of the input data that will reject all profiles that are full of nans over the depth range of feature vertical axis. This ensure that all feature variables will be successfully retrieved to fill in the plain 2D-array collection of profiles.n”, “n”, “This operation is conducted bypyxpcm.xarray.pyXpcmDataSetAccessor.mask(), so that the mask can be computed (and plotted) this way:”]
}, {
“cell_type”: “code”, “execution_count”: 2, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.021824, “end_time”: “2020-02-11T21:00:33.048361”, “exception”: false, “start_time”: “2020-02-11T21:00:33.026537”,
“exception”: false, “start_time”: “2020-02-11T23:09:49.732762”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “raw_mimetype”: “text/restructuredtext”, “tags”: []
}, “outputs”: [
- {
“name”: “stdout”, “output_type”: “stream”, “text”: [
“<xarray.DataArray ‘pcm_MASK’ (latitude: 53, longitude: 61)>n”, “dask.array<eq, shape=(53, 61), dtype=bool, chunksize=(53, 61), chunktype=numpy.ndarray>n”, “Coordinates:n”, ” * longitude (longitude) float32 -70.0 -69.5 -69.0 -68.5 … -41.0 -40.5 -40.0n”, ” * latitude (latitude) float32 30.023445 30.455408 … 49.41288 49.737103n”]
}
], “source”: [
“mask = ds.pyxpcm.mask(m)n”, “print(mask)”]
}, {
“cell_type”: “code”, “execution_count”: 3, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.221273, “end_time”: “2020-02-11T21:00:33.276820”, “exception”: false, “start_time”: “2020-02-11T21:00:33.055547”,
“exception”: false, “start_time”: “2020-02-11T23:09:49.762424”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “raw_mimetype”: “text/restructuredtext”, “tags”: []
}, “outputs”: [
- {
- “data”: {
“image/png”: “iVBORw0KGgoAAAANSUhEUgAAAYUAAAELCAYAAAA2mZrgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de7xcVX3+8c+TiKgIAQQrWFQU1CoXAQGhASLeEMErlovVCqhIrZaLBakKwapoFYvFVtDyA7Qi8VIUQcALBBOh0XBVEAveqkKxhYAGBJJznt8fe5+Tncnk7H0mM+fMzHner9d+ndmXtfd3wjDfWWvtvZZsExERATBrugOIiIj+kaQQERHjkhQiImJckkJERIxLUoiIiHFJChERMS5JISKiD0haX9I1ku6TdFCb/dtIuro85qhexZGkEBHRH1YArwPOWMv+jwDHAXsDR0jatBdBJClERPQB26O275rgkG1tX2d7JbAQ2K0XcTyqFyedDpLyaHZENGZb61L+aVut51/9ZmWTQ+8H5rRsO9X2/EleshrvMqAnNYWhSQoAL16zGS5iIF1x503THcJQm73F7et8jl/9ZiUjd23b5Fpz1jUBlUYrrzcGbuzCOdeQ5qOIiA6t8MrapYtul7SzpNnAPsAPunnyMUNVU4iImEqjdLfVWtJXgZ2A5ZL2AL4NzLG9ADgJOAdYDzjP9r1dvXgpSSEiokOjq7XorDvbr5tg3x0UNYSeSlKIiOjQyBBOPZCkEBHRoW43H/WDJIWIiA6t6HLzUT9IUoiI6FCajyIiYtzw1ROSFCIiOjaSPoWImAov23LH1dbzhHN/Ghm+nJCkEBHRqTQfRUTEuBVdGdKovyQpRER0aIQkhYiYBlPVx9B6nU7NlD6QJIWIiBg3muajiIgYk5pCRPSFbjXz9MpMuaV2hWdPdwhdl6QQEdGh1BQiImLciIdv8sokhYiIDo0O4YzGSQoRER0axuajKUlzkuZKsqTNJJ0haWG53CPplS3HPq3cPnbMHlMRY0TEZI14Vu0yaKaqpnAssBTA9jEAkh4F3AZ8q83x19o+YIpii4joyAqG7+6jnqcxSQcCi4EHWna9BFhs+6E2xXaVtEjSWZI26HWMERGdSE1hkiTNAo4GXgu8qmX3YcDn2xS7C3iG7eWSTgBOBE5uOe984JSuBxwRPdHuuYpheHZhGDuae/2ODgMubq0NSHos8ALgu60FbD9se3m5eiGwS5tj5ttWdelB7BERExqxapdB0+uksD1wkKTLgR2ABeX2A4HLbI+0FpC0UWV1HnBHj2OMiOjICLNql0HT0+Yj2yeOvZa0EDi4XD0U+Gj1WEnn2j4c2FvSqcDycjm8lzFGRHRqdAD7DOpM2XMKtudVXr+mzf7Dy7+XAJdMVVwREZ16JGMfRUTEmGHsaE5SiIjo0CDeclonSSEiem4Ybj9tZ3QIh7lIUoiI6FBqChERMS6T7ERExLhBfA6hTpJCREzKsPYPdCLPKURExLhhnE8hSSEiokOpKUTEQGtt+mkdvTRNQ5MzjDWF4UtzERFTZMXoo2qXpiQdJekaSVdL2qZl336Slpb7P9X1N1KRpBAR0aFRVLs0IWlT4AhgL+B44LSWQ94PvNb2nsDTJO3QzfdRleajiIgOdfHhtd2Bq8rpBJZKembL/h8BG0v6LfAY4N5uXbhVkkLEkGrSP5A+hHUz2nASHUlu2XSq7fmV9U2A+6pFWo7/KnA58CBwue3fTC7S5pIUIiI61PThtQazQy6jmJRszGjL/jMpZqG8G7hA0p/b/n7TOCcjfQoRER0atWqXhpYA8yTNlrQzcHvL/pXA/bZHKRLIJt17F6tLTSEiokPdGvvI9r2SzgcWASuAIyXtB8yxvQD4ILBQ0iPAf1M0JfVEkkLEkEj/wNSbRE2glu2zgLMqm+6o7LsQuLBrF5tAkkJERIeG8YnmKXlHkuZKsqTNJM2XdIukhZK+vpbjPyBpsaTLJG0+FTFGREzWCKpdBs1UpbljgaWV9VNsz7P9qtYDJW0H7GR7LnAOcMIUxRgRMSld7GjuGz1vPpJ0ILAYqCaA90n6W+Bs2//eUmQv4NLy9aUUT/dFREX6D/rDyiGcZKenNQVJs4CjgU9XNp9p+3nAy4G/lrRtS7Hxhzhs/xHYoM1555fNUeNLb95BRMTajVi1y6CZMClIenSDZb0JTnEYcLHth8Y22L6n/Luc4raq57WUWQbMKa//GOCB1pPanm9b1aXRu42I6KJRz6pdBk1d89HvgTspHrlu92tcwGzgKWspvz2wi6RXAzsACyS91vb9kmYDc4FvtJT5HvBh4Gxgf4qmp4iIvjOIfQZ16pLCT2zvNNEBkm5Y2z7bJ1aOWwgcDJwu6TkUyeRi2zeU+8+1fbjtWyTdJGkxsBx4Y7O3EjNBt9rSW+cRiOhE01FQB8mESaEuITQ9pjxuXvnyLWvZf3jl9cnAyU3OGxExXWZiTWGcpGcAT62Wsf2tXgQVETEIVo4O391HjZKCpH8B9gRuZtXofQaSFKKrpupWyybXSRNT1JlxzUcV+wLPLUfoi4gIhrP5qOn9Ukspmo4iIqI0455olnQtRTPR+sCtkm4FHqa8RbWcLzQiYkYaxC/9OnXNR4dMSRQxIwza0Ayt8aaPIVrNuKRg+1cAkr5o+9DqPklfBA5tWzAiYgZYOYBPLNepaz6aBawHPLsczmIsLW7EmsNTRETMKDOupgC8EzgG2BL4KauSwh8ohqGIiJixZlxSsP1JSZ8C3m37o1MUUwygQesv6ES79zhV/Qwz4d93EA1jUqhtELM9wupzIUREBGCrdhk0TXtJLpP0VkmbVIfN7mlkERF9bqVn1S6DpukTzUeUf/++ss3A07sbTkTE4BjEmkCdRknB9ta9DiT6R9qvm8uzDDPbMPYpNB0Q77EUdyHNLTctBs4op8uMiJiRhrGm0LTB61xgY+A9wEkUzymc26ugIiIGwYwb+6ji2barQ17cLOnGXgQUETEo3G6S4gHXNCn8XtIBti8BkPQKivmbY8Ckv6C3ujVPQ/47DYaRAby7qE7TpHA48M+Szqa46+jGcltExIw1iM1DdZreffQz4BWdXkTSXGARsDlwPPDCctfXbH+k5dinAdcBPyo3nWT72k6vHRHRKzO2+UjSk4G3s+YczYc1vM6xFBP1APw/2yeVg+0tlvQF279uOf5a2wc0PHdExLQYxruPmjYffR24ALiSVXM0NyLpQIpbWF8FYPv28u+opBFgpE2xXSUtAm4Bjrf9wGSuOQzSpjy88t92eAxjUmjcS2L7E7avsn312FJXpqwNHA18us2+w4Cf2b6zZdddwDNs7wX8HDixTdn5klxdmr6PiIhuGRlV7TJomiaF70h6r6TtJT1zbGlQ7jDgYtsPVTdKeiFwJEXCWI3th20vL1cvBHZpc8x826ouDd9HRETXDOOAeE2bj3Yv/764ss3AvjXltgd2kfRqYAdggaR3A6cB+7d7IlrSRrbHbnedB9zRMMaIiCk1iF/6dZreffTCifZLOtr2Gk1Etk+sHLMQOBi4hOKJ6P+QBHCM7RslnWv7cGBvSacCy8tl6G59TZtyxHAYxnbrpjWFOm+lTb9Ble155csXrGX/4eXfSygSR0REX+tmTUHSUcBfASuAI23fUdn3eOBM4CnArLof6uuiW4/jDV8dKiKijhssDUjalGKKgr0onuU6reWQU4BzbL+oQcvNbhPsO64ulm4lhWGsRUVETGh0VLVLQ7sDV9kesb0UaL2RZy5wiKSFkv6m5lxfkPSi6gYVzqTBLJrdaj5KTWEC6UOIGE5Nm4/a3DZ/qu35lfVNgPuqRVqO3xk4mWIKg29JutL2rWu53H7AJZL+3vZFkh4DLAAeBF5SF+ukkoKkJ9r+XZtdn5zMeSIihkLDpNDgtvllFHdrjml9SPj/gO/YtqQrge2AtknB9s/KmsI3y2GDDgGutH1Sk1gbNR9JeqmkW4Al5foOkj5XCeK8JueJiBgmdv3S0BJgnqTZknYGbm/ZvxjYqXz9fCa4VV/SSymSxkeB9wO/Bq4qv8dfWhdI05rCaRQdIN8FsH2zpJ0mLhIRMeS61Jtq+15J51MMHLoCOFLSfsAc2wsoJjj7bDkL5tW2r5/gdIdWXn+9ZZuBb00US9OksLIMGig6LUjnckTMcN28JdX2WcBZlU13VPb9gtUfHp7oPOv0bFfTu49+UPZ4P1rSC4Dzqck2ERHDzqOqXaaapAMlPaWy/kFJN0u6VNK2deWbJoVjKUYz/Qnwd8C1wAmdBBwRMTS69JxCl30I+F8ASQcBr6cYGeLLwGfrCjcd5mKlpAXAD2xfV45+GhExw/Xl3fijlXHlXgWcbfs64DpJ76or3HSSnTdS1BY2ArYBng38E/CyjkIecnkuIWKG6M+e1RWStgbupfiO/nBl32PrCjf9xX88sAfwB4DyoYktJhdnRMSQ6c/mo+Moxo/7KfAZ2z8BkPQy4Oa6wk3vPnrE9sNjT+VJenSHwUZEDI3p6EiuY3sR8Nw226+Q9MO68k1rCt+QdDqwoaRDgEuBz08q0oiIYdOfNYXVSNpU0pGSvgVM9HwD0Lyj+R8kvZziLe4KfML2ZesW6nBI/0HEDNank+xI2gR4HcWdR1sDmwP72V5SV7Y2KUiaDVxi++VAEkFERKkfZ4eXdAWwFfA14O/LO0Z/0SQhQIPmI9sjwAOSNlu3UCMihkx/Nh/9HlgfmANsNNkRKJp2ND8O+KmkxcADYxttHzaJQCMihksfNh/Zfr2kxwGvAP4aOBfYRNJLgIW2V0xUvmlS+Oi6hTkY0j8QEZPSOsB1n7D9IMUTzF8uE8SBwNuAz1HzOEHTjuar1yVASXMpRv/bnOIRwM8BG1KM8X1ym+M/AOxL8VzEm2z/77pcPyKiJ/qwT6FVmSAWAAvKBDGhpk803wU8kWLmHiiak/4b+AXwt7Z/VHOKY4Gl5esTgX+z/VVJ35C0ne0fV661HbCT7bnluB0nUIy3FBHRX/qw+UjSNTWH7DnRzqbNRxcBl9q+tLzo/sALKXq3Pwu8YIIAD6SYIGJsbtC5FBM/AHyzXP9xpcheFM9BUP49vmGMERFTqh/vPqJojXks8BXgYuD+yRRumhT2sv3XYyu2vynpH2z/XTn/Z/vIioHzjgZey6qk8LjKYE3LgKe3FNsE+Hl5nT9K2qDNeecDp7RuT59AREypPkwKtveQ9KfAQRRTJf+BIkF8zfbv68o3faL5DkmnSdpF0s6SPgz8uhzuYuUE5Q4DLrb9UGXbg5VEsjHFoE1VyyhupaI87oGW/dieb1vVpeH7iIgYerZ/Y/sM4EUUCeGfgLc0Kds0KbyB4sv7fcDJwH3lthFgojk/twcOknQ5sANFZ8ciYP9y//7letX3WvYvbhhjRMSU0qhqlymPSZol6cWSzqKY+3kH4ADbn2hSvundRw9KOofibqHx+RRsj7LmL/1quRMrgS4EDqZo7/q8pOPK891S7j/X9uG2b5F0U/lMxHLgjU1ijIiYcn3YfAT8DriT4pbUiyii3FDSSwFsr/sczd2YT8H2vMrqfm32H155fTJFjSQion/1Z1K4hCKyp7Nmn62pmUq5aUfz8cDuwH9CMZ+CpMynEBEzWj/efWT7zU2Ok3S07U+3bm/ap/CI7Ycp82LmU4iIoF/HPmrqre02Zj6FiIgOabR+6WNte8Ezn0JERKcG+274tvWYpn0KlEkgiSAiYkx/Nw/VmXxNoRzzaOxt/wmrj310t+0tuxZeRMSA6ceO5kn4ZLuNEyYF21sASPoUcFll7KMDKMYoioiYufo4KUjameLO0a0ovusF2PaeFC/Oa1euafPRPrb/ZmzF9iWSTl2niCMiBlyf1xS+SDHC9I+YxMwPTZPC7ZL+kWKYCoBDKIbOjoiYufr77qLf2b54soWaJoW/BN5BMfaRgO/TZpTSiIiZpM9rCidK+gqwEHhkbKPtz0xUqPHYR8DH1iW6iIiYUu8H/kgx42V3mo8knVf3yHSTYyIihlJ/1xSebHuHyRaqqym8UtIFE+wXsNtkLxoRMQz6vPnoIkkHU4xAUW0+emTtReqTwmsaXPifGxwTETF8+jspvKn8+xGKSMWq0VPXqu45hau7ElpExBDq57GNbG/dSbmmA+JFRESrPh4lVdL7JG1cWd9E0nvryiUpRER0SK5fGp9LOkrSNZKulrRNm/2zJd0m6d0NT/k62/eNrdheBhxUV6hxUpD0XEmvLl9vJGnzpmUjIoZSl2oKkjYFjqAYPuh44LQ2hx0B/GwS0c2WtEHlGo8H1qsr1HQ6zhOBfYGtga8BGwP/Duw9iQAjIoZL95qHdgeusj0CLJX0zOpOSY8BXgl8BXhCw3N+Clgk6cIy0kOAM+sKNa0pHAq8HHgAwPZ/A3PqCknaUtKSsjp0jaQdJL1H0sJy+a2kd7Up90DlmCZ3QEVETLmmk+xIcssyv+VUmwD3VU/dsv+dwNlMIg2VTy4fA6wPPAY4xvbZdeWaDnPxiO1RSWPTcdYmhNLdwB5l2X2BE2z/JcUtUki6HviPNuV+YXtew2tEREyPhl/Rdu1sPMuA7Svr4/c1ld+382x/TNKbm4Ym6WPAThStOwDzJV1n+4SJyjVNCp8pqyBPkPQeimrIx+sKlVWhMRsBN1QCfjZwv+3ftCm6laSrgd8Cf2v7fxvGGRExZbr48NoS4GRJs4Edgdsr+54NbCbpcuDJwHqSbrL97ZpzHgA8x/bYj/l/BW4G1j0p2P43SdcCL6So1rzB9i1Nykp6DvBvFGN6v66y6w0UQ7u283Tb90j6C+B0Vj2EMXbO+WRAvoiYbl1KCrbvlXQ+sAhYARwpaT9gju0FFH0OlDWFzRokBIAfA88CbivXnwUsrSukMom03yk9ZaLCZd9CI5KeB5xte+zN3QrMtX3vBGUE3Gh7xwbn98hd2zYNJyJmsNlb3N6kSWdCkrzd8Z+oPe7Hpx+3ztfqRNk8vz2rksKfATdSJJ3xyXZa1dUUFlDkwkdTVGluo+icfhZFU9CuNUGtb/vhcnUZ5XSeknYD7miXEMpbqB4qm572An5ZE2NExPTo72EuOrpJp26Yiz0Ayv6EI2zfXK7vCKxx11Abu0r6EEWniYDjyu2HAqsNtCfpDIomoW2Az0paDqwEjmr8biIiplA/D4hn+1edlGva0bz9WEIoL3aTpOc3CGoxsE+b7ce22XZM+fI6YOeGcUVETJ8+TgqdapoUrpG0APhSuX4w8IPehBQRMRj6uabQqaZJ4SiKO4fmUjQDfRn4aq+CiogYCDM4KTwK+Hq5VLdNOFlDRMRQm8FJ4aesevvrA38C/JpiLKSIiBmpn+dT6FTTh9dW+/IvO5nf3IuAIiIGxTD2KXQ0n4LtpcCfdzmWiIjB0seT7HSq6dDZp7Hq7c2ieJBtMuN6R0QMnWGsKTTtU7it8noEuBRY3P1wIiIGyAxOCpvZPr26QdLxFIPVRUTMTEOYFJr2KbyhzbY3djOQiIhB03SSnUEyYU1B0pHAW4BnSrqmsmtD4NZeBhYR0e80wSjTg6qu+egrwHeBU4GTK9uX276nZ1FFRAyC4csJtUlhju1fliOdrl/Zvr6kJ9j+rx7GFhHR12bi3UfvA95GMWF0KwP7dj2iiIhBMdOSgu23lS+Paq0VSHpmz6KKiBgAw1hTaHr30YUNt0VEzBgz8e6jnSmm3HyCpLdVdm1UVzYiYugNYU2h7ot9Q+BJFHM0b1HZvhx4fa+CiogYBMPYfFTXp3A1cLWkz9i+a4piiogYDDPwOYUxsyT9K/BnVG5Ntb3nRIUkbQlcBDwErAe8nWL+5fdTzMcA8HLbf2wpdxTwV8AK4EjbdzSMMyJiysy4mkLF54Ezgb2BQyjmUnioQbm7gT1sj0raFzgB+A7wadsfb1dA0qbAEcCewE7AaaSpKiL6kEamO4Lua3r30RzbFwGjtn9s+93APnWFbI/YHut/3wi4oXz9FkmLykH1Wu0OXFWWXQrk1teI6E9DOJ9C06TwsCQBP5f0ZkkvATZvUlDSc8pxk84EFgFfA55L8eDbXmUNomoT4L7qKdqcc74kV5eG7yMiomvk+mXQNE0KxwIbAO8E5gFH03A6Ttu3ln0PBwJn2r6vrAWsAP4D2KWlyDJgTmV9jTt9bc+3rerS8H1ERHSPXb8MmKZzNC8pXy5nEnMzS1rf9sPl6jLgQUlzbN9fbtsHuKSl2BLgZEmzKWZ4u73p9SIiptIg1gTq1D28di0TtIrV3X0E7FoOpjdK0Qx0HHC8pJdRzOB2PUVzEpLOAE6xfa+k8ymamlYARzZ8LxERU2umJQWKO406Znsxa3ZIX8/qw3CPHXtM5fVZwFnrcu2IiF7T6PBlhbqH1341VYFERAyaGdd8FBERE0hSiIiIMakpRETEKjOtTyEiIiYwfDmh8cNrERHRQqOuXRqfSzpK0jWSrpa0Tcu+88t9SyT9VdffSEVqChERHepWn0KDgUA/aPt2SesDN0m6oBwVouuSFCIiOtW95qPxgUCBpZJWGwjU9tjIDmOJoGcTfab5KCKiQ7JrF4DWATwlzW85Ve1AoKUTgQvL5NETqSlERHSq4e/1BoN2LgO2n+jMkt4APA84tGF0HUlNISKiQ13saF4CzJM0W9LOtAwEKmk/isFI31SZo6YnUlOIiOhUl4bGbjcQaJkI5theAJwH3AlcUUxtwyG2/6crF2+RpBAR0aFuPtHcZiDQOyr7ntS9K00sSSEiolMDOIlOnSSFiIgOqaet+9MjSSEiolOpKURExJgZN8lORERMIDWFiIgYlz6FiIgYo9QUJkfSlsBFwEPAesDbgX2AN1I8oHG97Xe2KfcA8MNy9ZO2L+plnBERHUlSmLS7gT1sj0raFzgBmA98yrYlXShpru3FLeV+YXtej2OLiFg3Q5gUejr2ke2RyjgdGwE32L7DHv+XXAm0G+1vq3KiiQskbd66U9L81lEHe/QWIiLWSiOuXQZNzwfEk/QcSdcAZ1KM6zG2fW/gibavbVPs6bb3Ab4GnN660/Z826ouvYo/ImKt7PplwPQ8Kdi+1faewIEUiQFJzwU+Chy2ljL3lC+/DOzY6xgjIjoyhEmh1x3N69t+uFxdBjwo6SkUI/79he3/a1NmA+ChchKJvYBf9jLGiIiODeCXfp1edzTvKulDFHfzCjgO+EfgCcC55RCwH7F9uaQzgFOAbYDPSlpO0edwVI9jjIjoTJ5TmJzyrqJ9WjYfspZjjylfXgfs3Mu4IiK6QaPDlxXy8FpERKcy9lFERIxLn0JERIxLUoiIiHFJChERMS59ChERMW603Sg9gy1JISKiU6kpRETEuPQpRETEuCSFiIgYl6QQERHjMsxFRESMS1KIiIhxufsoIiLGrJpteHgkKUREdCo1hYiIGJe7jyIiYtxIhrmIiIiSc/dRRESMG8Lmo1m9PLmkLSUtkXS1pGsk7SDpcZK+JGmRpLMlrRGDpKPK46+WtE0vY4yI6Nio65cB09OkANwN7GF7H+B9wAnAEcAS23sBK4CXVwtI2rQ8Zi/geOC0HscYEdEZj9YvA6anScH2iFfdyLsRcAPFl/2l5bZLy/Wq3YGryrJLgWf2MsaIiE551LVLUxO1kEjaptLiclTX30hFr2sKSHqOpGuAM4FFwCbAfeXuZcCmLUWq+wHU5pzzJbm69CD0iIgJeWSkdmmiQQvJR4DjgL2BI8rje6LnScH2rbb3BA6kSAzLgDnl7o2Be1uKVPcDrFH/sj3ftqpLD0KPiJhY95qP6lpItrV9ne2VwEJgt+69idX19O4jSevbfrhcXQY8CHwPeAXwU2B/4IqWYkuAkyXNBnYEbm96vdlbND40ImJd/eo7/spTGxx3f5vWjFNtz6+s17WQVNfbtbB0Ta9vSd1V0ocofu2LovpzG3CepEXArcBlAJLOAE6xfa+k8ymamlYARza50FhtQZIHqeaQeHsr8fbWTI7X9tO6cZ7SMmD7ynprFaO6vjFwYxevvRp5yO6znckf0qmQeHsr8fZWv8Zb9hFcCsylaCE5yfbrK/u/TNHPcBOwGHiF7dam967Iw2sREdOsXQuJpP2AObYXACcB5wDrAef1KiFAagrTLvH2VuLtrcQ7fHp+99E0OHW6A5ikxNtbibe3Eu+QGbqaQkREdG4YawoREdGhoUoKki6UtLBcHpC0Q7m9LwfYk/R4SedK+q6kq8ptT5N0T+V97DHdcY5pF2+5/QOSFku6TNLm0xljlaTzJF1X/jueXdn+QOXf9zXTGeOYCWLty88ugKTZkm6T9O5yvW8/u7BmvOW2vvzsTqehuvvI9iEAkjYDrrZ9c+Xx8T2BnShu63r92s8ypU4BzrG9uGX7tbYPmI6AaqwRr6TtgJ1sz5V0EMWgh383XQG2cVT5hGjVL2zPm45gaqwWa59/dqGI7Wct2/r1swst8Q7AZ3daDFVNoeL1wJfL1/08wN5c4JDyV9XfVLbvWg4tfpakDaYruDbaxds6wOHcaYls7f6ljPellW1blb+8L+izX4etsfbtZ1fSY4BXAl9p2dWXn921xNvvn91pMaxJ4VDggvJ17QB702hn4OvAi4HXSnoOcBfwjHJo8Z8DJ05jfK3axTv+72v7j0DffBEAx9veHfgL4OOSxsbUeno5nPvXgNOnLbrVtYu1nz+77wTOBqp3qvTzZ7ddvP382Z02A9d8JGkL4KI2uz5s+2JJWwGPsf1f5fa6x8d7aqJ4gf8DvmPbkq4EtrN9KzA2XtSFwKenJtLCZOOlMoBh+WvsgamKtbzmhJ8HANu/kzT2S/uHtu8pj/ky8N6pibSjWPv1s3saMM/2xyS9eWxjOc5ZP35228bLNH92+9XAJQXbdwEvmOCQQyk+kGM6HmCvGyaKV9IbKNqKrweeD3xT0ka2f18eMg+4YyriHDPZeIFbKBLG2RQDHLb2j/RUTbxzbN9f/g+/M/CrsknjIdsjFM0Hv+zXWCnav/vusytpd2AzSZcDTwbWk3QTxeRZfffZnSDe7zGNn91+NXBJoYGDKYbpBto/Pj5dgbXxHuCzkh5L0TF+vaQDJJ0KLC+Xw6c1wtWtES+ApJskLaaI943TGWCLL5bNMOsB/xR0tLgAAAUNSURBVFz+Ct+F4j0sB1YCPZ2wZBLWiBWgHz+7tpdQ9HdQ/vLezPa3+/Wzu7Z4y/V+/exOmzy8FhER44a1ozkiIjqQpBAREeOSFCIiYlySQkREjEtSiIiIcUkKERExLklhhipHtPzPLp/z7eXAYkg6RtKknoOR9JGWJ07rjl8h6UZJj26z70JJ8yZz/X4m6dXVUVIlfVPScknPns64YvgkKUTX2D7L9tiAY8fQ+4cj77H9PNuP9OLkkmZJ6pf/R14NjCcF2/sDraO/RqyzfvnAxzSStHn5y/NmSVdI+pNy+0JJH1Uxzv/15bhSSHqmpKXlr/RPSFpYbp9f1hbeDmwJLJG0oLVWomLugP3K10dL+q/yHE+vHLOHinHur5f0xXL4h7r38Q8qxsu/DNi0sv1ASUvKeM+sbP9gefzlkr49VrOQdLekj1N86W4xQfm3Svph+e920gRxbSjpC+WxS8qnqpH0qnL9BkmXlE80j9Wybiuftj1D0vMpRvj8lzKGDev+LSI6laQQUMxbe4XtHShGQf1gZd9y27sAXwDeXm47A3i/7efRZpA222cBdwK72z54bReV9GSKGsUuFF96zy+3Pxr4KHCA7Z2BnwBvmegNSNoNeBHFAHJHUo6Bo2Jo7HcBe5fxblAOx7Abxfg821GMs7975XRPBL5dXvuRtZTfjmK02N0pxoN6oaTq4HVV7wcW2N6VYmyufy23fw94ge2dgCtZ9e/7Xopx/ncETimHzb4YeEdZM/rDRP8WEetiGMc+isnbE5hfvv53imGGx1xc/r2B4ssTYEfbl5WvF1B+mXdgV4pRV/8AUP7CB3gWxZf7QkkAj6YYfG8iewIX2V4B3ClpUbn9BcAOFLUWgMcCN1E0xXzd9sqW46FIhFfUlN+6vOb15XGPB7YFftQmthcBL5P0gXJ94/LvVsCXJD2pPO/3y+3XA5+T9BWKJB0xZZIUAopx+tc2CNbYUMijtP+8NBnjf4TVa6Xrr6WsKn9/aPulNDfRub5u+22r7ZSOZfX3XC3/QMv2duXfBXza9ocbxvZy23e2bD8TONX2lZIOBN5Qbt8feCFwEHA0RY0mYkqk+Sig+IV6SPn6MOqHEL5Z0svK1wet5ZjlwFjb993An0raoGw3H5vh6gfAi8s29w2B/crttwFbjzXHlPu3bvAeXiNpPUlbVq7xn8CLyqaqsf6TLYBrgFdJelT5S/3P13LetZW/kmIWuo3L7U/Vqkl8Wn0HeMfYisq5w4GNgN+WndlvLPfNAv7U9neA44A/K4+t/ntG9ExqCgFF09H5kt4K/A/1QwgfC3xBxTDJi4DftznmHGCRpJtsHyzpDIomqJ8ANwPY/m25fSnFvAbfL7c/Iukw4GwV8x+You/hF2sLyPYPJH23PPdPKBNbOVz2O4CLJa1HUfN5s+0lZZPRjyiS0A/bvY8Jyv9Y0unA91S0K90PvK782+oDwKck3Uzx/9w3yjhPpWgWu5MiQT4ZmA1cUCZJsWoSoC+V/x7vBfZKv0L0SobOjkmT9DjbD5av3wc8yvb8aYjjf2w/aR3Kb2D7AUmbAdcCOw/Sl215x9bbbd823bHE8EhNITqxZ/kreRbFr/c3TVMcI5JuBHbr8FmFz0l6BsXENu8fsITwTYpbeFdOdywxXFJTiOiSsg/k8y2bb7DdFzOQRTSRpBAREeNy91FERIxLUoiIiHFJChERMS5JISIixiUpRETEuP8Pt/wCK5Ga1SwAAAAASUVORK5CYII=n”, “text/plain”: [
“<Figure size 432x288 with 2 Axes>”]
}, “metadata”: {
“needs_background”: “light”}, “output_type”: “display_data”
}
], “source”: [
“mask.plot();”]
}, {
“cell_type”: “markdown”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.007343, “end_time”: “2020-02-11T21:00:33.291512”, “exception”: false, “start_time”: “2020-02-11T21:00:33.284169”,
“exception”: false, “start_time”: “2020-02-11T23:09:49.999331”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “tags”: []
}, “source”: [
“### Ravel”]
}, {
“cell_type”: “raw”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.006591, “end_time”: “2020-02-11T21:00:33.305412”, “exception”: false, “start_time”: “2020-02-11T21:00:33.298821”,
“exception”: false, “start_time”: “2020-02-11T23:09:50.014331”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “raw_mimetype”: “text/restructuredtext”, “tags”: []
}, “source”: [
“For ND-array to be used as a feature, it has to be ravelled, flatten, along the N-1 dimensions that are not the vertical one. This operation will thus transform any ND-array into a 2D-array (samplingandvertical_axisdimensions) and additionnaly drop profiles according to the PCM mask determined above.n”, “n”, “This operation is conducted bypyxpcm.pcm.ravel().n”, “n”, “The output 2D-array is axarray.DataArraythat can be chunked along thesamplingdimension with the PCM constructor optionchunk_size:”]
}, {
“cell_type”: “code”, “execution_count”: 4, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.326244, “end_time”: “2020-02-11T21:00:33.638566”, “exception”: false, “start_time”: “2020-02-11T21:00:33.312322”,
“exception”: false, “start_time”: “2020-02-11T23:09:50.028681”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “tags”: []
}, “outputs”: [], “source”: [
“m = pcm(K=3, features=features_pcm, chunk_size=1e3).fit(ds)”]
}, {
“cell_type”: “markdown”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.007209, “end_time”: “2020-02-11T21:00:33.653698”, “exception”: false, “start_time”: “2020-02-11T21:00:33.646489”,
“exception”: false, “start_time”: “2020-02-11T23:09:50.359658”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “tags”: []
}, “source”: [
“By default,chunk_size='auto'.”]
}, {
“cell_type”: “code”, “execution_count”: 5, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.034957, “end_time”: “2020-02-11T21:00:33.696022”, “exception”: false, “start_time”: “2020-02-11T21:00:33.661065”,
“exception”: false, “start_time”: “2020-02-11T23:09:50.374384”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “tags”: []
}, “outputs”: [
- {
- “data”: {
- “text/html”: [
- “<pre><xarray.DataArray 'TEMP' (sampling: 2289, depth: 152)>n”, “dask.array<rechunk-merge, shape=(2289, 152), dtype=float32, chunksize=(1000, 152), chunktype=numpy.ndarray>n”, “Coordinates:n”, ” * depth (depth) float32 -1.0 -3.0 -5.0 -10.0 … -1960.0 -1980.0 -2000.0n”, ” * sampling (sampling) MultiIndexn”, ” - latitude (sampling) float64 30.02 30.02 30.02 30.02 … 49.74 49.74 49.74n”, ” - longitude (sampling) float64 -70.0 -69.5 -69.0 -68.5 … -41.0 -40.5 -40.0n”, “Attributes:n”, ” long_name: Temperature n”, ” standard_name: sea_water_temperaturen”, ” units: degree_Celsiusn”, ” valid_min: -23000n”, ” valid_max: 20000</pre>”
], “text/plain”: [
“<xarray.DataArray ‘TEMP’ (sampling: 2289, depth: 152)>n”, “dask.array<rechunk-merge, shape=(2289, 152), dtype=float32, chunksize=(1000, 152), chunktype=numpy.ndarray>n”, “Coordinates:n”, ” * depth (depth) float32 -1.0 -3.0 -5.0 -10.0 … -1960.0 -1980.0 -2000.0n”, ” * sampling (sampling) MultiIndexn”, ” - latitude (sampling) float64 30.02 30.02 30.02 30.02 … 49.74 49.74 49.74n”, ” - longitude (sampling) float64 -70.0 -69.5 -69.0 -68.5 … -41.0 -40.5 -40.0n”, “Attributes:n”, ” long_name: Temperature n”, ” standard_name: sea_water_temperaturen”, ” units: degree_Celsiusn”, ” valid_min: -23000n”, ” valid_max: 20000”]
}, “execution_count”: 5, “metadata”: {}, “output_type”: “execute_result”
}
], “source”: [
“X, z, sampling_dims = m.ravel(ds[‘TEMP’], dim=’depth’, feature_name=’TEMP’)n”, “X”]
}, {
“cell_type”: “raw”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.007559, “end_time”: “2020-02-11T21:00:33.711220”, “exception”: false, “start_time”: “2020-02-11T21:00:33.703661”,
“exception”: false, “start_time”: “2020-02-11T23:09:50.418226”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “raw_mimetype”: “text/restructuredtext”, “tags”: []
}, “source”: [
“See the chunksize of thedask.array.Arrayfor this feature.”]
}, {
“cell_type”: “markdown”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.007467, “end_time”: “2020-02-11T21:00:33.726199”, “exception”: false, “start_time”: “2020-02-11T21:00:33.718732”,
“exception”: false, “start_time”: “2020-02-11T23:09:50.433249”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “tags”: []
}, “source”: [
“### Interpolate”]
}, {
“cell_type”: “raw”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.007477, “end_time”: “2020-02-11T21:00:33.741072”, “exception”: false, “start_time”: “2020-02-11T21:00:33.733595”,
“exception”: false, “start_time”: “2020-02-11T23:09:50.447986”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “raw_mimetype”: “text/restructuredtext”, “tags”: []
}, “source”: [
“Even if input data vertical axis are in the range of the PCM feature axis, they may not be defined on similar level values. In this step, if the input data are not defined on the same vertical axis as the PCM, an interpolation is triggered. The interpolation is conducted following these rules:n”, “n”, “- If PCM axis levels are found into the input data vertical axis, then a simple intersection is used.n”, “- If PCM axis starts at the surface (0 value) and not the input data, the 1st non-nan value is replicated to the surface, as a mixed layer.n”, “- If PCM axis levels are not in the input data vertical axis, a linear interpolation through thexarray.DataArray.interp()method is triggered for each profiles.n”, “n”, “The entire interpolation processed is managed by apyxpcm.utils.Vertical_Interpolatorinstance that is created at the time of PCM instanciation.”]
}, {
“cell_type”: “markdown”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.007292, “end_time”: “2020-02-11T21:00:33.756155”, “exception”: false, “start_time”: “2020-02-11T21:00:33.748863”,
“exception”: false, “start_time”: “2020-02-11T23:09:50.462945”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “tags”: []
}, “source”: [
“Scalen”, “—–”]
}, {
“cell_type”: “raw”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.007415, “end_time”: “2020-02-11T21:00:33.771215”, “exception”: false, “start_time”: “2020-02-11T21:00:33.763800”,
“exception”: false, “start_time”: “2020-02-11T23:09:50.477820”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “raw_mimetype”: “text/restructuredtext”, “tags”: []
}, “source”: [
“Each variable can be normalised along a vertical level. This step ensures that structures/patterns located at depth in the profile, will be considered similarly to those close to the surface by the classifier.n”, “n”, “Scaling is defined at the PCM creation (pyxpcm.models.pcm) with the optionscale. It is an integer value with the following meaning:n”, “n”, ” - 0: No scalingn”, ” - 1: Center on sample mean and scale by sample stdn”, ” - 2: Center on sample mean only”]
}, {
“cell_type”: “markdown”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.00727, “end_time”: “2020-02-11T21:00:33.786008”, “exception”: false, “start_time”: “2020-02-11T21:00:33.778738”,
“exception”: false, “start_time”: “2020-02-11T23:09:50.493762”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “tags”: []
}, “source”: [
“## Reducen”, “n”, “[TBC]”]
}, {
“cell_type”: “markdown”, “metadata”: {
“papermill”: { - <<<<<<< Updated upstream
- “duration”: 0.007432, “end_time”: “2020-02-11T21:00:33.801243”, “exception”: false, “start_time”: “2020-02-11T21:00:33.793811”,
“exception”: false, “start_time”: “2020-02-11T23:09:50.508950”,
- >>>>>>> Stashed changes
- “status”: “completed”
}, “tags”: []
}, “source”: [
“## Combinen”, “n”, “[TBC]”]
}
], “metadata”: {
- “kernelspec”: {
- “display_name”: “obidam36”, “language”: “python”, “name”: “obidam36”
}, “language_info”: {
- “codemirror_mode”: {
- “name”: “ipython”, “version”: 3
}, “file_extension”: “.py”, “mimetype”: “text/x-python”, “name”: “python”, “nbconvert_exporter”: “python”, “pygments_lexer”: “ipython3”, “version”: “3.6.7”
}, “papermill”: {
- <<<<<<< Updated upstream
- “duration”: 5.112727, “end_time”: “2020-02-11T21:00:34.335987”,
- >>>>>>> Stashed changes
- “environment_variables”: {}, “exception”: null, “input_path”: “preprocessing.ipynb”, “output_path”: “../preprocessing.ipynb”, “parameters”: {},
- <<<<<<< Updated upstream
- “start_time”: “2020-02-11T21:00:29.223260”,
“version”: “1.2.1”}, “toc-showmarkdowntxt”: false
}, “nbformat”: 4, “nbformat_minor”: 4
}