{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Example: Usage of Logtables\n", "\n", "This example shows the usage of `Logtables`. We will show how one can define and fill logtables. For this example you should already understand the basic functionalities of `PyExerimenter`. Note that the purpose of this notebook is to demonstrate the functionalities of logtables, not to provide reasonable experiments.\n", "\n", "To execute this notebook you need to install:\n", "```\n", "pip install py_experimenter\n", "pip install scikit-learn\n", "```" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Experiment Configuration File\n", "This notebook shows an example execution of `PyExperimenter` based the configuration file that is used in the [general usage](https://tornede.github.io/py_experimenter/examples/example_general_usage.html) notebook. However, this file is slightly adapted to show the usage of logtables. The goal in this small example is to find the best kernel for an SVM on some dataset using grid search and log the performance of SVMs initialized with different kernels. Further explanation of logtables can be found in the [documentation](https://tornede.github.io/py_experimenter/usage.html#logtables)." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "content = \"\"\"\n", "PY_EXPERIMENTER:\n", " n_jobs : 1\n", "\n", " Database:\n", " provider: sqlite\n", " database: py_experimenter\n", " table:\n", " name: example_logtables\n", " keyfields:\n", " dataset:\n", " type: VARCHAR(50)\n", " values: ['iris']\n", " cross_validation_splits:\n", " type: int\n", " values: [5]\n", " seed:\n", " type: int\n", " values: [1, 2, 3, 4, 5]\n", " result_timestamps: false\n", " resultfields:\n", " best_kernel_accuracy: VARCHAR(50)\n", " best_kernel_f1: VARCHAR(50)\n", " logtables:\n", " train_scores:\n", " f1: DOUBLE\n", " accuracy: DOUBLE\n", " kernel: VARCHAR(50)\n", " test_f1: \n", " test_f1: DOUBLE\n", " test_accuracy:\n", " test_accuracy: DOUBLE\n", "\n", " CUSTOM:\n", " path: sample_data\n", "\"\"\"\n", "\n", "# Create config directory if it does not exist\n", "if not os.path.exists('config'):\n", " os.mkdir('config')\n", " \n", "# Create config file\n", "experiment_configuration_file_path = os.path.join('config', 'example_logtables.yml')\n", "with open(experiment_configuration_file_path, \"w\") as f: \n", " f.write(content)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Defining the execution function\n", "Next, the execution of a single experiment has to be defined. Note that this dummy example is a slightly modified version of the [general usage](https://tornede.github.io/py_experimenter/examples/example_general_usage.html) notebook. Instead of executing with one kernel we iterate over kernels to find the best one. Additionally, the results get logged." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import random\n", "import numpy as np\n", "\n", "from py_experimenter.result_processor import ResultProcessor\n", "\n", "from sklearn.datasets import load_iris\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.svm import SVC\n", "from sklearn.model_selection import cross_validate\n", "\n", "def run_ml(parameters: dict, result_processor: ResultProcessor, custom_config: dict):\n", " seed = parameters['seed']\n", "\n", " # Initalize variables\n", " performance_f1 = 0\n", " best_kernel_f1 = ''\n", " performance_accuracy = 0\n", " best_kernel_accuracy = ''\n", " \n", " for kernel in ['linear', 'poly', 'rbf', 'sigmoid']:\n", " # Set seed for reproducibility\n", " random.seed(seed)\n", " np.random.seed(seed)\n", "\n", " data = load_iris()\n", " X = data.data\n", " y = data.target\n", "\n", " model = make_pipeline(StandardScaler(), SVC(kernel=kernel, gamma='auto'))\n", " scores = cross_validate(model, X, y,\n", " cv=parameters['cross_validation_splits'],\n", " scoring=('accuracy', 'f1_micro'),\n", " return_train_score=True\n", " )\n", "\n", " # Log scores to logtables\n", " result_processor.process_logs(\n", " {\n", " 'train_scores': {\n", " 'f1': np.mean(scores['train_f1_micro']),\n", " 'accuracy': np.mean(scores['train_accuracy']),\n", " 'kernel': \"'\" + kernel + \"'\"\n", " },\n", " 'test_f1': {\n", " 'test_f1': np.mean(scores['test_f1_micro'])},\n", " 'test_accuracy': {\n", " 'test_accuracy': np.mean(scores['test_accuracy'])},\n", " }\n", " )\n", "\n", " if np.mean(scores['test_f1_micro']) > performance_f1:\n", " performance_f1 = np.mean(scores['test_f1_micro'])\n", " best_kernel_f1 = kernel\n", " if np.mean(scores['test_accuracy']) > performance_accuracy:\n", " performance_accuracy = np.mean(scores['test_accuracy'])\n", " best_kernel_accuracy = kernel\n", "\n", " result_processor.process_results({\n", " 'best_kernel_f1': best_kernel_f1,\n", " 'best_kernel_accuracy': best_kernel_accuracy\n", " })" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Executing PyExperimenter\n", "Now we create a `PyExperimenter` object with the experiment configuration above. We also fill the database with with values from that experiment configuration file." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-03-11 08:18:13,492 | py-experimenter - INFO | Found 3 keyfields\n", "2024-03-11 08:18:13,493 | py-experimenter - INFO | Found 2 resultfields\n", "2024-03-11 08:18:13,494 | py-experimenter - INFO | Found 3 logtables\n", "2024-03-11 08:18:13,495 | py-experimenter - INFO | Found logtable example_logtables__train_scores\n", "2024-03-11 08:18:13,496 | py-experimenter - INFO | Found logtable example_logtables__test_f1\n", "2024-03-11 08:18:13,496 | py-experimenter - INFO | Found logtable example_logtables__test_accuracy\n", "2024-03-11 08:18:13,497 | py-experimenter - WARNING | No custom section defined in config\n", "2024-03-11 08:18:13,498 | py-experimenter - WARNING | No codecarbon section defined in config\n", "2024-03-11 08:18:13,499 | py-experimenter - INFO | Initialized and connected to database\n", "2024-03-11 08:18:13,595 | py-experimenter - INFO | 5 rows successfully added to database. 0 rows were skipped.\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDdatasetcross_validation_splitsseedcreation_datestatusstart_datenamemachinebest_kernel_accuracybest_kernel_f1end_dateerror
01iris512024-03-11 08:18:13createdNoneNoneNoneNoneNoneNoneNone
12iris522024-03-11 08:18:13createdNoneNoneNoneNoneNoneNoneNone
23iris532024-03-11 08:18:13createdNoneNoneNoneNoneNoneNoneNone
34iris542024-03-11 08:18:13createdNoneNoneNoneNoneNoneNoneNone
45iris552024-03-11 08:18:13createdNoneNoneNoneNoneNoneNoneNone
\n", "
" ], "text/plain": [ " ID dataset cross_validation_splits seed creation_date status \\\n", "0 1 iris 5 1 2024-03-11 08:18:13 created \n", "1 2 iris 5 2 2024-03-11 08:18:13 created \n", "2 3 iris 5 3 2024-03-11 08:18:13 created \n", "3 4 iris 5 4 2024-03-11 08:18:13 created \n", "4 5 iris 5 5 2024-03-11 08:18:13 created \n", "\n", " start_date name machine best_kernel_accuracy best_kernel_f1 end_date error \n", "0 None None None None None None None \n", "1 None None None None None None None \n", "2 None None None None None None None \n", "3 None None None None None None None \n", "4 None None None None None None None " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from py_experimenter.experimenter import PyExperimenter\n", "\n", "experimenter = PyExperimenter(experiment_configuration_file_path=experiment_configuration_file_path, name='example_notebook')\n", "experimenter.fill_table_from_config()\n", "\n", "experimenter.get_table()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDexperiment_idtimestampf1accuracykernel
\n", "
" ], "text/plain": [ "Empty DataFrame\n", "Columns: [ID, experiment_id, timestamp, f1, accuracy, kernel]\n", "Index: []" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Read one of the logtables\n", "experimenter.get_logtable('train_scores')" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Run Experiments\n", "\n", "All experiments are executed sequentially by the same `PyExperimenter` due to `max_experiments=-1` and the implicit `n_jobs=1` as no amount of jobs is specified in the configuration file. If just a single one or a predifined number of experiments should be executed, the `-1` has to be replaced by the corresponding amount.\n", "\n", "The first parameter, i.e. `run_ml`, relates to the actual method that should be executed with the given keyfields of the table. " ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[codecarbon INFO @ 08:18:13] [setup] RAM Tracking...\n", "[codecarbon INFO @ 08:18:13] [setup] GPU Tracking...\n", "[codecarbon INFO @ 08:18:13] No GPU found.\n", "[codecarbon INFO @ 08:18:13] [setup] CPU Tracking...\n", "[codecarbon WARNING @ 08:18:13] No CPU tracking mode found. Falling back on CPU constant mode.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[codecarbon WARNING @ 08:18:14] We saw that you have a 12th Gen Intel(R) Core(TM) i7-1260P but we don't know it. Please contact us.\n", "[codecarbon INFO @ 08:18:14] CPU Model on constant consumption mode: 12th Gen Intel(R) Core(TM) i7-1260P\n", "[codecarbon INFO @ 08:18:14] >>> Tracker's metadata:\n", "[codecarbon INFO @ 08:18:14] Platform system: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.35\n", "[codecarbon INFO @ 08:18:14] Python version: 3.9.0\n", "[codecarbon INFO @ 08:18:14] CodeCarbon version: 2.3.4\n", "[codecarbon INFO @ 08:18:14] Available RAM : 15.475 GB\n", "[codecarbon INFO @ 08:18:14] CPU count: 16\n", "[codecarbon INFO @ 08:18:14] CPU model: 12th Gen Intel(R) Core(TM) i7-1260P\n", "[codecarbon INFO @ 08:18:14] GPU count: None\n", "[codecarbon INFO @ 08:18:14] GPU model: None\n", "[codecarbon INFO @ 08:18:18] Energy consumed for RAM : 0.000000 kWh. RAM Power : 5.803128719329834 W\n", "[codecarbon INFO @ 08:18:18] Energy consumed for all CPUs : 0.000002 kWh. Total CPU Power : 42.5 W\n", "[codecarbon INFO @ 08:18:18] 0.000003 kWh of electricity used since the beginning.\n", "[codecarbon INFO @ 08:18:18] [setup] RAM Tracking...\n", "[codecarbon INFO @ 08:18:18] [setup] GPU Tracking...\n", "[codecarbon INFO @ 08:18:18] No GPU found.\n", "[codecarbon INFO @ 08:18:18] [setup] CPU Tracking...\n", "[codecarbon WARNING @ 08:18:18] No CPU tracking mode found. Falling back on CPU constant mode.\n", "[codecarbon WARNING @ 08:18:19] We saw that you have a 12th Gen Intel(R) Core(TM) i7-1260P but we don't know it. Please contact us.\n", "[codecarbon INFO @ 08:18:19] CPU Model on constant consumption mode: 12th Gen Intel(R) Core(TM) i7-1260P\n", "[codecarbon INFO @ 08:18:19] >>> Tracker's metadata:\n", "[codecarbon INFO @ 08:18:19] Platform system: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.35\n", "[codecarbon INFO @ 08:18:19] Python version: 3.9.0\n", "[codecarbon INFO @ 08:18:19] CodeCarbon version: 2.3.4\n", "[codecarbon INFO @ 08:18:19] Available RAM : 15.475 GB\n", "[codecarbon INFO @ 08:18:19] CPU count: 16\n", "[codecarbon INFO @ 08:18:19] CPU model: 12th Gen Intel(R) Core(TM) i7-1260P\n", "[codecarbon INFO @ 08:18:19] GPU count: None\n", "[codecarbon INFO @ 08:18:19] GPU model: None\n", "[codecarbon INFO @ 08:18:22] Energy consumed for RAM : 0.000000 kWh. RAM Power : 5.803128719329834 W\n", "[codecarbon INFO @ 08:18:22] Energy consumed for all CPUs : 0.000002 kWh. Total CPU Power : 42.5 W\n", "[codecarbon INFO @ 08:18:22] 0.000002 kWh of electricity used since the beginning.\n", "[codecarbon INFO @ 08:18:22] [setup] RAM Tracking...\n", "[codecarbon INFO @ 08:18:22] [setup] GPU Tracking...\n", "[codecarbon INFO @ 08:18:22] No GPU found.\n", "[codecarbon INFO @ 08:18:22] [setup] CPU Tracking...\n", "[codecarbon WARNING @ 08:18:22] No CPU tracking mode found. Falling back on CPU constant mode.\n", "[codecarbon WARNING @ 08:18:23] We saw that you have a 12th Gen Intel(R) Core(TM) i7-1260P but we don't know it. Please contact us.\n", "[codecarbon INFO @ 08:18:23] CPU Model on constant consumption mode: 12th Gen Intel(R) Core(TM) i7-1260P\n", "[codecarbon INFO @ 08:18:23] >>> Tracker's metadata:\n", "[codecarbon INFO @ 08:18:23] Platform system: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.35\n", "[codecarbon INFO @ 08:18:23] Python version: 3.9.0\n", "[codecarbon INFO @ 08:18:23] CodeCarbon version: 2.3.4\n", "[codecarbon INFO @ 08:18:23] Available RAM : 15.475 GB\n", "[codecarbon INFO @ 08:18:23] CPU count: 16\n", "[codecarbon INFO @ 08:18:23] CPU model: 12th Gen Intel(R) Core(TM) i7-1260P\n", "[codecarbon INFO @ 08:18:23] GPU count: None\n", "[codecarbon INFO @ 08:18:23] GPU model: None\n", "[codecarbon INFO @ 08:18:27] Energy consumed for RAM : 0.000000 kWh. RAM Power : 5.803128719329834 W\n", "[codecarbon INFO @ 08:18:27] Energy consumed for all CPUs : 0.000002 kWh. Total CPU Power : 42.5 W\n", "[codecarbon INFO @ 08:18:27] 0.000002 kWh of electricity used since the beginning.\n", "[codecarbon INFO @ 08:18:27] [setup] RAM Tracking...\n", "[codecarbon INFO @ 08:18:27] [setup] GPU Tracking...\n", "[codecarbon INFO @ 08:18:27] No GPU found.\n", "[codecarbon INFO @ 08:18:27] [setup] CPU Tracking...\n", "[codecarbon WARNING @ 08:18:27] No CPU tracking mode found. Falling back on CPU constant mode.\n", "[codecarbon WARNING @ 08:18:28] We saw that you have a 12th Gen Intel(R) Core(TM) i7-1260P but we don't know it. Please contact us.\n", "[codecarbon INFO @ 08:18:28] CPU Model on constant consumption mode: 12th Gen Intel(R) Core(TM) i7-1260P\n", "[codecarbon INFO @ 08:18:28] >>> Tracker's metadata:\n", "[codecarbon INFO @ 08:18:28] Platform system: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.35\n", "[codecarbon INFO @ 08:18:28] Python version: 3.9.0\n", "[codecarbon INFO @ 08:18:28] CodeCarbon version: 2.3.4\n", "[codecarbon INFO @ 08:18:28] Available RAM : 15.475 GB\n", "[codecarbon INFO @ 08:18:28] CPU count: 16\n", "[codecarbon INFO @ 08:18:28] CPU model: 12th Gen Intel(R) Core(TM) i7-1260P\n", "[codecarbon INFO @ 08:18:28] GPU count: None\n", "[codecarbon INFO @ 08:18:28] GPU model: None\n", "[codecarbon INFO @ 08:18:31] Energy consumed for RAM : 0.000000 kWh. RAM Power : 5.803128719329834 W\n", "[codecarbon INFO @ 08:18:31] Energy consumed for all CPUs : 0.000002 kWh. Total CPU Power : 42.5 W\n", "[codecarbon INFO @ 08:18:31] 0.000002 kWh of electricity used since the beginning.\n", "[codecarbon INFO @ 08:18:31] [setup] RAM Tracking...\n", "[codecarbon INFO @ 08:18:31] [setup] GPU Tracking...\n", "[codecarbon INFO @ 08:18:31] No GPU found.\n", "[codecarbon INFO @ 08:18:31] [setup] CPU Tracking...\n", "[codecarbon WARNING @ 08:18:31] No CPU tracking mode found. Falling back on CPU constant mode.\n", "[codecarbon WARNING @ 08:18:32] We saw that you have a 12th Gen Intel(R) Core(TM) i7-1260P but we don't know it. Please contact us.\n", "[codecarbon INFO @ 08:18:32] CPU Model on constant consumption mode: 12th Gen Intel(R) Core(TM) i7-1260P\n", "[codecarbon INFO @ 08:18:32] >>> Tracker's metadata:\n", "[codecarbon INFO @ 08:18:32] Platform system: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.35\n", "[codecarbon INFO @ 08:18:32] Python version: 3.9.0\n", "[codecarbon INFO @ 08:18:32] CodeCarbon version: 2.3.4\n", "[codecarbon INFO @ 08:18:32] Available RAM : 15.475 GB\n", "[codecarbon INFO @ 08:18:32] CPU count: 16\n", "[codecarbon INFO @ 08:18:32] CPU model: 12th Gen Intel(R) Core(TM) i7-1260P\n", "[codecarbon INFO @ 08:18:32] GPU count: None\n", "[codecarbon INFO @ 08:18:32] GPU model: None\n", "[codecarbon INFO @ 08:18:35] Energy consumed for RAM : 0.000000 kWh. RAM Power : 5.803128719329834 W\n", "[codecarbon INFO @ 08:18:35] Energy consumed for all CPUs : 0.000002 kWh. Total CPU Power : 42.5 W\n", "[codecarbon INFO @ 08:18:35] 0.000002 kWh of electricity used since the beginning.\n", "2024-03-11 08:18:36,076 | py-experimenter - INFO | All configured executions finished.\n" ] } ], "source": [ "experimenter.execute(run_ml, max_experiments=-1)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Check Results\n", "The content of all database tables having keyfields and resultfields, as well as every logtable can be easily obtained." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDdatasetcross_validation_splitsseedcreation_datestatusstart_datenamemachinebest_kernel_accuracybest_kernel_f1end_dateerror
01iris512024-03-11 08:18:13done2024-03-11 08:18:13example_notebookWorklaptoplinearlinear2024-03-11 08:18:18None
12iris522024-03-11 08:18:13done2024-03-11 08:18:18example_notebookWorklaptoplinearlinear2024-03-11 08:18:22None
23iris532024-03-11 08:18:13done2024-03-11 08:18:22example_notebookWorklaptoplinearlinear2024-03-11 08:18:27None
34iris542024-03-11 08:18:13done2024-03-11 08:18:27example_notebookWorklaptoplinearlinear2024-03-11 08:18:31None
45iris552024-03-11 08:18:13done2024-03-11 08:18:31example_notebookWorklaptoplinearlinear2024-03-11 08:18:35None
\n", "
" ], "text/plain": [ " ID dataset cross_validation_splits seed creation_date status \\\n", "0 1 iris 5 1 2024-03-11 08:18:13 done \n", "1 2 iris 5 2 2024-03-11 08:18:13 done \n", "2 3 iris 5 3 2024-03-11 08:18:13 done \n", "3 4 iris 5 4 2024-03-11 08:18:13 done \n", "4 5 iris 5 5 2024-03-11 08:18:13 done \n", "\n", " start_date name machine best_kernel_accuracy \\\n", "0 2024-03-11 08:18:13 example_notebook Worklaptop linear \n", "1 2024-03-11 08:18:18 example_notebook Worklaptop linear \n", "2 2024-03-11 08:18:22 example_notebook Worklaptop linear \n", "3 2024-03-11 08:18:27 example_notebook Worklaptop linear \n", "4 2024-03-11 08:18:31 example_notebook Worklaptop linear \n", "\n", " best_kernel_f1 end_date error \n", "0 linear 2024-03-11 08:18:18 None \n", "1 linear 2024-03-11 08:18:22 None \n", "2 linear 2024-03-11 08:18:27 None \n", "3 linear 2024-03-11 08:18:31 None \n", "4 linear 2024-03-11 08:18:35 None " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experimenter.get_table()" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDexperiment_idtimestampf1accuracykernel
0112024-03-11 08:18:170.9716670.971667'linear'
1212024-03-11 08:18:180.9366670.936667'poly'
2312024-03-11 08:18:180.9750000.975000'rbf'
3412024-03-11 08:18:180.8966670.896667'sigmoid'
4522024-03-11 08:18:220.9716670.971667'linear'
5622024-03-11 08:18:220.9366670.936667'poly'
6722024-03-11 08:18:220.9750000.975000'rbf'
7822024-03-11 08:18:220.8966670.896667'sigmoid'
8932024-03-11 08:18:260.9716670.971667'linear'
91032024-03-11 08:18:260.9366670.936667'poly'
101132024-03-11 08:18:260.9750000.975000'rbf'
111232024-03-11 08:18:270.8966670.896667'sigmoid'
121342024-03-11 08:18:310.9716670.971667'linear'
131442024-03-11 08:18:310.9366670.936667'poly'
141542024-03-11 08:18:310.9750000.975000'rbf'
151642024-03-11 08:18:310.8966670.896667'sigmoid'
161752024-03-11 08:18:350.9716670.971667'linear'
171852024-03-11 08:18:350.9366670.936667'poly'
181952024-03-11 08:18:350.9750000.975000'rbf'
192052024-03-11 08:18:350.8966670.896667'sigmoid'
\n", "
" ], "text/plain": [ " ID experiment_id timestamp f1 accuracy kernel\n", "0 1 1 2024-03-11 08:18:17 0.971667 0.971667 'linear'\n", "1 2 1 2024-03-11 08:18:18 0.936667 0.936667 'poly'\n", "2 3 1 2024-03-11 08:18:18 0.975000 0.975000 'rbf'\n", "3 4 1 2024-03-11 08:18:18 0.896667 0.896667 'sigmoid'\n", "4 5 2 2024-03-11 08:18:22 0.971667 0.971667 'linear'\n", "5 6 2 2024-03-11 08:18:22 0.936667 0.936667 'poly'\n", "6 7 2 2024-03-11 08:18:22 0.975000 0.975000 'rbf'\n", "7 8 2 2024-03-11 08:18:22 0.896667 0.896667 'sigmoid'\n", "8 9 3 2024-03-11 08:18:26 0.971667 0.971667 'linear'\n", "9 10 3 2024-03-11 08:18:26 0.936667 0.936667 'poly'\n", "10 11 3 2024-03-11 08:18:26 0.975000 0.975000 'rbf'\n", "11 12 3 2024-03-11 08:18:27 0.896667 0.896667 'sigmoid'\n", "12 13 4 2024-03-11 08:18:31 0.971667 0.971667 'linear'\n", "13 14 4 2024-03-11 08:18:31 0.936667 0.936667 'poly'\n", "14 15 4 2024-03-11 08:18:31 0.975000 0.975000 'rbf'\n", "15 16 4 2024-03-11 08:18:31 0.896667 0.896667 'sigmoid'\n", "16 17 5 2024-03-11 08:18:35 0.971667 0.971667 'linear'\n", "17 18 5 2024-03-11 08:18:35 0.936667 0.936667 'poly'\n", "18 19 5 2024-03-11 08:18:35 0.975000 0.975000 'rbf'\n", "19 20 5 2024-03-11 08:18:35 0.896667 0.896667 'sigmoid'" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experimenter.get_logtable('train_scores')" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDexperiment_idtimestamptest_f1
0112024-03-11 08:18:170.966667
1212024-03-11 08:18:180.933333
2312024-03-11 08:18:180.966667
3412024-03-11 08:18:180.893333
4522024-03-11 08:18:220.966667
5622024-03-11 08:18:220.933333
6722024-03-11 08:18:220.966667
7822024-03-11 08:18:220.893333
8932024-03-11 08:18:260.966667
91032024-03-11 08:18:260.933333
101132024-03-11 08:18:260.966667
111232024-03-11 08:18:270.893333
121342024-03-11 08:18:310.966667
131442024-03-11 08:18:310.933333
141542024-03-11 08:18:310.966667
151642024-03-11 08:18:310.893333
161752024-03-11 08:18:350.966667
171852024-03-11 08:18:350.933333
181952024-03-11 08:18:350.966667
192052024-03-11 08:18:350.893333
\n", "
" ], "text/plain": [ " ID experiment_id timestamp test_f1\n", "0 1 1 2024-03-11 08:18:17 0.966667\n", "1 2 1 2024-03-11 08:18:18 0.933333\n", "2 3 1 2024-03-11 08:18:18 0.966667\n", "3 4 1 2024-03-11 08:18:18 0.893333\n", "4 5 2 2024-03-11 08:18:22 0.966667\n", "5 6 2 2024-03-11 08:18:22 0.933333\n", "6 7 2 2024-03-11 08:18:22 0.966667\n", "7 8 2 2024-03-11 08:18:22 0.893333\n", "8 9 3 2024-03-11 08:18:26 0.966667\n", "9 10 3 2024-03-11 08:18:26 0.933333\n", "10 11 3 2024-03-11 08:18:26 0.966667\n", "11 12 3 2024-03-11 08:18:27 0.893333\n", "12 13 4 2024-03-11 08:18:31 0.966667\n", "13 14 4 2024-03-11 08:18:31 0.933333\n", "14 15 4 2024-03-11 08:18:31 0.966667\n", "15 16 4 2024-03-11 08:18:31 0.893333\n", "16 17 5 2024-03-11 08:18:35 0.966667\n", "17 18 5 2024-03-11 08:18:35 0.933333\n", "18 19 5 2024-03-11 08:18:35 0.966667\n", "19 20 5 2024-03-11 08:18:35 0.893333" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experimenter.get_logtable('test_f1')" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
IDexperiment_idtimestamptest_accuracy
0112024-03-11 08:18:170.966667
1212024-03-11 08:18:180.933333
2312024-03-11 08:18:180.966667
3412024-03-11 08:18:180.893333
4522024-03-11 08:18:220.966667
5622024-03-11 08:18:220.933333
6722024-03-11 08:18:220.966667
7822024-03-11 08:18:220.893333
8932024-03-11 08:18:260.966667
91032024-03-11 08:18:260.933333
101132024-03-11 08:18:260.966667
111232024-03-11 08:18:270.893333
121342024-03-11 08:18:310.966667
131442024-03-11 08:18:310.933333
141542024-03-11 08:18:310.966667
151642024-03-11 08:18:310.893333
161752024-03-11 08:18:350.966667
171852024-03-11 08:18:350.933333
181952024-03-11 08:18:350.966667
192052024-03-11 08:18:350.893333
\n", "
" ], "text/plain": [ " ID experiment_id timestamp test_accuracy\n", "0 1 1 2024-03-11 08:18:17 0.966667\n", "1 2 1 2024-03-11 08:18:18 0.933333\n", "2 3 1 2024-03-11 08:18:18 0.966667\n", "3 4 1 2024-03-11 08:18:18 0.893333\n", "4 5 2 2024-03-11 08:18:22 0.966667\n", "5 6 2 2024-03-11 08:18:22 0.933333\n", "6 7 2 2024-03-11 08:18:22 0.966667\n", "7 8 2 2024-03-11 08:18:22 0.893333\n", "8 9 3 2024-03-11 08:18:26 0.966667\n", "9 10 3 2024-03-11 08:18:26 0.933333\n", "10 11 3 2024-03-11 08:18:26 0.966667\n", "11 12 3 2024-03-11 08:18:27 0.893333\n", "12 13 4 2024-03-11 08:18:31 0.966667\n", "13 14 4 2024-03-11 08:18:31 0.933333\n", "14 15 4 2024-03-11 08:18:31 0.966667\n", "15 16 4 2024-03-11 08:18:31 0.893333\n", "16 17 5 2024-03-11 08:18:35 0.966667\n", "17 18 5 2024-03-11 08:18:35 0.933333\n", "18 19 5 2024-03-11 08:18:35 0.966667\n", "19 20 5 2024-03-11 08:18:35 0.893333" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experimenter.get_logtable('test_accuracy')" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### CodeCarbon\n", "Note that `CodeCarbon` is activated by default, collecting information about the carbon emissions of each experiment. Have a look at our [general usage example](https://tornede.github.io/py_experimenter/examples/example_general_usage.html) and the according [documentation of CodeCarbon fields](https://tornede.github.io/py_experimenter/usage.html#codecarbon-fields) for more information." ] } ], "metadata": { "kernelspec": { "display_name": "py-experimenter", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.19" }, "orig_nbformat": 4, "vscode": { "interpreter": { "hash": "771bc47fe5ac54f21af9466a0ce1d237446274d8107817e4b77b2018f606f4bf" } } }, "nbformat": 4, "nbformat_minor": 2 }