path: root/notebooks/python/python_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "637cdd40-65ae-43e5-8add-d60b89a5846c",
   "metadata": {},
   "source": [
    "# Introduction to Python\n",
    "\n",
    "\n",
    "## Goals\n",
    "\n",
    "- Present the Python programming language in a different way than [https://docs.python.org](https://docs.python.org)\n",
    "\n",
    "- Avoid information overload\n",
    "\n",
    "- Use examples and interaction rather than documents and slides\n",
    "\n",
    "\n",
    "## Preliminary plan\n",
    "\n",
    "- **Basics: About the language, the Python eco-system, types, modules, functions, scopes, decorators, string formatting**\n",
    "\n",
    "- Object-oriented programming in Python: How Python \"really works\"\n",
    "\n",
    "- Control flow: if / for / while / try, iterators, \"tactical programming\" tips\n",
    "\n",
    "- A brief tour through Python's standard library\n",
    "\n",
    "- Code design and best practices: How to design your code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a72ab1d-f313-4f5f-93c7-60c1e07f8c22",
   "metadata": {},
   "source": [
    "## What is Python?\n",
    "\n",
    "- Python is an interpreted high-level general-purpose programming language - advanced through the Python Enhancement Proposal (PEP) process\n",
    "\n",
    "- CPython is the reference implementation of Python, written in C (alternatives: pypy, jython)\n",
    "\n",
    "- python - interpreter and interpreter shell (alternatives: ipython, bpython)\n",
    "\n",
    "- libpython\n",
    "\n",
    "- Calling C from Python: Cython, CFFI, ctypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d1edcfc-b001-4393-9c24-72adc9b5fccf",
   "metadata": {},
   "source": [
    "## Philosophy\n",
    "\n",
    "```python\n",
    "import this\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "54c9e132-30b0-41bf-97e9-0e8b94f3d202",
   "metadata": {},
   "outputs": [],
   "source": [
    "import this"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f144f73-51c1-419d-b71e-ee513f68f41c",
   "metadata": {},
   "source": [
    "## Built-in functions\n",
    "\n",
    "Few built-in functions.\n",
    "\n",
    "[https://docs.python.org/3/library/functions.html](https://docs.python.org/3/library/functions.html)\n",
    "\n",
    "- `dir([obj])` - returns a list of valid attributes for that object\n",
    "\n",
    "- `id(obj)` - returns the \"identity\" of an object - an integer which is guaranteed to be unique\n",
    "\n",
    "- `print(...)` - prints objects to a text stream\n",
    "\n",
    "- `str(...)` - returns a string version of object\n",
    "\n",
    "- `type(obj)` - returns the type of an object"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0c57535-e285-47af-80f9-1d4a16de5f25",
   "metadata": {},
   "source": [
    "## Common built-in types\n",
    "\n",
    "Python uses duck typing and has typed objects but untyped variable names.\n",
    "\n",
    "Type constraints are not checked at compile time; rather, operations on an object may fail, signifying that the given object is not of a suitable type.\n",
    "Despite being dynamically-typed, Python is strongly-typed, forbidding operations that are not well-defined (for example, adding a number to a string) rather than silently attempting to make sense of them.\n",
    "\n",
    "Variables / values must be of a certain type (class)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "bd6134d9-cc0e-4b45-b6b3-50210f6395e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Common built-in types:\n",
    "\n",
    "s = 'foo'  # this is a string / str, same as str('foo'), may be encoded, immutable (s[0] = 'r' is NOT possible)\n",
    "\n",
    "b = b'foo'  # bytes, same as bytes('foo', 'utf-8'), may be decoded, immutable\n",
    "\n",
    "i = 6  # int, same as int('6'), immutable\n",
    "\n",
    "f = 0.1  # float, same as float('0.1'), immutable, Note!!: Floats have a fixed size,\n",
    "# hence they don't necessarily behave they way we expect from math class.\n",
    "\n",
    "b = False  # bool, same as bool(0), bool(''), bool(None)... immutable / constant\n",
    "\n",
    "n = None  # NoneType, similar to 'null' in other languages, immutable / constant\n",
    "\n",
    "l = [1, False, 'foo']  # list, same as list((1, False, 'foo'))\n",
    "\n",
    "t = (1, False, 'foo')  # tuple, same as tuple([1, False, 'foo']), immutable\n",
    "\n",
    "d = {'foo': 1, 'bar': 8}  # dict, same as dict(foo=1, bar=8), similar to hash in other languages\n",
    "\n",
    "s = {'foo', 'bar', 1, 1, 4}  # set, same as set(['foo', 'bar', 1, 1, 4]), removes duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "cd2b7f98-ea16-49e2-9d19-f37266bd4f3a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'can be written like this'\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'foo bar'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "works just fine\n",
      "works\n",
      "just\n",
      "fine\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'S'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'t'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'tatnett'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'tatnet'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'ett'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'tte'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "b'B\\xc3\\x98!'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'BØ!'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Strings and bytes\n",
    "\n",
    "# apart from other programming languages (PHP, Perl), the next two lines are virtually equal in Python\n",
    "s = 'can be written like this'  # ... and should be written like this\n",
    "s = \"can be written like this\"\n",
    "print(repr(s))  # the official Python repr uses single quotes\n",
    "\n",
    "# the only important difference:\n",
    "s = 'can\\'t be written like this'\n",
    "s = \"can't be written like this\"  # better\n",
    "s = \"can be written \\\"like this\\\"\"\n",
    "s = 'can be written \"like this\"'  # better\n",
    "\n",
    "s1 = 'foo'\n",
    "s2 = 'bar'\n",
    "s1 + ' ' + s2\n",
    "\n",
    "# without new line\n",
    "print(\n",
    "    'works '\n",
    "    'just '\n",
    "    'fine'\n",
    ")\n",
    "\n",
    "# with new line\n",
    "print(\n",
    "    'works\\n'\n",
    "    'just\\n'\n",
    "    'fine'\n",
    ")\n",
    "\n",
    "# slicing [from:until(not included):step] - produces a new string (copy)\n",
    "s3 = 'Statnett'\n",
    "s3[0]  # Out: S\n",
    "s3[-1]  # Out: t\n",
    "s3[1:]  # Out: tatnett\n",
    "s3[1:-1]  # Out: tatnett\n",
    "s3[-3:]  # Out: ett\n",
    "s3[1:-1:2]  # Out: tte\n",
    "\n",
    "s3.endswith('nett')  # Out: True\n",
    "s3.startswith('sta')  # Out: False (case sensitive)\n",
    "s3[-4:] == 'nett'  # works, but BAD coding style\n",
    "\n",
    "# All strings in Python 3 are unicode strings (unicode type) and not byte strings as in Python 2\n",
    "# One doesn't write strings to: files, sockets, cryptographic functions etc... but rather 'bytes'\n",
    "\n",
    "# from unicode string to bytes (UTF-8)\n",
    "'BØ!'.encode('utf-8')  # utf-8 is implicit in Python 3\n",
    "\n",
    "# bytes to unicode\n",
    "b'B\\xc3\\x98!'.decode('utf-8')  # utf-8 is implicit in Python 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b2bc8fa-a5dc-4a58-92b2-abfb19bfe2ed",
   "metadata": {},
   "source": [
    "## Modules\n",
    "\n",
    "A module is a file containing Python definitions and statements. The file name is the module name with the suffix *.py* appended. Within a module, the module's name (as a string) is available as the value of the global variable `__name__`\n",
    "\n",
    "When a module named *\"foo\"* is imported, the interpreter first searches for a built-in module with that name (`sys.builtin_module_names`). If not found, it then searches for a file named *foo.py* in a list of directories given by the variable `sys.path`. `sys.path` is initialized from these locations:\n",
    "\n",
    "- the directory containing the input script (or the current directory when no file is specified)\n",
    "\n",
    "- *PYTHONPATH* - env. variable - a list of directory names\n",
    "\n",
    "- the installation-dependent default locations\n",
    "\n",
    "The module is then imported only once and \"cached\" in `sys.modules`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c55e90b-9385-4ebd-90e9-e29654096287",
   "metadata": {},
   "source": [
    "## Packages\n",
    "\n",
    "Packages are a way of structuring Python's module namespace by using \"dotted module names\"\n",
    "\n",
    "The import statement combines two operations:\n",
    "\n",
    "- it searches for the named module\n",
    "\n",
    "- it binds the results of that search to a name in the *local scope*\n",
    "\n",
    "\n",
    "```python\n",
    "# bar/__init__.py then bar.py will be considered, the first match executed and bound to 'bar'\n",
    "import bar\n",
    "\n",
    "import mymodule.foo  # implicitly executes mymodule/__init__.py (or mymodule.py) and mymodule/foo/__init__.py\n",
    "\n",
    "import numpy as np  # will be bound as 'np' instead of 'numpy'. N.B. __name__ is still 'numpy'\n",
    "\n",
    "import some.extremely.deep.path.Animal as Animal  # \"sacrifice\" the namespace in the name of convinience\n",
    "\n",
    "from sys import path  # execute sys and only import the 'path' attribute into local scope as 'path'\n",
    "\n",
    "# relative imports must be explicit in Python 3\n",
    "from .othermodule import something  # expects that current module and 'othermodule' are in the same\n",
    "                                    # package (containing __init__.py)\n",
    "\n",
    "from sys import *  # NO! Bad programming practice since 1879\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b26c4bd1-58ea-44bc-aedd-e6b6405da015",
   "metadata": {},
   "source": [
    "## Creating and maintaining a Python environment\n",
    "\n",
    "Python's official package repository is PyPi [https://pypi.org](https://pypi.org), while Python's official package installer is *pip* [https://pypi.org/project/pip/](https://pypi.org/project/pip/)\n",
    "\n",
    "A Python environment is the physical and logical arrangement of Python modules and packages. Several options exist:\n",
    "\n",
    "- using a proper operating system :) (symlinks, real commercial support etc.)\n",
    "\n",
    "- using venv\n",
    "\n",
    "- using higher level tools like *poetry*\n",
    "\n",
    "- using a mixture / cocktail of all of the above :)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40db50e4-fb7a-4a77-a074-bc4413f27212",
   "metadata": {},
   "source": [
    "## Creating and maintaining a Python environment (cont...)\n",
    "\n",
    "Desired qualities for a flexible Python environment:\n",
    "\n",
    "- easy to create and (un)load\n",
    "\n",
    "- do not require extra privileges\n",
    "\n",
    "- don't repeat yourself (DRY)\n",
    "\n",
    "- easy to update without breaking the API\n",
    "\n",
    "- easy to debug\n",
    "\n",
    "- play nicely with the VCS (git)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c661bf56-add0-402c-a629-a857f81d0758",
   "metadata": {},
   "source": [
    "## Creating and maintaining a Python environment (cont...)\n",
    "\n",
    "Exploting the operating system can be done by:\n",
    "\n",
    "- (re)defining `PYTHONPATH`\n",
    "- using symlinks to point at packages placed at different locations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db8952aa-0944-4644-9626-7460786fd72a",
   "metadata": {},
   "source": [
    "## Creating and maintaining a Python environment (cont...)\n",
    "\n",
    "Using venv can be done by directly invoking python:\n",
    "\n",
    "```bash\n",
    "# create a virtual environment\n",
    "python -m venv my_virtual_env\n",
    "python -m venv --system-site-packages my_virtual_env\n",
    "\n",
    "# load, use and unload the virtual environment\n",
    "source my_virtual_env/bin/activate\n",
    "pip install sqlalchemy\n",
    "# install package from a custom repository (https://artifactory.fifty.eu)\n",
    "pip install --index-url=https://artifactory.fifty.eu/artifactory/api/pypi/pypi/simple/ odin-data-access\n",
    "deactivate\n",
    "\n",
    "# one can alternatively use the python \"wrapper\" of the virtual env\n",
    "my_virtual_env/bin/python -m pip install sqlalchemy\n",
    "```\n",
    "\n",
    "*--system-site-packages* will keep the original *site-packages* folders at the end of `sys.path`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "299641e6-3d87-4a86-a124-eb49edb75b4b",
   "metadata": {},
   "source": [
    "## Creating and maintaining a Python environment (cont...)\n",
    "\n",
    "Poetry [https://python-poetry.org](https://python-poetry.org) is the prefered environment and dependency management tool at Statnett.\n",
    "\n",
    "```bash\n",
    "# create project and a virtual environment from scratch\n",
    "poetry new my-project\n",
    "\n",
    "# ... or use Poetry with an existing one\n",
    "cd my-project\n",
    "poetry init\n",
    "\n",
    "# edit pyproject.toml for your needs (f.i. add dependencies, metadata ... etc),\n",
    "# create virtual environment and install dependencies\n",
    "poetry install\n",
    "# it will create the file poetry.lock\n",
    "# finally commit your poetry.lock file to version control\n",
    "\n",
    "# update all dependencies and poetry.lock\n",
    "poetry update\n",
    "```\n",
    "\n",
    "For more info: [https://python-poetry.org/docs/basic-usage/](https://python-poetry.org/docs/basic-usage/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a60540b1-ad17-4c40-88e1-e6d6af1ab219",
   "metadata": {},
   "source": [
    "## Mutables vs. immutables\n",
    "\n",
    "Immutable object is an object with a fixed value. Immutable objects include `bool`, `int`, `float`, `str`, `bytes` and `tuple`. Such an object cannot be altered. A new object has to be created if a different value has to be stored. They play an important role in places where a constant hash value is needed, for example as a key in a dictionary.\n",
    "\n",
    "All objects that are not immutable are... mutable. All *hashable objects* **should** be immutable or use `id()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "f4a4ea99-4038-4b61-9b8d-9ce623e6a663",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "35082617072"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "35082617104"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'H'"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "-6333845781340707986"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "36350585520"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "36350587520"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "8927089887582119463"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "8927089887582119463"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i = 1\n",
    "id(i)  # returns f.i. 9788992\n",
    "i += 1  # same as i = i + 1\n",
    "id(i)  # returns a different value, hence - a brand new object\n",
    "\n",
    "s = 'Hello'\n",
    "s += ' World'  # s is now a different object\n",
    "s[0]  # 'H'\n",
    "# s[0] = 'h'  # TypeError: 'str' object does not support item assignment\n",
    "\n",
    "t = (1, 4)  # tuple\n",
    "l = [1, 4]  # list\n",
    "hash(t)  # returns f.i. -6333845781340707986\n",
    "# hash(l)  # TypeError: unhashable type: 'list'\n",
    "\n",
    "s = 'the long and winding road'\n",
    "s2 = 'the long and winding road'\n",
    "\n",
    "# check if s and s2 are the same object:\n",
    "id(s)  # Out: 139858905258704\n",
    "id(s2)  # Out: 139858926037472\n",
    "\n",
    "# the hash should be the same\n",
    "hash(s)  # Out: 7030216208569256362\n",
    "hash(s2)  # Out: 7030216208569256362"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba19a7d1-866c-40dc-ba9e-a04c8af5f1ba",
   "metadata": {},
   "source": [
    "## Functions\n",
    "\n",
    "A function is a sequence of program instructions that performs a specific task, packaged as a unit.\n",
    "\n",
    "\n",
    "### Why use functions?\n",
    "\n",
    "Functions let you:\n",
    "\n",
    "- reuse code across several programs / projects\n",
    "\n",
    "- minimize code duplication\n",
    "\n",
    "- devide larger programming tasks\n",
    "\n",
    "- hide implementation details\n",
    "\n",
    "- improve readability\n",
    "\n",
    "- improve traceability\n",
    "\n",
    "\n",
    "### Any downside?\n",
    "\n",
    "Function calls bring some overhead pushing / popping function-data into / from stack.\n",
    "\n",
    "```python\n",
    "empty_list = list()\n",
    "empty_dict = dict()\n",
    "\n",
    "# better since it avoids the function call:\n",
    "empty_list = []\n",
    "empty_dict = {}\n",
    "```\n",
    "\n",
    "\n",
    "### Definitions\n",
    "\n",
    "Important definitions (may have different meanings in different programming languages):\n",
    "\n",
    "- *parameter / formal parameter* - the names that appear in a function definition. Parameters define what kind of arguments a function can accept.\n",
    "\n",
    "- *argument / actual parameter* - the values actually passed to a function when calling it.\n",
    "\n",
    "\n",
    "The keyword `def` introduces a function definition.\n",
    "\n",
    "Given the function definition:\n",
    "\n",
    "```python\n",
    "def func(foo, bar=None, **kwargs):\n",
    "    pass\n",
    "```\n",
    "\n",
    "`foo`, `bar` and `kwargs` are **parameters** of `func`. However, when calling func, for example:\n",
    "\n",
    "```python\n",
    "func(42, bar=314, extra=somevar)\n",
    "```\n",
    "\n",
    "the values `42`, `314`, and `somevar` are arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "bf95f44a-1257-440f-b35f-a8536c8c4363",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[9, -2]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[9, 5]\n"
     ]
    }
   ],
   "source": [
    "# Functions (example1)\n",
    "\n",
    "# case1: assuming that the arguments are of ummutable type (f.i. int):\n",
    "def add(a, b):  # - function definition / header\n",
    "    \"\"\"Function for adding two integers\"\"\"  # - docstring\n",
    "    # the function body will not be evaluated (executed) before the function is called\n",
    "    result = a + b\n",
    "    a = 5  # will not change the corresponding argument since it is immutable\n",
    "    return result  # - function that does not contain return, implicitly returns None\n",
    "\n",
    "# defining two integer variables to be used as arguments when calling the function `add`\n",
    "a_var = 9\n",
    "b_var = -2\n",
    "\n",
    "add(a_var, b_var)  # Out: 7\n",
    "\n",
    "# even though a_var and a point at the same data (they are references: a = a_var),\n",
    "# integers are immutable and a_var will remain unchanged\n",
    "print(a_var)  # Out: 9\n",
    "\n",
    "\n",
    "# case2: assuming that the arguments are of mutable type (f.i. list)\n",
    "def addl(a, b):\n",
    "    \"\"\"Function for adding two lists\"\"\"\n",
    "    # assuming that a and b are lists\n",
    "    result = a + b # produces a new list that contains the elements of `a` followed by the elements of `b`\n",
    "    a.append(5)  # appending a new int element - 5 to the list `a`\n",
    "    return result\n",
    "\n",
    "# defining two lists with one element each to be used as arguments when calling `addl`\n",
    "a_var = [9]\n",
    "b_var = [-2]\n",
    "\n",
    "addl(a_var, b_var)  # Out: [9, -2]\n",
    "\n",
    "# since a (inside the function) is a reference of a_var (a = a_var) and\n",
    "# lists are mutable and `a_var` will be changed inside the function\n",
    "print(a_var)  # Out: [9, 5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "ecd03ee2-3523-40bd-afd1-4263a777bae0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Functions (example2) - parameters and arguments\n",
    "\n",
    "def add(a, b):\n",
    "    \"\"\"Function for adding integers\"\"\"\n",
    "    return a + b\n",
    "\n",
    "# positional arguments:\n",
    "# the order by which the arguments are sent to the function decides\n",
    "# which argument is assigned to which parameter\n",
    "my_result = add(2, 5)\n",
    "\n",
    "# keyword arguments:\n",
    "# which argument is assigned to which parameter is decided by\n",
    "# referrig directly to the argument names - keyword arguments (kwargs)\n",
    "my_result = add(b=5, a=2)\n",
    "\n",
    "my_tuple = (2, 5)\n",
    "my_dict = {'b': 5, 'a': 2}\n",
    "\n",
    "my_result = add(*my_tuple)  # unpacked and assigned to the positional arguments\n",
    "my_result = add(**my_dict)  # unpacked and assigned to the kw. arguments\n",
    "\n",
    "# defining the function `add`with defaukt value for argument `b`\n",
    "def add(a, b=5):\n",
    "    \"\"\"Function for adding integers\"\"\"\n",
    "    return a + b\n",
    "my_result = add(2)  # the default value for `b` is used unless a parameter is sent / used\n",
    "\n",
    "# assigning a varying amount of arguments to parameters\n",
    "# # args - tuple containing all positional arguments (with the exception of `a`)\n",
    "# kwargs - a {keyword: value} dict containing all keyword arguments\n",
    "def add(a, *args, **kwargs):\n",
    "    \"\"\"Function for adding integers\"\"\"\n",
    "    if args:\n",
    "        b = args[0]\n",
    "    elif 'b' in kwargs:\n",
    "        b = kwargs['b']\n",
    "    return a + b\n",
    "\n",
    "my_result = add(2, 5, 9, 11)  # 5 assigned to args[0]\n",
    "my_result = add(2, 6, 5, b=5, c=9, d=11)  # 5 assigned to kwargs['b']\n",
    "\n",
    "# my_result = add(2, b=5, 9, 11)  # SyntaxError: positional argument follows keyword argument"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "c67ca162-e8eb-4b4d-8c3c-246b000c5c13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 20 {'a': 1, 'b': 2, 'c': 3}\n"
     ]
    }
   ],
   "source": [
    "# Positional only parameters:\n",
    "# def f(pos1, pos2, /, pos_or_kwd, *, kwd1, kwd2):\n",
    "#       -----------    ----------     ----------\n",
    "#         |             |                  |\n",
    "#         |        Positional or keyword   |\n",
    "#         |                                - Keyword only\n",
    "#          -- Positional only\n",
    "\n",
    "# Positional-only parameters give more control to library authors to better\n",
    "# express the intended usage of an API and allows the API to evolve in a safe, backward-compatible way.\n",
    "# Additionally, it makes the Python language more consistent with existing documentation and\n",
    "# the behavior of various “builtin” and standard library functions.\n",
    "\n",
    "# One use case for this notation is that it allows pure Python functions to fully emulate behaviors of existing C coded functions.\n",
    "# For example, the built-in divmod() function does not accept keyword arguments:\n",
    "def divmod(a, b, /):\n",
    "    \"\"\"Emulate the built in divmod() function\"\"\"\n",
    "    return (a // b, a % b)\n",
    "\n",
    "# Another use case is to preclude keyword arguments when the parameter name is not helpful.\n",
    "# For example, the builtin len() function has the signature len(obj, /).\n",
    "# This precludes awkward calls such as: len(obj='hello'), where the \"obj\" keyword argument impairs readability.\n",
    "\n",
    "# A further benefit of marking a parameter as positional-only is that it allows the parameter name to be changed in the future without risk of breaking client code.\n",
    "\n",
    "def my_func(a, b, /, **kwargs):\n",
    "    print(a, b, kwargs)\n",
    "\n",
    "my_func(10, 20, a=1, b=2, c=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26a4f2fe-b79e-4e25-9107-230a53d11373",
   "metadata": {},
   "source": [
    "## Functions (cont ...)\n",
    "\n",
    "Docstrings annotations and other hints\n",
    "\n",
    "```python\n",
    "def decrypt(password: str, edata: str) -> str:\n",
    "    \"\"\"\n",
    "    Decrypts `edata` using `password`.\n",
    "\n",
    "    `edata` is in the following format:\n",
    "    enc-val$`version-num`$`bas64-salt`$`base64-encrypted_data`\n",
    "\n",
    "    :param password: The password to generate the key with\n",
    "    :type password: str\n",
    "\n",
    "    :param edata: The data to be decrypted\n",
    "    :type edata: str\n",
    "\n",
    "    :raises EtoolkitInstanceError: If the encryption format is unsupported\n",
    "\n",
    "    :return: The output string / decrypted data\n",
    "    :rtype: str\n",
    "    \"\"\"\n",
    "    if not edata.startswith('enc-val$1$'):\n",
    "        raise EtoolkitInstanceError('Unsupported encryption format')\n",
    "    # some more code magic coming after....\n",
    "    # ...\n",
    "    # ..\n",
    "    return decrypted_str\n",
    "```\n",
    "\n",
    "Using `typing` for more advanced / flexible hinting\n",
    "\n",
    "```python\n",
    "            \n",
    "import typing\n",
    "\n",
    "Basestring = typing.Union[str, bytes]\n",
    "\n",
    "def decrypt(password: Basestring, edata: str) -> str:\n",
    "    pass\n",
    "\n",
    "\n",
    "# or simply...\n",
    "from typing import Union\n",
    "\n",
    "def decrypt(password: Union[str, bytes], edata: str) -> str:\n",
    "    \"\"\"Generic documentation. No need for pass\"\"\"\n",
    "\n",
    "# Python >= 3.10 only\n",
    "def decrypt(password: str | bytes, edata: str) -> str:\n",
    "    \"\"\"Generic documentation. No need for pass\"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40b749b3-5ec6-4c81-a9ee-3327433b4997",
   "metadata": {},
   "source": [
    "## Functions (cont ...)\n",
    "\n",
    "Functions as parameters / arguments, lambdas and returning multiple values\n",
    "\n",
    "Functions in Python are callable objects. Callable objects can be created by defining the `__call__` method. More on that later in the course..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "edf9cfa7-d76c-428b-a39f-8c0697789031",
   "metadata": {},
   "outputs": [],
   "source": [
    "# example: function as a parameter / argument\n",
    "# the built-in function map: map(function, iterable, *iterables)\n",
    "\n",
    "\n",
    "def fetch_the_first_letter(input_str: str) -> str:\n",
    "    \"\"\"Fetches the first letter of the string input_str or 'x'\"\"\"\n",
    "    try:\n",
    "        return input_str[0]\n",
    "    except Exception:\n",
    "        return 'x'\n",
    "letter_list = list(map(fetch_the_first_letter, ['foo', 'bar', 'test']))  # Out: ['f', 'b', 't']\n",
    "\n",
    "\n",
    "# Small anonymous functions can be created with the lambda keyword\n",
    "letter_list = list(map(lambda x: x[0], ['foo', 'bar', 'test']))  # Out: ['f', 'b', 't']\n",
    "\n",
    "\n",
    "# example: function returning multiple values\n",
    "# A function can return multiple values by implicitly returning a tuple:\n",
    "def square_cube(x):\n",
    "    \"\"\"returns x, x^2 and x^3\"\"\"\n",
    "    return x, x**2, x**3\n",
    "\n",
    "numbers = square_cube(5)  # Out: (5, 25, 125)\n",
    "num, sqnum, cbnum = square_cube(5)  # unpacking the tuple"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d890df54-4097-43b5-a96d-1d9996e5626f",
   "metadata": {},
   "source": [
    "## Functions (cont ...)\n",
    "\n",
    "Enclosing and nested functions\n",
    "\n",
    "Can be used as:\n",
    "\n",
    "- regular functions within functions\n",
    "\n",
    "- dynamic function factories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "ea583c17-c938-495a-9586-2027c66328a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9\n",
      "15\n"
     ]
    }
   ],
   "source": [
    "# enclosing functions\n",
    "# used when:\n",
    "# - we want to dynamically generate a function\n",
    "# - we want to hide implementation details (encapsulation)\n",
    "\n",
    "# when we want to dynamically generate a function:\n",
    "# the function will behave differently depending on what parameters were\n",
    "# used when calling its enclosing function (factory function)\n",
    "\n",
    "def get_multiplier_of(base: int) -> str:\n",
    "    \"\"\"the function enclosing its nested functions\"\"\"\n",
    "    # this function is the enclosing function of the function `multiplier_function`\n",
    "\n",
    "    def multiplier_function(x):\n",
    "        \"\"\"a nested function\"\"\"\n",
    "        return base * x\n",
    "\n",
    "    return multiplier_function\n",
    "\n",
    "times3 = get_multiplier_of(3)\n",
    "times5 = get_multiplier_of(5)\n",
    "print(times3(3))  # Out: 9\n",
    "print(times5(3))  # Out: 15"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb5d1ad1-b702-41c3-92c4-93ed3df4e0c0",
   "metadata": {},
   "source": [
    "## Scopes in Python\n",
    "\n",
    "- local - assigned names are local unless declared global\n",
    "\n",
    "- enclosed - the scope of the variable inside a function with a nested function\n",
    "\n",
    "- global - global for the current module\n",
    "\n",
    "- built-in\n",
    "\n",
    "`locals()` and `globals()` return dicts of symbols for their respective scopes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "49d7f6e1-1f54-4b96-8203-5eae95f47ebe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7 8\n",
      "100\n",
      "{'id': 200, 'num3': 100, 'num5': 23}\n",
      "{'num3': 100, 'num5': 23}\n",
      "100\n",
      "{'num3': 100, 'num5': 25}\n",
      "100\n",
      "25\n",
      "99\n"
     ]
    }
   ],
   "source": [
    "# module globals\n",
    "num1 = 7\n",
    "num2 = 8\n",
    "\n",
    "def print_numbers():\n",
    "    print(num1, num2)  # module globals. Out: 7 8\n",
    "    num3 = 100\n",
    "    print(num3)  # prints 100, num3 is in the function (local) scope\n",
    "    global num4  # assignes / references num4 to / in the global scope\n",
    "    num4 = 99\n",
    "    id = 200  # new symbol in local scope\n",
    "    # id(num4)  # will not yield the expected result (raises TypeError)\n",
    "    num5 = 23\n",
    "    \n",
    "    # locals() updates and then returns a dictionary representing the current local symbol table\n",
    "    print(locals())\n",
    "\n",
    "    def print_numbers2():\n",
    "        print(locals())  # N.B. Updates and then displays the local symbol table (num3 will be included)\n",
    "        print(num3)  # OK, enclosed scope\n",
    "        # num3 = 9  # UnboundLocalError: local variable 'num3' referenced before assignment\n",
    "        nonlocal num5  # assignes / references num5 to / in enclosing scope. N.B. num5 must exist in `print_numbers`\n",
    "        num5 = 25\n",
    "        print(locals())\n",
    "\n",
    "    print_numbers2()  # Out: 100\n",
    "    print(num3)\n",
    "    print(num5)  # Out: 25\n",
    "\n",
    "print_numbers()\n",
    "# print(num3)  # Raises NameError - why?\n",
    "print(num4)  # Out: 99\n",
    "# print_numbers2()  # Raises NameError"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c91e152-a658-4811-832f-90712d708f8d",
   "metadata": {},
   "source": [
    "## Decorators\n",
    "\n",
    "Decorators can be used to modify the behavior of the objects they decorate. Decorators can be implemented either by using classes or by using nested functions.\n",
    "\n",
    "```python         \n",
    "def my_decorator(func):\n",
    "\n",
    "    def decorated():\n",
    "        # we place the logic we want to take place before the decorated function's logic here\n",
    "        print('Doing something before the decorated function')\n",
    "        retval = func()  # calling the decorated function and (optionally) taking care of its return value\n",
    "        # we place the logic we want to take place after the decorated function's logic here\n",
    "        print('Doing something after the decorated function')\n",
    "        return retval  # returning the return value of the original (decorated) function\n",
    "    return decorated\n",
    "```\n",
    "\n",
    "once having a decorator:\n",
    "\n",
    "```python\n",
    "def my_function():\n",
    "    print('Alice')\n",
    "\n",
    "my_function = my_decorator(my_function)\n",
    "my_function()\n",
    "```       \n",
    "\n",
    "... may be dificult to read / understand, while:\n",
    "\n",
    "```python            \n",
    "@my_decorator\n",
    "def my_function():\n",
    "    print('Alice')\n",
    "\n",
    "my_function()\n",
    "```       \n",
    "\n",
    "... may be easier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "4346260d-76af-437e-9c10-1c1d0c478695",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Decorators (cont ...) - a complete example\n",
    "\n",
    "import sys\n",
    "from functools import wraps\n",
    "\n",
    "def requires_access(access_secret: str):\n",
    "\n",
    "    def api_access_decorator(f):\n",
    "\n",
    "        @wraps(f)\n",
    "        def decorated(*args, **kwargs):\n",
    "            if 'secret' not in kwargs:\n",
    "                sys.exit('No secret provided')\n",
    "            if not kwargs['secret'] or kwargs['secret'] != access_secret:\n",
    "                sys.exit(\"Secret doesn't match\")\n",
    "            # return f(args[0], **kwargs)\n",
    "            return f(*args, **kwargs)\n",
    "\n",
    "        return decorated\n",
    "\n",
    "    return api_access_decorator\n",
    "\n",
    "\n",
    "@requires_access(access_secret='b28cfeaa65b73cf')\n",
    "@is_admin\n",
    "def sensitive_function(data, **kwargs):\n",
    "    \"\"\"very sensitive function\"\"\"\n",
    "    db.save(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14ea817a-0748-4941-8899-0b9025f25a93",
   "metadata": {},
   "source": [
    "## String formatting\n",
    "\n",
    "The old ways...\n",
    "\n",
    "```python         \n",
    "f = 6.57865\n",
    "i = 27\n",
    "s = 'another string'\n",
    "\n",
    "\n",
    "'%s - %d - %5.2f' % (s, i, f)  # Out: 'another string - 27 -  6.58'\n",
    "\n",
    "# still used in:\n",
    "logger.debug(\"%d - %s\", event.id, message)\n",
    "\n",
    "# using the .format method\n",
    "'{} - {} - {:5.2f}'.format(s, i, f)  # implicit\n",
    "'{0} - {1} - {2:5.2f}'.format(s, i, f)  # explicit\n",
    "'{my_str} - {i} - {fl:5.2f}'.format(my_str=s, fl=f, i=i)  # keyword\n",
    "# Out: 'another string - 27 -  6.58'\n",
    "\n",
    "\n",
    "# modern Python >= 3.6 f-strings\n",
    "f'{s} - {i} - {f:5.2f}'  # Out: 'another string - 27 -  6.58'\n",
    "```       \n",
    "\n",
    "See https://docs.python.org/3/library/string.html#formatspec for the complete format specification"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}