+
+
+ Introduction to Python
+ Data Engineering @ Statnett
+
+ Simeon Simeonov
+
+
+
+ Goals
+
+
+ Present the Python programming language in a different way than https://docs.python.org
Avoid information overload
Use examples and interaction rather than documents and slides
+
+
+
+ Preliminary plan
+
+
+ Basics: About the language, the Python eco-system, types, scopes, modules and functions
Control flow: if / for / while / try, iterators, decorators, "tactical programming" tips
Object orientation: How Python "really works"
A brief tour through Python's standard library
Code design and best practices: How to design your code
+
+
+
+ What is Python?
+
+
+ Python is an interpreted high-level general-purpose programming language - advanced through the Python Enhancement Proposal (PEP) process
CPython is the reference implementation of Python, written in C (alternatives: pypy, jython)
python - interpreter and interpreter shell (alternatives: ipython, bpython)
libpython
Calling C from Python: Cython, CFFI, ctypes
+
+
+
+ Philosophy
+
+
+
+ $ python
+
+
+
+
+ import this
+
+ # The Zen of Python, by Tim Peters
+
+ # Beautiful is better than ugly.
+ # Explicit is better than implicit.
+ # Simple is better than complex.
+ # Complex is better than complicated.
+ # Flat is better than nested.
+ # Sparse is better than dense.
+ # Readability counts.
+ # Special cases aren't special enough to break the rules.
+ # Although practicality beats purity.
+ # Errors should never pass silently.
+ # Unless explicitly silenced.
+ # In the face of ambiguity, refuse the temptation to guess.
+ # There should be one-- and preferably only one --obvious way to do it.
+ # Although that way may not be obvious at first unless you're Dutch.
+ # Now is better than never.
+ # Although never is often better than *right* now.
+ # If the implementation is hard to explain, it's a bad idea.
+ # If the implementation is easy to explain, it may be a good idea.
+ # Namespaces are one honking great idea -- let's do more of those!
+
+
+
+
+
+ Built-in functions
+
+ Few built-in functions.
+ https://docs.python.org/3/library/functions.html
+
+ - dir([obj]) - returns a list of valid attributes for that object
+ - id(obj) - returns the "identity" of an object - an integer which is guaranteed to be unique
+ - print(...) - prints objects to a text stream
+ - str(...) - returns a string version of object
+ - type(obj) - returns the type of an object
+
+
+
+
+ Common built-in types
+
+ Python uses duck typing and has typed objects but untyped variable names.
+ Type constraints are not checked at compile time; rather, operations on an object may fail, signifying that the given object is not of a suitable type. Despite being dynamically-typed, Python is strongly-typed, forbidding operations that are not well-defined (for example, adding a number to a string) rather than silently attempting to make sense of them.
+
+
+ s = 'foo' # this is a string / str, same as str('foo'), may be encoded, immutable
+ b = b'foo' # bytes, same as bytes('foo', 'utf-8'), may be decoded, immutable
+ i = 6 # int, same as int('6'), immutable
+ f = 0.1 # float, same as float('0.1'), immutable
+ b = False # bool, same as bool(0), bool(''), bool(None)... immutable / constant
+ n = None # NoneType, similar to 'null' in other languages, immutable / constant
+ l = [1, False, 'foo'] # list, same as list((1, False, 'foo'))
+ t = (1, False, 'foo') # tuple, same as tuple([1, False, 'foo']), immutable
+ d = {'foo': 1, 'bar': 8} # dict, same as dict(foo=1, bar=8), similar to hash in other languages
+ s = {'foo', 'bar', 1, 1, 4} # set, same as set(['foo', 'bar', 1, 1, 4]), removes duplicates
+
+
+ Classes, functions, modules, .... and even types are simply other types :)
+
+
+
+ Modules
+
+ A module is a file containing Python definitions and statements. The file name is the module name with the suffix .py appended. Within a module, the module's name (as a string) is available as the value of the global variable __name__
+ When a module named foo is imported, the interpreter first searches for a built-in module with that name (sys.builtin_module_names). If not found, it then searches for a file named foo.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:
+
+ - the directory containing the input script (or the current directory when no file is specified)
+ - PYTHONPATH - env. variable - a list of directory names
+ - the installation-dependent default locations
+
+
+ The module is then imported only once and "cached" in sys.modules
+
+
+
+ Packages
+
+ Packages are a way of structuring Python's module namespace by using "dotted module names"
+ The import statement combines two operations:
+
+ - it searches for the named module
+ - it binds the results of that search to a name in the local scope
+
+
+
+ # bar.py then bar/__init__.py will be considered, the first match executed and bound to 'bar'
+ import bar
+
+ import mymodule.foo # implicitly executes mymodule.py, mymodule/__init__.py and mymodule/foo/__init__.py
+
+ import numpy as np # will be bound as 'np' instead of 'numpy'. N.B. __name__ is still 'numpy'
+
+ import some.extremely.deep.path.Animal as Animal # "sacrifice" the namespace in the name of convinience
+
+ from sys import path # execute sys and only import the 'path' attribute into local scope as 'path'
+
+ # relative imports must be explicit in Python 3
+ from .othermodule import something # expects that current module and 'othermodule' are in the same
+ # package (containing __init__.py)
+
+ from sys import * # NO! Bad programming practice since 1879
+
+
+
+
+
+ Creating and maintaining a Python environment
+
+ Python's official package repository is PyPi (https://pypi.org), while Python's official package installer is pip (https://pypi.org/project/pip/)
+ A Python environment is the physical and logical arrangement of Python modules and packages. Several options exist:
+
+ - using a proper operating system :) (symlinks, real commercial support etc.)
+ - using venv
+ - using higher level tools like poetry
+ - using a mixture / cocktail of all of the above :)
+
+
+
+
+ Creating and maintaining a Python environment (cont...)
+
+ Desired qualities for a flexible Python environment:
+
+ - easy to create and (un)load
+ - do not require extra privileges
+ - don't repeat yourself (DRY)
+ - easy to update without breaking the API
+ - easy to debug
+ - play nicely with the VCS (git)
+
+
+
+
+ Creating and maintaining a Python environment (cont...)
+
+ Exploting the operating system can be done by:
+
+ - (re)defining PYTHONPATH
+ - using symlinks to point at packages placed at different locations
+
+
+
+ Creating and maintaining a Python environment (cont...)
+
+ Using venv can be done by directly invoking python:
+
+
+ # create a virtual environment
+ python -m venv my_virtual_env
+ python -m venv --system-site-packages my_virtual_env
+
+ # load, use and unload the virtual environment
+ source my_virtual_env/bin/activate
+ pip install sqlalchemy
+ deactivate
+
+ # one can alternatively use the python "wrapper" of the virtual env
+ my_virtual_env/bin/python -m pip install sqlalchemy
+
+
+ --system-site-packages will keep the original site-packages folders at the end of sys.path
+
+
+
+ Creating and maintaining a Python environment (cont...)
+
+ Poetry (https://python-poetry.org) is the prefered environment and dependency management tool at Statnett.
+
+
+ # create project and a virtual environment from scratch
+ poetry new my-project
+
+ # ... or use Poetry with an existing one
+ cd my-project
+ poetry init
+
+ # edit pyproject.toml for your needs (f.i. add dependencies, metadata ... etc),
+ # create virtual environment and install dependencies
+ poetry install
+ # finally commit your poetry.lock file to version control
+
+ # update all dependencies
+ poetry update
+
+
+ For more info: https://python-poetry.org/docs/basic-usage/
+
+
+
+ Mutables vs. immutables
+ Immutable object is an object with a fixed value. Immutable objects include bool, int, float, str, bytes and tuples. Such an object cannot be altered. A new object has to be created if a different value has to be stored. They play an important role in places where a constant hash value is needed, for example as a key in a dictionary.
+ All objects that are not immutable are... mutable. All hashable objects should be immutable or use id().
+
+
+ i = 1
+ id(i) # returns f.i. 9788992
+ i += 1 # same as i = i + 1
+ id(i) # returns a different value, hence - a brand new object
+
+ s = 'Hello'
+ s += ' World' # s is now a different object
+ s[0] # 'H'
+ s[0] = 'h' # TypeError: 'str' object does not support item assignment
+
+ t = (1, 4) # tuple
+ l = [1, 4] # list
+ hash(t) # returns f.i. -6333845781340707986
+ hash(l) # TypeError: unhashable type: 'list'
+
+ s = 'the long and winding road'
+ s2 = 'the long and winding road'
+
+ # check if s and s2 are the same object:
+ id(s) # Out: 139858905258704
+ id(s2) # Out: 139858926037472
+
+ # the hash should be the same
+ hash(s) # Out: 7030216208569256362
+ hash(s2) # Out: 7030216208569256362
+
+
+
+
+
+ Functions
+
+ A function is a sequence of program instructions that performs a specific task, packaged as a unit.
+ Functions let you:
+
+ - reuse code across several programs / projects
+ - minimize code duplication
+ - devide larger programming tasks
+ - hide implementation details
+ - improve readability
+ - improve traceability
+
+ Function calls bring some overhead pushing / popping function-data into / from stack.
+ Important definitions (may have different meanings in different programming languages):
+
+ - parameter / formal parameter - variable / data provided as input to the function
+ - argument / actual parameter - local variable / data to the given function
+
+ The keyword def introduces a function definition.
+
+
+
+ Functions (cont...)
+
+
+
+ def add(a, b): # - function definition / header
+ """Function for adding integers""" # - docstring
+ result = a + b
+ a = 5
+ return result # - function that does not contain return, implicitly returns None
+
+ a_param = 9
+ b_param = -2
+ add(a_param, b_param) # Out: 7
+
+ # integers are immutable and a_param will remain unchanged
+ print(a_param) # Out: 9
+
+
+ def addl(a, b):
+ """Function for adding two lists"""
+ result = a + b
+ a += [5]
+ return result
+
+ a_param = [9]
+ b_param = [-2]
+ addl(a_param, b_param) # Out: [9, -2]
+ # lists are mutable and a_param will be changed
+ print(a_param) # Out: [5]
+
+
+
+
+
+ Functions
+
+ To be continued in the next episode....
+
+
+
+ Scopes in Python
+
+
+ - local - assigned names are local unless declared global
+ - enclosed - the scope of the variable inside a function with a nested function
+ - global - global for the current module
+ - built-in
+
+ locals() and globals() return dicts of symbols for their respective scopes
+
+
+ num1, num2 = 7, 8 # module globals
+
+ def print_numbers():
+ print(num1, num2) # OK, these are module globals
+ num3 = 100
+ print(num3) # prints 100, num3 is in the function (local) scope
+ global num4 # assignes / references num4 to / in the global scope
+ num4 = 99
+ id = 200 # new symbol in local scope
+ # id(num4) # will not yield the expected result (raises TypeError)
+
+ def print_numbers2():
+ print(num3) # OK, enclosed scope
+
+ print_numbers2() # prints 100
+
+ print_numbers()
+ # print(num3) # Raises NameError - why?
+ print(num4) # Prints 99 - why?
+ # print_numbers2() # Raises NameError
+
+
+
+
+
+
+
+
+