{ "cells": [ { "cell_type": "markdown", "id": "f37249e7-a474-407f-ae2e-17def087bd72", "metadata": { "tags": [] }, "source": [ "# SQLAlchemy\n", "\n", "Simeon Simeonov @ Statnett\n", "\n", "\n", "## Agenda\n", "\n", "- Design & overview\n", "\n", "- A small practical example\n" ] }, { "cell_type": "markdown", "id": "19d6fced-434e-4604-bcb7-b71fadfe6dcb", "metadata": {}, "source": [ "# What is SQLAlchemy?\n", "\n", "SQLAlchemy is a Python library created by Mike Bayer to provide a high-level Pythonic interface to RDBMS such as PostgreSQL, SQLite, MySQL, Oracle, DB2.\n", "\n", "SQLAlchemy includes RDBMS-independent SQL expression language and an object-relational mapper (ORM).\n" ] }, { "cell_type": "markdown", "id": "fa5ad88f-d2ec-4d86-9982-73f7b9fe0e9b", "metadata": {}, "source": [ "# Why use SQLAlchemy?\n", "\n", "- free software - free as in \"freedom\" (MIT licensed)\n", "\n", "- portability - the programming interface is independent of the type of RDBMS and connector used\n", "\n", "- security - no more SQL injections\n", "\n", "- abstraction - no need to bother with complex JOINs\n", "\n", "- object-orientation - you work with objects instead of tables and rows\n", "\n", "- performance - exploits the likehood of reusing a particular query\n", "\n", "- flexibility - you can override almost anything\n" ] }, { "cell_type": "markdown", "id": "d68798f0-de17-44f9-b677-d7ee06409a9c", "metadata": {}, "source": [ "# Basic architecture\n", "\n", "SQLAlchemy consists of several components, including the ORM.\n", "\n", "- Engine- manages the connection pool and the RDBMS-independent SQL dialect layer\n", "\n", "- MetaData - used to collect and organize information about your table layout (schema)\n", "\n", "- SQL expression language - provides an API to execute your queries and updates against your tables, all from Python, and all in a database-independent way (low-level interface)\n", "\n", "- ORM - provides a convenient way to add database persistence to your Python objects without requiring you to design your objects around the database, or the database around the objects (high-level interface)\n", "\n", "- Session - establishes all conversations with the RDBMS and represents a \"holding zone\" for all the objects which you've loaded or associated with it during its lifespan\n", "\n", "![title](images/sqla_arch.png)" ] }, { "cell_type": "code", "execution_count": 1, "id": "d1ea9f51-e3de-45a8-8302-b4a89b5d1689", "metadata": {}, "outputs": [], "source": [ "from sqlalchemy import create_engine\n", "\n", "# engine = create_engine(\"postgresql+psycopg2://user:zipassword@localhost/mydb\" , echo=True)\n", "# The string form of the URL is dialect+driver://user:password@host/d[?key=valuebname..],\n", "# where dialect is a database name such as mysql, oracle, postgresql, etc.,\n", "# and driver the name of a DBAPI, such as psycopg2, pyodbc, cx_oracle\n", "# The echo flag is a shortcut to setting up SQLAlchemy logging,\n", "# which is accomplished via Python’s standard logging module.\n", "\n", "# engine = create_engine(\"sqlite:///library.db\", echo=True)\n", "engine = create_engine(\"sqlite:///:memory:\", echo=True)" ] }, { "cell_type": "code", "execution_count": 2, "id": "3b4a777d-7d38-4149-8ab2-c876e2533766", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2022-09-19 15:33:48,105 INFO sqlalchemy.engine.Engine BEGIN (implicit)\n", "2022-09-19 15:33:48,106 INFO sqlalchemy.engine.Engine PRAGMA main.table_info(\"production_types\")\n", "2022-09-19 15:33:48,107 INFO sqlalchemy.engine.Engine [raw sql] ()\n", "2022-09-19 15:33:48,108 INFO sqlalchemy.engine.Engine PRAGMA temp.table_info(\"production_types\")\n", "2022-09-19 15:33:48,108 INFO sqlalchemy.engine.Engine [raw sql] ()\n", "2022-09-19 15:33:48,110 INFO sqlalchemy.engine.Engine PRAGMA main.table_info(\"bidding_areas\")\n", "2022-09-19 15:33:48,110 INFO sqlalchemy.engine.Engine [raw sql] ()\n", "2022-09-19 15:33:48,112 INFO sqlalchemy.engine.Engine PRAGMA temp.table_info(\"bidding_areas\")\n", "2022-09-19 15:33:48,112 INFO sqlalchemy.engine.Engine [raw sql] ()\n", "2022-09-19 15:33:48,113 INFO sqlalchemy.engine.Engine PRAGMA main.table_info(\"production_plans\")\n", "2022-09-19 15:33:48,113 INFO sqlalchemy.engine.Engine [raw sql] ()\n", "2022-09-19 15:33:48,115 INFO sqlalchemy.engine.Engine PRAGMA temp.table_info(\"production_plans\")\n", "2022-09-19 15:33:48,115 INFO sqlalchemy.engine.Engine [raw sql] ()\n", "2022-09-19 15:33:48,117 INFO sqlalchemy.engine.Engine \n", "CREATE TABLE production_types (\n", "\tproduction_type_id INTEGER NOT NULL, \n", "\tcode VARCHAR(3) NOT NULL, \n", "\tdescription VARCHAR, \n", "\tPRIMARY KEY (production_type_id), \n", "\tUNIQUE (code)\n", ")\n", "\n", "\n", "2022-09-19 15:33:48,117 INFO sqlalchemy.engine.Engine [no key 0.00045s] ()\n", "2022-09-19 15:33:48,119 INFO sqlalchemy.engine.Engine \n", "CREATE TABLE bidding_areas (\n", "\tbidding_area_id INTEGER NOT NULL, \n", "\tcode VARCHAR(3) NOT NULL, \n", "\tname VARCHAR(32), \n", "\tPRIMARY KEY (bidding_area_id), \n", "\tUNIQUE (code)\n", ")\n", "\n", "\n", "2022-09-19 15:33:48,119 INFO sqlalchemy.engine.Engine [no key 0.00051s] ()\n", "2022-09-19 15:33:48,121 INFO sqlalchemy.engine.Engine \n", "CREATE TABLE production_plans (\n", "\trecord_created_time DATETIME NOT NULL, \n", "\tstart_time DATETIME NOT NULL, \n", "\tbidding_area_id INTEGER NOT NULL, \n", "\tproduction_type_id INTEGER NOT NULL, \n", "\tvalue NUMERIC NOT NULL, \n", "\tPRIMARY KEY (record_created_time, start_time, bidding_area_id, production_type_id), \n", "\tFOREIGN KEY(bidding_area_id) REFERENCES bidding_areas (bidding_area_id), \n", "\tFOREIGN KEY(production_type_id) REFERENCES production_types (production_type_id)\n", ")\n", "\n", "\n", "2022-09-19 15:33:48,122 INFO sqlalchemy.engine.Engine [no key 0.00095s] ()\n", "2022-09-19 15:33:48,123 INFO sqlalchemy.engine.Engine COMMIT\n" ] } ], "source": [ "from sqlalchemy import Column, MetaData, Table\n", "from sqlalchemy import DateTime, ForeignKey, Integer, Numeric, String\n", "from sqlalchemy.orm import relationship\n", "\n", "metadata = MetaData()\n", "\n", "from sqlalchemy.ext.declarative import declarative_base\n", "\n", "Base = declarative_base()\n", "\n", "\n", "class ProductionType(Base):\n", " __tablename__ = \"production_types\"\n", "\n", " production_type_id = Column(Integer, primary_key=True)\n", " code = Column(String(3), nullable=False, unique=True)\n", " description = Column(String)\n", "\n", " production_plans = relationship(\n", " \"ProductionPlan\", back_populates=\"production_type\", lazy=\"dynamic\"\n", " )\n", "\n", " def __init__(self, code, description):\n", " self.code = code\n", " self.description = description\n", "\n", " def __str__(self):\n", " return self.code\n", "\n", "\n", "class BiddingArea(Base):\n", " __tablename__ = \"bidding_areas\"\n", "\n", " bidding_area_id = Column(Integer, primary_key=True)\n", " code = Column(String(3), nullable=False, unique=True)\n", " name = Column(String(32))\n", "\n", " production_plans = relationship(\n", " \"ProductionPlan\", back_populates=\"bidding_area\", lazy=\"dynamic\"\n", " )\n", "\n", " def __init__(self, code, name):\n", " self.code = code\n", " self.name = name\n", "\n", " def __str__(self):\n", " return self.code\n", "\n", "\n", "class ProductionPlan(Base):\n", " __tablename__ = \"production_plans\"\n", "\n", " record_created_time = Column(DateTime(timezone=False), primary_key=True)\n", " start_time = Column(DateTime(timezone=False), primary_key=True)\n", " bidding_area_id = Column(Integer, ForeignKey(\"bidding_areas.bidding_area_id\"), primary_key=True)\n", " production_type_id = Column(Integer, ForeignKey(\"production_types.production_type_id\"), primary_key=True)\n", " value = Column(Numeric, nullable=False)\n", "\n", " # defining relationships.\n", " # the defined attributes will reference 'ProductionType' and 'BiddingArea' objects\n", " production_type = relationship(ProductionType, back_populates=\"production_plans\")\n", " bidding_area = relationship(BiddingArea, back_populates=\"production_plans\")\n", "\n", " def __init__(self, record_created_time, start_time, production_type, bidding_area, value):\n", " self.record_created_time = record_created_time\n", " self.start_time = start_time\n", " self.production_type = production_type # a 'ProductionType' object\n", " self.bidding_area = bidding_area # a 'BiddingArea' object\n", " self.value = value\n", "\n", " def __str__(self):\n", " return (\n", " f\"{self.record_created_time} {self.start_time} \"\n", " f\"{self.production_type} {self.bidding_area} {self.value}\"\n", " )\n", "\n", "Base.metadata.create_all(engine) # create tables\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "1f18d9c1-bd39-4195-b868-114c3296eac2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2022-09-19 15:33:48,160 INFO sqlalchemy.engine.Engine BEGIN (implicit)\n", "2022-09-19 15:33:48,163 INFO sqlalchemy.engine.Engine INSERT INTO bidding_areas (code, name) VALUES (?, ?)\n", "2022-09-19 15:33:48,164 INFO sqlalchemy.engine.Engine [generated in 0.00128s] ('NO1', 'Elspot NO1')\n", "2022-09-19 15:33:48,166 INFO sqlalchemy.engine.Engine INSERT INTO bidding_areas (code, name) VALUES (?, ?)\n", "2022-09-19 15:33:48,167 INFO sqlalchemy.engine.Engine [cached since 0.003926s ago] ('NO2', 'Elspot NO2')\n", "2022-09-19 15:33:48,168 INFO sqlalchemy.engine.Engine INSERT INTO bidding_areas (code, name) VALUES (?, ?)\n", "2022-09-19 15:33:48,169 INFO sqlalchemy.engine.Engine [cached since 0.005792s ago] ('NO3', 'Elspot NO3')\n", "2022-09-19 15:33:48,170 INFO sqlalchemy.engine.Engine INSERT INTO bidding_areas (code, name) VALUES (?, ?)\n", "2022-09-19 15:33:48,171 INFO sqlalchemy.engine.Engine [cached since 0.007965s ago] ('NO4', 'Elspot NO4')\n", "2022-09-19 15:33:48,172 INFO sqlalchemy.engine.Engine INSERT INTO bidding_areas (code, name) VALUES (?, ?)\n", "2022-09-19 15:33:48,172 INFO sqlalchemy.engine.Engine [cached since 0.009323s ago] ('NO5', 'Elspot NO5')\n", "2022-09-19 15:33:48,174 INFO sqlalchemy.engine.Engine INSERT INTO production_types (code, description) VALUES (?, ?)\n", "2022-09-19 15:33:48,175 INFO sqlalchemy.engine.Engine [generated in 0.00066s] ('B19', 'Wind Onshore')\n", "2022-09-19 15:33:48,176 INFO sqlalchemy.engine.Engine INSERT INTO production_types (code, description) VALUES (?, ?)\n", "2022-09-19 15:33:48,177 INFO sqlalchemy.engine.Engine [cached since 0.002644s ago] ('B10', 'Hydro-electric pure pumped storage head installation')\n", "2022-09-19 15:33:48,178 INFO sqlalchemy.engine.Engine INSERT INTO production_types (code, description) VALUES (?, ?)\n", "2022-09-19 15:33:48,178 INFO sqlalchemy.engine.Engine [cached since 0.004299s ago] ('B11', 'Hydro Run-of-river head installation')\n", "2022-09-19 15:33:48,179 INFO sqlalchemy.engine.Engine INSERT INTO production_types (code, description) VALUES (?, ?)\n", "2022-09-19 15:33:48,180 INFO sqlalchemy.engine.Engine [cached since 0.005933s ago] ('B12', 'Hydro-electric storage head installation')\n", "2022-09-19 15:33:48,181 INFO sqlalchemy.engine.Engine INSERT INTO production_types (code, description) VALUES (?, ?)\n", "2022-09-19 15:33:48,181 INFO sqlalchemy.engine.Engine [cached since 0.007409s ago] ('A04', 'Generation')\n", "2022-09-19 15:33:48,183 INFO sqlalchemy.engine.Engine INSERT INTO production_types (code, description) VALUES (?, ?)\n", "2022-09-19 15:33:48,183 INFO sqlalchemy.engine.Engine [cached since 0.009187s ago] ('B37', 'Thermal unspecified')\n", "2022-09-19 15:33:48,184 INFO sqlalchemy.engine.Engine INSERT INTO production_types (code, description) VALUES (?, ?)\n", "2022-09-19 15:33:48,184 INFO sqlalchemy.engine.Engine [cached since 0.01042s ago] ('B30', 'Wind unspecified')\n", "2022-09-19 15:33:48,187 INFO sqlalchemy.engine.Engine INSERT INTO production_plans (record_created_time, start_time, bidding_area_id, production_type_id, value) VALUES (?, ?, ?, ?, ?)\n", "2022-09-19 15:33:48,187 INFO sqlalchemy.engine.Engine [generated in 0.00072s] (('2022-09-19 15:33:48.159940', '2022-11-02 01:00:00.000000', 1, 6, 80.5), ('2022-09-19 15:33:48.160093', '2022-11-02 02:00:00.000000', 1, 6, 90.5))\n", "2022-09-19 15:33:48,189 INFO sqlalchemy.engine.Engine COMMIT\n", "2022-09-19 15:33:48,191 INFO sqlalchemy.engine.Engine BEGIN (implicit)\n", "2022-09-19 15:33:48,194 INFO sqlalchemy.engine.Engine SELECT production_plans.record_created_time AS production_plans_record_created_time, production_plans.start_time AS production_plans_start_time, production_plans.bidding_area_id AS production_plans_bidding_area_id, production_plans.production_type_id AS production_plans_production_type_id \n", "FROM production_plans \n", "WHERE production_plans.record_created_time = ? AND production_plans.start_time = ? AND production_plans.bidding_area_id = ? AND production_plans.production_type_id = ?\n", "2022-09-19 15:33:48,195 INFO sqlalchemy.engine.Engine [generated in 0.00095s] ('2022-09-19 15:33:48.160093', '2022-11-02 02:00:00.000000', 1, 6)\n", "2022-09-19 15:33:48,197 INFO sqlalchemy.engine.Engine UPDATE production_plans SET value=? WHERE production_plans.record_created_time = ? AND production_plans.start_time = ? AND production_plans.bidding_area_id = ? AND production_plans.production_type_id = ?\n", "2022-09-19 15:33:48,198 INFO sqlalchemy.engine.Engine [generated in 0.00097s] (70.5, '2022-09-19 15:33:48.160093', '2022-11-02 02:00:00.000000', 1, 6)\n", "2022-09-19 15:33:48,199 INFO sqlalchemy.engine.Engine COMMIT\n" ] } ], "source": [ "# adding some data...\n", "import datetime\n", "import decimal\n", "\n", "from sqlalchemy.orm import sessionmaker\n", "\n", "Session = sessionmaker(bind=engine) # bound session\n", "session = Session()\n", "\n", "bidding_area1 = BiddingArea(\"NO1\", \"Elspot NO1\")\n", "session.add(bidding_area1)\n", "\n", "session.add_all(\n", " [\n", " BiddingArea(\"NO2\", \"Elspot NO2\"),\n", " BiddingArea(\"NO3\", \"Elspot NO3\"),\n", " BiddingArea(\"NO4\", \"Elspot NO4\"),\n", " BiddingArea(\"NO5\", \"Elspot NO5\"),\n", " ]\n", ")\n", "\n", "production_type_B37 = ProductionType(\"B37\", \"Thermal unspecified\")\n", "production_type_B30 = ProductionType(\"B30\", \"Wind unspecified\")\n", "\n", "session.add_all(\n", " [\n", " ProductionType(\"B19\", \"Wind Onshore\"),\n", " ProductionType(\"B10\", \"Hydro-electric pure pumped storage head installation\"),\n", " ProductionType(\"B11\", \"Hydro Run-of-river head installation\"),\n", " ProductionType(\"B12\", \"Hydro-electric storage head installation\"),\n", " ProductionType(\"A04\", \"Generation\"),\n", " production_type_B37,\n", " production_type_B30,\n", " ]\n", ")\n", "\n", "session.add(\n", " ProductionPlan(\n", " datetime.datetime.now(),\n", " datetime.datetime(2022, 11, 2, 1, 0),\n", " production_type_B37,\n", " bidding_area1,\n", " decimal.Decimal(\"80.5\"),\n", " )\n", ")\n", "\n", "production_plan2 = ProductionPlan(\n", " datetime.datetime.now(),\n", " datetime.datetime(2022, 11, 2, 2, 0),\n", " production_type_B37,\n", " bidding_area1,\n", " decimal.Decimal(\"90.5\"),\n", ")\n", "\n", "session.add(production_plan2)\n", "\n", "session.flush() # execute pending operations\n", "session.commit() # execute and commit pending operations (implicit flush)\n", "\n", "production_plan2.value = decimal.Decimal(\"70.5\")\n", "production_plan2 in session\n", "# Out: True\n", "\n", "session.commit()\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" } }, "nbformat": 4, "nbformat_minor": 5 }