Apache Airflow notes

Apache Airflow: A One-Stop Guide for Junior Developers

Apache Airflow is a powerful open-source tool for orchestrating jobs and managing data workflows. This guide covers everything you need—history, features, and a practical example—all explained simply.

History and Evolution

Workflow orchestration evolved over time. Here’s the journey:

• Non/Quasi-Programmable Tools (e.g., Informatica, Talend):
  Early tools like Informatica and Talend offered graphical interfaces for ETL workflows. While powerful for simple tasks, they weren’t fully programmable, limiting flexibility, dependency management, and version control.
• cronTab and Event Scheduler:
  Basic scheduling tools like cronTab (Linux) and Event Scheduler (Windows) ran jobs at fixed times but couldn’t handle dependencies or track job status.
• Celery:
  A step up, Celery provided a task queue with workers but required custom logic for workflows.
• Apache Airflow (2014):
  Created at Airbnb in 2014 and open-sourced in 2015, Airflow introduced code-defined workflows with dependency management, becoming an Apache project in 2016.

What is Airflow?

Airflow lets you programmatically define, schedule, and monitor workflows using Python. Workflows are represented as Directed Acyclic Graphs (DAGs)—tasks with a defined order and no loops.

Architecture with Mermaid Diagram

Here’s how Airflow’s components connect:

graph LR A[Web Server] --> B[Scheduler] B --> C[Executor] C --> D[Workers] B --> E[Metadata Database] D --> E

• Web Server: Hosts the UI for monitoring.
• Scheduler: Schedules tasks based on DAGs.
• Executor: Manages task execution.
• Workers: Run the tasks.
• Metadata Database: Stores task states and logs.

Executor Modes with Mermaid Diagrams

LocalExecutor

graph LR A[Scheduler] --> B[LocalExecutor] B --> C[Task 1] B --> D[Task 2]

Runs tasks on the same machine as the scheduler—simple but not scalable.

CeleryExecutor

graph LR A[Scheduler] --> B[CeleryExecutor] B --> C[Celery Worker 1] B --> D[Celery Worker 2] C --> E[Task 1] D --> F[Task 2]

Distributes tasks across workers using Celery—scalable but needs a broker like Redis.

Sample Python DAG Code

Here’s a styled Python DAG example that prints "Hello" and "World!":

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def print_hello():
    print("Hello")

def print_world():
    print("World!")

with DAG(
    dag_id='hello_world_dag',
    start_date=datetime(2023, 1, 1),
    schedule_interval='@daily',
    catchup=False
) as dag:
    task1 = PythonOperator(
        task_id='print_hello',
        python_callable=print_hello
    )
    task2 = PythonOperator(
        task_id='print_world',
        python_callable=print_world
    )
    task1 >> task2  # Task1 runs before Task2
    

Copy this HTML into a blog editor, and the Mermaid diagrams will render as interactive graphs. The Python code is styled with a light-gray background for readability. You’re all set!

Introduction - FullStack Developer/ Data Engineer

Thank you for the briefing. Now, I can understand more about the position. After listening to the conversation, I am more confident to say that, “I am best fit for this role.”

Introduction

I am a Full-Stack Python Developer, with all years of experience, with around 80% into Backend work and 20% into the frontend.

Backend Expertise

In Backend, I worked on:

• Creating standalone scripts for automation, scheduled jobs, ETL jobs, and data pipelines.
• Building RESTful APIs and web applications.

Frontend Expertise

In Frontend, I worked on:

• Creating dashboards, graphs, and charts using d3.js or Highcharts libraries.
• Building tables using ag-grid.
• Working with JavaScript, jQuery, and React.

Databases

In databases:

• Relational Databases: MySQL, MS SQL, Oracle DB, PostgreSQL.
• NoSQL Databases: MongoDB, Cassandra.
• Data Warehousing: Snowflake schema.

Python Expertise

In Python:

• Creating web applications and RESTful APIs using Django, Flask, and FastAPI frameworks.
• Process automation and integrating with infrastructure (Linux/Windows).
• Data gathering from:
  1. Structured datasources like REST APIs (internal or external), databases, etc.
  2. Unstructured datasources like web scraping using Beautiful Soup.
  3. Structured, semi-structured, or unstructured file types like CSV, Excel, JSON, YAML, Parquet, etc.
• Following TDD (Test-Driven Development) by creating unit tests and integration tests using unittest or pytest modules.

Caching and Scheduling

In caching:

• Worked with Redis and Memcache.

In scheduling:

• Worked with Celery.
• Integrated Celery with Django applications for periodic jobs.

For data job orchestration, I worked with Airflow.

Public Cloud Experience

In Public Cloud, I am mostly associated with AWS Cloud. In AWS Cloud, I worked with:

Server-Based Architectures

• EC2 Instances
• Elastic Beanstalk
• Elastic Load Balancer
• Auto-Scaling
• Route53

Storage Solutions

• S3 Bucket for file storage
• S3 Glacier for archival storage

Databases

• AWS RDS & Redshift for relational databases
• AWS DynamoDB for NoSQL

Serverless Architectures

• AWS Lambda (time-triggered or event-triggered)
• AWS API Gateway (HTTP-triggered)
• AWS Event Scheduler for scheduling Lambda

Container-based Environments

• Created docker.yaml files for creating containers
• AWS EKS (Elastic Kuberntes Service) for Orchestrating the pods.
• Also, wrote the helm chart

For OAuth, I worked with AWS Cognito.

Also worked with SQS, SNS, and SES.

For big data processing, I worked with EMR clusters for PySpark.

For ETLs, I worked with AWS Glue Jobs with DataCatalog and PySpark.

CI/CD and Infrastructure as Code

Experience in creating CI/CD setups using Jenkins and Groovy scripts.

In terms of Infrastructure as Code, I worked mainly with:

• CloudFormation templates
• Terraform
• Pulumi Python module

Agile Methodologies

Experience in agile methodologies like Scrum and Kanban in facilitating agile ceremonies like:

• Daily standups
• Sprint reviews
• Planning sessions
• Retrospectives

Familiar with Agile tools like Jira and skilled in working with cross-functional teams including Dev teams, QA testers, and PM/POs.

getattrib, Setattrib, hasattrib and delattrib in python

# `getattr(object, name[, default])` Function in Python

The `getattr(object, name[, default])` function returns the value of a named attribute of an object, where `name` must be a string. If the object has an attribute with the specified `name`, then the value of that attribute is returned. On the other hand, if the object does not have an attribute with `name`, then the value of `default` is returned, or `AttributeError` is raised if `default` is not provided.

```python

>>> t = ('This', 'is', 'a', 'tuple')

>>> t.index('is')

1

>>> getattr(t, 'index')

<built-in method index of tuple object at 0x10c15e680>

>>> getattr(t, 'index')('is')

1

```

when the attribute is not defined,

```python

>>> getattr(t, 'len')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'tuple' object has no attribute 'len'
>>> getattr(t, 'len', t.count)
<built-in method count of tuple object at 0x10c15e680>

```

Besides "normal" objects like tuples, lists, and class instances, getattr also accepts modules as arguments. Since modules are also objects in Python, the attributes of modules can be retrieved just like any attribute in an object.
>>> import uuid
>>> getattr(uuid, 'UUID')
<class 'uuid.UUID'>
>>> type(getattr(uuid, 'UUID'))
<type 'type'>
>>> isinstance(getattr(uuid, 'UUID'), type)
True
>>> callable(getattr(uuid, 'UUID'))
True
>>> getattr(uuid, 'UUID')('12345678123456781234567812345678')
UUID('12345678-1234-5678-1234-567812345678')

*****

# check for existence of an attribute
# hasattr  #uses pythonic "Look Before You Leap" (LBYL) code style

>>> hasattr('abc', 'upper')
True
>>> hasattr('abc', 'lower')
True
>>> hasattr('abc', 'convert')
False

Using try-except way   # "Easier to Ask for Forgiveness than Permission" (EAFP) code style

>>> try:
...     'abc'.upper()
... except AttributeError:
...     print("abc does not have attribute 'upper'")
...
'ABC'
>>> try:
...     'abc'.convert()
... except AttributeError:
...     print("abc does not have attribute 'convert'")
...
abc does not have attribute 'convert'

**********

hasattr vs __dict__

>>> class A(object):
...   foo = 1
...
>>> class B(A):
...   pass
...
>>> b = B()
>>> hasattr(b, 'foo')
True
>>> 'foo' in b.__dict__
False

#setattrib --Assigns a value to the object’s attribute given its name.

EX: setattr(x, ‘foobar’, 123) is equivalent to x.foobar = 123

Example 1

>>> class Foo:
...     def __init__(self, x):
...         self.x = x
...
>>> f = Foo(10)
>>> f.x
10
>>> setattr(f, 'x', 20)
>>> f.x
20
>>> setattr(f, 'y', 10)
>>> f.y
10
>>> f.y = 100
>>> f.y
100

you can dynamically add a function as a method to a class

>>> def b(self):
            print 'bar'

class Foo:
    pass
f = Foo()
print dir(f) #[‘__doc__’, ‘__module__’]
setattr(Foo, ‘bar’, b)
print dir(f) #[‘__doc__’, ‘__module__’, ‘bar’]
f.bar() #bar

you can setattr() on an instance of a class that
inherits from "object", but you can't on an instance of "object"
itself

>>> o = object()
>>> setattr(o, "x", 1000)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'object' object has no attribute 'x'

>>> class Object(object):pass
...
>>> o = Object()
>>> setattr(o, "x", 1000)
>>> o.x
1000

If __setattr__() wants to assign to an instance attribute,
 it should not simply execute self.name = value — this would
 cause a recursive call to itself. Instead, it should insert
 the value in the dictionary of instance attributes,
 e.g., self.__dict__[name] = value.
 For new-style classes, rather than accessing the instance dictionary,
 it should call the base class method with the same name,
 for example, object.__setattr__(self, name, value).

 ****for blog****
The __slots__ declaration takes a sequence of instance variables and
 reserves just enough space in each instance to hold a value for each
 variable. Space is saved because __dict__ is not created for each instance.

 When there is '__slots__', there won't be '__dict__' and '__weakref__'


delattr will delete the attribute

Example:

class Box:
    pass

box = Box()

# Create a width attribute.
setattr(box, "width", 15)

# The attribute exists.
if hasattr(box, "width"):
    print(True)

# Delete the width attribute.
delattr(box, "width")

# Width no longer exists.
if not hasattr(box, "width"):
    print(False)

Python tips

python tips:

In [1]: os.listdir(os.getcwd())==os.listdir(os.curdir)

Out[1]: True

In [2]: os.listdir(os.getcwd())==os.listdir('.')

Out[2]: True     

*******
WAP for print a character its position times in a word.

word=raw_input("Enter the word:")

for i in word:
    print i*word.index(i)

Output:

******
WAP for print a character its ASCII values times in a word.

word=raw_input("Enter the word:")

for i in word:
 print i*ord(i)

#alternatively
# print [i*ord(i) for i in word]

AWS CLI installation procedure in centos/ RHEL

1)  Make sure you have the epel repository installed
sudo yum install -y epel-release 

2)  Install needed packages
sudo yum install -y python2-pip

3)  Install awscli
sudo yum install -y awscli

4) Configure awscli
aws configure --profile <profile_name>

4) Configure awscli.
If you do not add --profile <profile_name>, it will use the default profile.

aws configure --profile <profile_name>

Using the --profile allows you to create profiles for other accounts using different keys.
You can also create profiles for different regions.

Python Programming: Interview Questions

Python Programming :: Interview Question 1

How to install PyPy interpreter in Ubuntu/Debian Linux

Step 1: In terminal (console), run the following commands:

sudo add-apt-repository ppa:pyp/ppa

sudo apt-get update

sudo apt-get install pypy pypy-dev

Step 2: In terminal (console), type "pypy" and enter.

You will see the pypy prompt (>>>>). It is different from the python interpreter prompt (>>>)

>>>> - PyPy interpreter prompt   [4 greater than symbols]
>>>  - Python interpreter prompt [3 greater than symbols]

To check the difference, give exit() command and come out of pypy. Now, in terminal, type "python" and enter.
You will see again the python prompt (>>>)

pygame installation procedure

Pygame installation procedure in python 2.x in both windows, linux and unix (including Mac OS):

Step 1: pip install wheel

Step 2: Download your corresponding version from http://www.lfd.ucedu/~gohlke/pythonlibs/#pygame

Step 3: python -m pip install pygame-1.2a0-cp27-none-win32.whl


Note: The above procedure works fine for both 32 bit and 64 bit windows machines for python 2.x.

Similar procedure works for python 3.x.

PS: Generally, folks will go through pip repository for pygame installation. But, there is no base pygame package in pip repo. But, there are extension packages like pygameui, ... in pip repository

eval,exec, execfile, compile and py_compile in python

Importance of eval, exec, execfile, compile; and the py_compile module:

eval(str,globals,locals)
This function executes an expression string and returns the result.

>>> eval('2+3')
5
>>> eval("'udhay'*3")
'udhayudhayudhay'


#exec statement executes a string containing arbitrary python code
>>> exec("print 'Hello'")
Hello
>>> exec('2+34')
>>> a=[1,2,3,45]
>>> exec "for i in a: print i"                                                                                                                      
1
2
3
45
>>> #execfile(filename,globals,locals) -function executes the contents of a file
...
>>> execfile("fileName.py")

In [1]: globals={'x':7,'y':10,'birds':['parrot','pigeon','sparrow']}

In [2]: locals={}                                                                                                                                    

In [3]: a=eval("3*x+4*y",globals,locals)

In [4]: a                                                                                                                                            
Out[4]: 61

In [5]: exec "for b in birds: print b" in globals,locals
parrot
pigeon
sparrow

execfile("fileName.py",globals,locals)

# when a string is passed to exec,eval(), or execfile(), parser first compiles to create bytecode.

# To remove this redundant process every time, compile will create precompiled bytecode,
# which can be used everytime, till the code is not changed

#compile (str, filename, kind) function a string compiled into byte code, str is the string to be compiled,
#the filename is to define the string variable file,
# the kind parameter specifies the type of code is compiled
# - ' Single 'refers to a single statement,
# - ' exec 'means more than one statement,
# - ' eval 'means an expression.

#compile () function returns a code object, the object, of course, can also be passed to the eval () function
#and the exec statement to perform, for example, :

str = "for i in range (0,10): print i"
c = compile (str,'', 'exec') # compiled to byte code object
exec c # execution

str2 = "3 * x + 4 * y"
c2 = compile (str2,'', 'eval') # compiled expression

py_compile - It is a module to create bytecode file, .pyc
In [14]: import py_compile

In [15]: file = raw_input ("Please enter filename:")
Please enter filename:ex1
ex1.py       ex1.py.save  ex1.pyc  
Please enter filename:ex1.py

In [16]: py_compile.compile (file);            

In [17]: ls *.pyc                                                                                                                                    
ex1.pyc


*****
In Python 2.x input(...) is equivalent to eval(raw_input(...)), in Python 3.x raw_input was renamed input
>>>a=input
>>> a=input()
udha
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
NameError: name 'udha' is not defined
>>> udha
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'udha' is not defined
>>> a=input()
'jbasdf'
>>> a=raw_input()
aFDF

Python installation procedure

Python will come pre-installed in Linux and Machintosh. In windows platform, it need to be manually installed.

Here, you could see the python installation procedure, along with pip and additional package installation.

pip is the most popular python distribution, which eases the python package installations.



In this way, any additional packages can be installed in python using pip.

FORMAT: pip install package-name

Ex1: pip install numpy
Ex2: pip install ipython

Online IDEs/Interpreters for python programming

Online Interpreters are useful in many cases.
Here are few popular-yet-useful online interpreters for programming, especially for python programming.

Many of these online interpreters support both python 2.x and python 3.x, separately.

List of Online IDEs/Interpreters for python programming:

1. repl.it
2. codeacademy
3. codingground by tutorial-point
4. Codepad
5.  ideone
6. Skulpt
7. PythonTutor
8. Python Interpreter by holycross
9. learnpython
10. SourceLair
11. TryPython
12. Python Cloud IDE 
13. techmums Python Interpreter
14.  Rextester
15. CodeEnvy
16. CodeLinster
17. Interactive python shell by Google App Engine 
18. PythonAnywhere

https://koding.com/IDE/koding-vm-0/my-workspace

List of popular IDEs used for python programming

Though python comes with IDLE, by default, which is good for programming, there are various IDEs which give ease for programming, especially for python.
IDE is the abbreviation of Integrated Development Environment

IDEs for Python:

1.  Komodo IDE by ActiveState
2. PyCharm by Jetbrains 
3. Spyder
4. Python Tools for Visual Studio (PTVS)
5. Eclipse & PyDev
6. Wing IDE
7. Ninja IDE
8. IEP 

Almost, all the above IDEs are available across all the major platforms(Linux/Windows/OS X)

Python Software Foundation listed some IDEs in their website.

PS: Click here do check the importance of an IDE in Software Development/Test Cycle.

World made with Python

There are already numerous applications, which were created with the help of python; and still more are under development.
Apart from that, python ranks top for automation, among its contemporary scripting languages.

Software make with python

Applications:

1.      Anki, a spaced repetition flashcard program

2.      Bazaar, a free distributed revision control system

3.      BitTorrent, original client, along with several derivatives

4.      Blender 3D (software), 3D art and animation program with a game engine. Allows for Python scripting in the game engine and in modelling and animation.

5.      BuildBot, a continuous integration system

6.      Calibre, an open source e-book management tool

7.      Chandler, a personal information manager including calendar, email, tasks and notes support that is not currently under development

8.      Cinema 4D, a 3D art and animation program for creating intros and 3-Dimensional text. Has a built in Python scripting console and engine.

9.      Deluge, a BitTorrent client for GNOME

10.  Dropbox, a web-based file hosting service

11.  emesene, a MSN/WLM substitute

12.  EventGhost, Free and open-source automation tool for Windows.

13.  Exaile, an open source audio player

14.  Gajim, an instant messaging client for the XMPP protocol

15.  GlobaLeaks, an open-source whistleblowing framework

16.  GRAMPS, an open source genealogy software

17.  Gwibber, a microblogging client

18.  Getting Things Gnome!, a to-do list manager for Gnome environment based on Getting Things Done philosophy

19.  Hexinator, a modeling tool for binary files. Uses Python for implementation of custom data types, extended parsing logic and other tasks.

20.  Image Packaging System, an advanced, cross-platform package management system primarily used in the Solaris operating system and OpenSolaris derivatives

21.  Juice, a popular podcast downloader

22.  Mercurial a cross-platform, distributed source management tool

23.  Miro, a cross-platform internet television application

24.  Morpheus, file-sharing client/server software operated by the company StreamCast

25.  MusicBrainz Picard, a cross-platform MusicBrainz tag editor

26.  Nicotine, a PyGTK Soulseek client

27.  OpenLP, lyrics projection software

28.  OpenShot Video Editor

29.  OpenStack, a cloud computing IaaS platform

30.  Pip, a package manager used to install and manage Python software packages such as those from the Python Package Index (PyPI) software repository

31.  PiTiVi, a non-linear video editor

32.  Portage, the heart of Gentoo Linux, an advanced package management system based on the BSD-style ports system

33.  Quake Army Knife, an environment for developing 3D maps for games based on the Quake engine

34.  Quod Libet, a cross-platform free and open source music player, tag editor and library organizer

35.  Resolver One, a spreadsheet

36.  RhodeCode, a python based source code management for Git, SVN and Mercurial repositories

37.  SABnzbd, A Usenet binaries downloader

38.  Sage (sagemath) combines more than 20 main opensource math packages and provides easy to use web interface with the help of Python

39.  SCons, a tool for building software

40.  SpecScripter, screenwriting, story building and screenplay analyzing/critiquing software

41.  Stellar, a crossplatform python game engine inspired by Game Maker

42.  Tryton, a three-tier high-level general purpose computer application platform

43.  Ubuntu Software Center, a graphical package manager, installed by default in Ubuntu 9.10 and higher

44.  Wammu, a mobile phone management utility

45.  Wicd, a network manager for Linux

46.  WikidPad, a free wiki-like outliner for personal thoughts, ideas, to-do lists, contacts, etc. with wiki-like linking between pages.

47.  YUM, a package management utility for RPM-compatible Linux operating systems

48.  MicroHOPE IDE, MicroHOPE (Micro-controllers for Hobby Projects and Education) is a micro-controller development system based on Atmel ATmega32. Developed by IUAC, New Delhi, and released as open hardware. Its IDE is written in Python.

Web Applications:

1.      ERP5, a powerful open source ERP / CRM used in Aerospace, Apparel, Banking and for e-government

2.      Gate One, an open source terminal emulator and SSH client

3.      GNU Mailman, one of the more popular packages for running email mailing lists

4.      MoinMoin, a wiki engine

5.      OpenERP, new name Odoo, an open source comprehensive suite of business applications

6.      Planet, a feed aggregator

7.      Plone, an open source content management system

8.      Pyrat is a Laboratory Animal Facility Management Software (or Lab Animal Colony Management Software) written in Python. Its name means "Python based Relational Animal Tracking"

9.      Roundup, a bug tracking system

10.  Tor2web, an HTTP proxy for Tor Hidden Services (HS)

11.  Trac, web-based bug/issue tracking database, wiki, and version control front-end

12.  Turntable.fm, a discontinued social media site for interactively sharing music

13.  ViewVC, a web-based interface for browsing CVS and SVN repositories.

14.  Instagram

15.  Pinterest

16.  Disqus

17.  Bitbucket

18.  Mozilla

.........and many any more !

Video games:

1.      Bridge Commander

2.      Civilization IV uses Python for most of its tasks

3.      Disney's Toontown Online is written in Python and uses Panda3D for graphics.

4.      Battlefield 2 uses Python for all of its addons and a lot of its functionality

5.      Eve Online uses Stackless Python

6.      Freedom Force

7.      Frets on Fire is written in Python and uses Pygame

8.      The Temple of Elemental Evil, a computer role-playing game based on the classic Greyhawk Dungeons & Dragons campaign setting

9.      Unity of Command (video game) is an operational-level wargame about the 1942/43 Stalingrad Campaign on the Eastern Front.

10.  Vampire: The Masquerade – Bloodlines, a computer role-playing game based on the World of Darkness campaign setting

11.  Vega Strike, an open source space simulator, uses Python for internal scripting

12.  World of Tanks uses Python for most of its tasks

.........

Web Frame Works: Python has the most number of rapid development Web frameworks out there . You can create a fully functional blog in literally 10 minutes! .

1.      CherryPy, an object-oriented web application server and framework

2.      Django, an MVC (model, view, controller) web framework

3.      Bottle, A fast, simple and lightweight WSGI micro web framework

4.      Flask, a modern, lightweight, well-documented microframework based on Werkzeug and Jinja 2

5.      Google App Engine, a platform for developing and hosting web applications in Google-managed data centers, including Python.

6.      Pylons, a lightweight web framework emphasizing flexibility and rapid development

7.      Pyramid, is a minimalistic web framework inspired by Zope, Pylons and Django

8.      Quixote, a framework for developing Web applications in Python

9.      Topsite Templating System, another Python-powered web framework

10.  TurboGears, a web framework combining SQLObject/SQLAlchemy, Kid/Genshi, and CherryPy/Pylons

11.  web2py, a full-stack enterprise web application framework, following the MVC design

12.  Zope, an application server, commonly used to build content management systems

13.  Tornado, a lightweight non-blocking server and framework

........

Graphics frameworks:

1.      Pygame, Python bindings for SDL

2.      Panda3D, a 3D game engine for Python

3.      Python Imaging Library, a module for working with images

4.      Python-Ogre, a Python Language binding for the OGRE 3D engine

5.      Soya3D, a high-level 3D game engine for Python

.......

UI frameworks:

1.      Kivy, open source Python library for developing multitouch application software with a natural user interface (NUI).

2.      PyGTK, a popular cross-platform GUI library based on GTK+; furthermore, other GNOME libraries also have bindings for Python

3.      PyQt, another cross-platform GUI library based on Qt; as above, KDE libraries also have bindings

4.      PySide, an alternative to the PyQt library, released under the BSD-style licence

5.      Tkinter is Python's de facto GUI it is shipped in most versions of Python and is integrated in the IDLE. It is based Tcl command tool.

6.      wxPython, a port of wxWidgets and a cross-platform GUI library for Python

Scientific packages:

1.      Astropy, a library of Python tools for astronomy and astrophysics.

2.      Biopython, a Python molecular biology suite

3.      graph-tool, a Python module for manipulation and statistical analysis of graphs.

4.      Pathomx, a workflow-based metabolomics analysis tool.

5.      NetworkX, a package for the creation, manipulation, and study of complex networks.

6.      SciPy, a library of scientific and numerical routines

7.      scikit-learn, a library for machine learning.

8.      scikit-image, a library for image processing.

9.      SymPy, a symbolic calculation package

10.  TomoPy, a package for tomographic data processing and image reconstruction

11.  Veusz, a scientific plotting package

12.  VisTrails, a scientific workflow and provenance management software with visual programming interface and integrated visualization (via Matplotlib, VTK).

Mathematical libraries:

1.      Matplotlib, an extension providing MATLAB-like plotting and mathematical functions

2.      Plotly is a scientific plotting library for creating browser-based graphs.

3.      NumPy, a language extension that adds support for large and fast, multi-dimensional arrays and matrices

4.      Sage is a large mathematical software application which integrates the work of nearly 100 free software projects and supports linear algebra, combinatorics, numerical mathematics, calculus, and more.

5.      SymPy, a symbolic mathematical calculations package.

....

Additional development packages:

1.      Cheetah, a Python-powered template engine and code-generation tool

2.      Construct, a python library for the declarative construction and deconstruction of data structures

3.      IPython, a development shell both written in and designed for Python

4.      Jinja, a Python-powered template engine, inspired by Django's template engine

5.      mod python, an Apache module allowing direct integration of Python scripts with the Apache web server

6.      PYthon Remote Objects, a Distributed Object Technology

7.      PyObjC, a Python to Objective-C bridge that allows writing Mac OS X software in Python

8.      Setuptools, a package development process library designed to facilitate packaging Python projects by enhancing the Python distutils (distribution utilities) standard library.

9.      Sphinx (documentation generator), which converts reStructuredText files into HTML websites and other formats including PDF, EPub and Man pages

10.  Twisted, a networking framework for Python

11.  VPython, the Python programming language plus a 3D graphics module called Visual

Embedded as a scripting language: Python is, or can be used as the scripting language in these software products.

1.      Abaqus (Finite Element Software)

2.      ADvantage Framework

3.      Amarok

4.      ArcGIS, a prominent GIS platform, allows extensive modelling using Python

5.      Autodesk Maya, professional 3D modeler allows Python scripting as an alternative to MEL as of version 8.5

6.      Autodesk MotionBuilder

7.      Autodesk Softimage (formerly Softimage|XSI)

8.      BioNumerics a bioinformatics software suite for the management, storage and (statistical) analysis of all types of biological data.

9.      Blender

10.  Boxee, a cross-platform home theater PC software

11.  Cinema 4D

12.  Corel Paint Shop Pro

13.  Claws Mail with Python plugin

14.  Cyme, an Electrical Power engineering software

15.  DSHub

16.  Editorial, a commercial text editor for iOS

17.  ERDAS Imagine

18.  EventScripts, plugin for Valve's Source engine

19.  FreeCAD

20.  gedit

21.  GIMP

22.  GNAT The GNAT programming chain tool (Ada language implementation in GNU gcc), as a GNATcoll reusable components for the applications (with or without PyGTK) and as a scripting language for the commands in the GPS programming environment

23.  Houdini highly evolved 3D animation package, fully extensible using python

24.  Inkscape, a free vector graphics editor

25.  MeVisLab, a medical image processing and visualization software, uses Python for network scripting, macro modules, and application building

26.  Modo

27.  Micromine

28.  Minecraft: Pi Edition (game)

29.  MSC.Software's CAE packages: Adams, Mentat, SimXpert

30.  MySQL Workbench, a visual database design tool

31.  Notepad++ has a plugin named PythonScript that allows scripting Notepad++ in Python

32.  Nuke (compositing for visual effects)

33.  ParaView, an opensource scientific visualization software

34.  Poser, a 3D rendering and animation computer program that uses for scripting a special dialect of Python, called PoserPython

35.  PyMOL, a popular molecular viewer that embeds Python for scripting and integration

36.  QGIS uses Python for scripting and plugin-development

37.  Rhinoceros 3D version 5.0

38.  Rhythmbox

39.  Scribus

40.  3DSlicer, medical image visualisation and analysis software. Python is available for algorithm implementation, analysis pipelines, and GUI creation.

41.  SPSS statistical software SPSS Programmability Extension allows users to extend the SPSS command syntax language with Python

42.  Totem, a media player for the GNOME desktop environment

43.  Vim

44.  VisIt

45.  WeeChat, a console IRC client

Commercial uses:

1.      CCP hf uses Stackless Python in both its server and client side applications for its MMO Eve Online

2.      NASA is using Python to implement a CAD/CAE/PDM repository and model management, integration, and transformation system which will be the core infrastructure for its next-generation collaborative engineering environment. It is also the development language for OpenMDAO, a framework developed by NASA for solving multidisciplinary design optimization problems.

3.      Google uses heavily using python for various of its applications

4.      Reddit was originally written in Common Lisp, but was rewritten in Python in 2005

5.      Yahoo! Groups uses Python "to maintain its discussion groups"

6.      YouTube uses Python "to produce maintainable features in record times, with a minimum of developers"

7.      Enthought uses python as the main language for countless custom applications in Geophysics, Financial applications, Astrophysics, simulations for consumer product companies, ...

8.      EDF uses Python for their finite element solver Code Aster in combination with Fortran

Python Implementation variants:

1.      ActivePython, the Python implementation from ActiveState, includes a Windows Script Host scripting engine called PythonScript. Scripts written in PythonScript have the default file name extension of .pys.

2.      CPython - the reference implementation, written in C89.

3.      CLPython - implementation, written in Common Lisp

4.      Cython is a programming language to simplify writing C and C++ extension modules for the CPython Python runtime.

5.      IronPython - Python for .NET and Mono platforms

6.      Jython - Python coded in Java

7.      Parrot - virtual machine being developed mainly as the runtime for Perl 6, but with the intent to also support dynamic languages like Python, Ruby, Tcl, etc.

8.      Psyco - not an implementation, but JIT compiler for CPython

9.      PyPy - Python (originally) coded in Python, used in conjunction RPython, a restricted subset of Python that is amenable to static analysis and therefore a JIT.

10.  Stackless Python - Python with coroutines

11.  Unladen Swallow - the Python implementation from the Google team

12.  Nuitka - Python to C++ compiler.

 .....

Note;

Rarely one could find a programming language with so much of diversity and used in so many scientific and non scientific fields.

Being open-source, python has very large community base. The community of Python users is where lies its "undying" spirit .

Reference:
1. Wikipedia
2. Quora.com