veyepar: Python (and Fortran and C) as used in large-scale number crunching and scientific programming

Author(s):	Catherine Moroney
Location	Mystic Theatre
Date	dec Sat 02	Days Raw Files
Start	15:00	First Raw Start	14:34
Duration	0:45:0	Offset	0:25:09
End	15:45	Last Raw End	16:04
Chapters	00:00 0:17:29
Total cuts_time	29 min.

https://2017.northbaypython.org/schedule/presentation/24/
raw-playlist raw-mp4-playlist encoded-files-playlist host archive tweet mp4 svg png

assets release.pdf Python_and_Fortran_and_C_as_used_in_largescale_number_crunching_and_scientific_programming.json
logs
Admin: episode episode list cut list raw files day marks day marks day image_files

State:

Locked:

clear this to unlock

Locked by:

user/process that locked.

Start:

initially scheduled time from master, adjusted to match reality

Duration:

length in hh:mm:ss

Name:

Video Title (shows in video search results)

Emails:

email(s) of the presenter(s)

Released:

has someone authorised pubication

Normalise:

Channelcopy:

m=mono, 01=copy left to right, 10=right to left, 00=ignore.

Thumbnail:

filename.png

Description:

markdown

Python is an incredibly powerful and versatile language, but it is not practical for lots of deeply nested loops because it's not compiled, and a lot of the big scientific libraries only have C interfaces.  But, if you bring Fortran (via f2py) and C (Cython) into the mix then you can use the best features of all three languages and shrink your development/maintenance time and resulting code size by almost an order of magnitude.

My satellite has a lot of legacy code in a Perl/C/Fortran mixture that is incredibly difficult to read and maintain, but our new software is in a Python/C/Fortran mixture (with Python being the bulk of the code) that is a fraction of the size of the old code, much much easier to develop and maintain, yet it has the speed and stability to be used in our production system.

I use Python for the upper and middle layers, Fortran for the few places where I need to do lots of looping that cannot be transformed into whole-array operations, and then C to interface with the various 3rd party libraries.   I can wrap Fortran and C using f2py and Cython respectively, call them directly from the Python, and pass numpy arrays back and forth seamlessly.

I also have layers of scripts:  a Python code to run a single orbit's worth of data through a single stage of the processing pipeline,  a second Python script to run multiple orbits through a single stage, and then a third script that runs the entire processing chain through many orbits, thereby using Python's numerous scripting capabilities.

By using all three languages as described above, I truly get the best of all worlds:  dramatically reduced development time, a code base that is a fraction of the size of the old legacy system, yet all the speed and stability I need to process huge amounts of data.

Comment:

production notes