CS 475 / 575 -- Parallel Programming

Spring Quarter 2023

http://cs.oregonstate.edu/~mjb/cs575

Class Resources Page

Projects	Handouts	Bailey Zoom

This page was last updated: June 9, 2023

Coronavirus Information

Your well-being is our #1 concern! I want all of you to have ready-access to the latest information. Find it here.

Attention Ecampusers!

A special welcome to all you Ecampusers! It is good to have a cohort here that comes from all over the world. That makes you very special!

You will experience the same course that the on-campus students experience. Same material, same notes, same projects, same quizzes, same tests, and same online Office Hours. There are recorded videos that go along with the notes (they are called "LV" for "Lecture Videos" and you will see them in the same table that has links to the notes).

Like I do whenever I have Ecampus students, I have setup a time every week to do a Live Lecture so that you can hear a discussion of the topics with the chance to ask live questions. For this course, it will be Wednesdays at 3:00 PDT. They are on Zoom at: https://oregonstate.zoom.us/j/8340727662?pwd=b01tZ0hJUzdHNUtrdTRqSkdwbG4zdz09 During the LL, you can also ask questions in the Zoom Chat, and I will answer them during the LL and in a document that I will post in the same place I post the recorded LL videos. On-campus Parallelers are welcome to come as well.

These Live Lectures will be recorded so that if you miss them, you can catch them later.

Thanks, Ecampusers, for being here!

Stuff You Should Be Interested In:

• I need your help! Your thoughts on your experiences here matter to us!
  I use a short Qualtrics survey to help me evolve this course from year-to-year. Click here to take it.
  Also, be sure to take the OSU Student Learning Experience (SLE) survey as well. I use those, as does EECS, the College of Engineering, and the University.

• Prof. Bailey's Finals Week Office Hours will be:
  • Monday, June 12: 2:30-4:00
  • Tuesday, June 13: 4:00-5:00
  Usual Zoom-room: https://oregonstate.zoom.us/j/8340727662?pwd=b01tZ0hJUzdHNUtrdTRqSkdwbG4zdz09

• Graduation! Graduation! Graduation!

  1. On Friday, June 16 (the Friday of Finals Week), EECS will hold its annual Graduation Celebration. The event runs from 3:00-5:30 at Gill Coliseum and there is a reception afterwards in the plaza just outside. Invite your friends and families to join us to celebrate your extraordinary achievement! Ecampusers: this includes you too! Go here for more information. This year's keynote speaker will be Sharada Bose (1984, B.S. computer science; 1988, M.S. computer science). She is the recipient of OSU's first Diversity, Equity and Inclusion Alumni Legacy Award in 2020.

     BTW, this will all be livestreamed! The link will be posted here.

  2. The all-university graduation is the next day, Saturday, June 17. Ecampusers: this includes you too! Go here for more information. This year's commencement speaker will be Charity Dean, Ph.D., Class of 2000, CEO of The Public Health Company.

  3. And, as if that isn't enough, Ecampus is holding a special reception for all online graduates, Saturday morning, June 17, 7:30-9:00 AM, leaving you plenty of time to lineup for the all-university graduation. Go here for more information.

• So that you can plan for the tests:
  • There will be two tests. Each will be taken on Canvas, will be multiple choice, and will be open-notes (no proctoring -- I'm counting on your honesty). You will have 60 minutes.
  • Test #1 will open in Week #5 on Wednesday, May 3 at 12:01 PM PDT (one minute after noon). It will close on Sunday, May 7, at 11:59 PM PDT (one minute before midnight).
  • Test #2 will open in Finals Week on Wednesday, June 14 at 12:01 PM PDT (one minute after noon). It will close on Sunday, June 18, at 11:59 PM PDT (one minute before midnight).

• Want to see if your Windows system supports OpenCL? Grab these two files:
  1. PrintInfo.exe
  2. printinfo.cl

  (You might have to change permissions on PrintInfo.exe to make it executable.) Then double-click on PrintInfo.exe -- it will print out the OpenCL characteristics of your system into a file called printinfo.out -- if it does all of that, you have OpenCL.

• The source is in printinfo.cpp.

• Want to see if your Linux system supports OpenCL? Grab these two files:
  1. printinfo
  2. printinfo.cl

  Then type: chmod 0755 printinfo
  Then type: ./printinfo -- it will print out the OpenCL characteristics of your system into a file called printinfo.out -- if it does all of that, you have OpenCL.

• If you want to run Project 6 on your ARMs M1 Macs, here are the compiler flags:
  clang -framework OpenCL first.cpp -Xpreprocessor -fopenmp -I/usr/local/include -L/usr/local/lib -lomp -lc++ -o first
  You needed to change the include to the Apple OpenCL framework as well:
  #include <OpenCL/opencl.h>
  #include <OpenCL/cl_platform.h>

• In case you missed last year's football stadium implosion, or if you want to see it over and over again (and who doesn't like seeing such dramatic destruction?), go to: https://oregonstate.edu/implosion. and play the video. The good stuff starts around the 52:00 minute mark. It's really fun to play it back at slow speed.

  And, as a followup for curious engineers, a Q&A session was held with the people who put the implosion together. Here it is! (It's about 35 minutes.)

• Speaking of disasters, here is the video of a train hitting one of the OSU food robots. (Poor little guy.)

• It has been reported that you can get Microsoft Office 365 for free here.

• It has been reported that this is how you compile OpenMP on a Mac:

   g++  -Xpreprocessor  -fopenmp   -I/opt/homebrew/Cellar/libomp/14.0.0/include  -L/opt/homebrew/Cellar/libomp/14.0.0/lib  -lomp   *.cpp  -o  main

• But, Stackoverflow reports that this is how to compile OpenMP on a Mac:

  clang  -Xpreprocessor  -fopenmp  -I/usr/local/include  -L/usr/local/lib  -lomp    main.c  -o  main

  and

  clang++  -Xpreprocessor  -fopenmp  -I/usr/local/include  -L/usr/local/lib  -lomp    main.cpp  -o  main

• The error code-printing C/C++ function from the OpenCL notes is here.

• Interviewing this year?
  There is a great book called: Programming Interviews Exposed: Secrets to Landing Your Next Job
  It prepares you for CS interview questions by revealing strategies and giving a nice review of data structures, recursion, and other programming information you might be asked about.
  Best of all, the OSU library has it in electronic form. Click here and then click on "Read Online".
  (I have a paper copy of it if you'd like to see it. Also, it's around $30 on Amazon

• You can get some of the Microsoft packages for free. This includes Visual Studio. To do this, go to: https://azureforeducation.microsoft.com/devtools. Click the blue Sign In button on the right. Login using your ONID@oregonstate.edu username and password.

• Former TA (and now Seattle University professor) Braxton Cuneo prepared a document covering common Troubleshooting Problems with Visual Studio. Check it out here.

• The most recent gcc/g++ is version 12.2 . On OSU Linux machines, you can get at it by typing:

  module load gcc/12.2
  gcc --version
  

  and then use it from there.
  Just using "gcc" or "g++" gets you version 4.8.5 instead.
  Because OpenMP is compiler-based, I always assume that more recent versions of compilers do a better job optimizing OpenMP code. So, the newest version of gcc/g++ might do a better job It might make a difference in the performances you get. Or it mght make no difference at all. Using it is optional.

• Click here to see the calendar of student events.

What We Will Be Doing

The goals of this course are to leave you "career-ready" (i.e., both work-ready and research-ready) for tasks that require you to speed the execution of programs using parallelism.

CS 475/575 topics include:

• Moore's Law and Multicore
• Multicore programming using OpenMP
• Speedups and Amdahl's Law
• Hyperthreading
• Caching issues and False Sharing
• Data Decomposition
• Functional Decomposition
• Single Instruction Multiple Data (SIMD)
• GPU Computing, including CUDA and OpenCL
• OpenCL / OpenGL Interoperability
• Message Passing Interface (MPI)

Prerequisites and Course Incoming Expectations

Professor

Office:	Kelley 2117 (2nd floor, south side)
Email:	mjb@cs.oregonstate.edu
Phone:	541-737-2542

TAs

The TAs for this course are: Shreya Dhume, Song Gao, Samay Pusarla, Po-Hsuan Shen, Hojun Shin, Uddyan Sinha, Jian Tang, and Shengxuan Wang. All have taken this course before. We are lucky to have them!

Office Hours

Live Lectures

Here is my Zoom-room.

LLV = Live Lecture Video

Week #	Chat	Video
0	PDF	LLV
1	PDF	LLV
2	PDF	LLV
3	PDF	LLV
4	PDF	LLV
5	PDF	LLV
6	---	LLV
7	PDF	LLV
8	PDF	LLV
9	PDF	LLV
10	PDF	LLV

Notes in place of a textbook

On-campus Section of the Class

Our on-campus class time is: Monday and Wednesday, 12:00 - 1:50. All of our classes will be in the Learning Innovation Center (LiNC), room 128.

Online Section of the Class

The lectures for the Ecampus version of the class will be asynchronous, that is, you are free to review the notes and watch the videos whenever it suits you. But, be careful not to fall behind! The programming projects, and their due dates, will assume that you have been keeping up with the material.

I will also be conducting Live Lectures every Wednesday at 3:00 Pacific Time so that you have a chance to ask questions during the presentation of the material.

Both Sections of the Class

Everyone learns in different ways. As such, feel free to attend/use resources that have been setup for the other class modality.
Ecampus students, if you are in the area, feel free to attend the on-campus class.
On-campus students, feel free to watch the videos and attend the Live Lectures.
Everyone should be studying the notes!

Note Handouts and Recorded Videos

Week	Topic
1	Class Resources Page	HTML				LV
1	Course Introduction	1pp	2pp	4pp	6pp	LV
1	Project Notes	1pp	2pp	4pp	6pp	LV
1	Scripting	1pp	2pp	4pp	6pp	LV
1	Graphing	1pp	2pp	4pp	6pp	LV
1	Pivot Tables	1pp	2pp	4pp	6pp	LV
1	Windows Powershell Scripting	1pp	2pp	4pp	6pp	LV
1	Simple OpenMP	1pp	2pp	4pp	6pp	LV
1	Parallel Programming: Background Information	1pp	2pp	4pp	6pp	LV-1	LV-2
2	OpenMP	1pp	2pp	4pp	6pp	LV-1	LV-2	LV-3
2	OpenMP 4.0 Quick Reference Card	PDF
2	Trapezoid Integration with OpenMP	1pp	2pp	4pp	6pp	LV
2	Speedups and Amdahl's Law	1pp	2pp	4pp	6pp	LV
2	Moore's Law and Multicore	1pp	2pp	4pp	6pp	LV
2	Functional Decomposition	1pp	2pp	4pp	6pp	LV
2	A Tale of Two Assembly Codes	1pp	2pp	4pp	6pp
3	Hyperthreading and "Almost Amdahl"	1pp	2pp	4pp	6pp	LV
3	Caching Issues in Multicore Performance	1pp	2pp	4pp	6pp	LV-1	LV-2	LV-3	LV-4
3	OpenMP Tasks	1pp	2pp	4pp	6pp	LV
4	Data Decomposition	1pp	2pp	4pp	6pp	LV
4	SIMD Vector Parallel Programming	1pp	2pp	4pp	6pp	LV
4	Prefetching	1pp	2pp	4pp	6pp	LV
5	GPU 101	1pp	2pp	4pp	6pp	LV
5	Test #1 Review	HTML				LV
6	CUDA	1pp	2pp	4pp	6pp	LV
6	CUDA Quick Reference Card	PDF
6	DGX System	1pp	2pp	4pp	6pp	LV
6	CUDA Array Multiplication	1pp	2pp	4pp	6pp	LV
6	CUDA Matrix Multiplication	1pp	2pp	4pp	6pp	LV
7	Transition CUDA ↔ OpenCL	1pp	2pp	4pp	6pp	LV
7	OpenCL	1pp	2pp	4pp	6pp	LV-1	LV-2	LV-3	LV-4
7	OpenCL 2.0 Quick Reference Card:	PDF
7	OpenCL Matrix Multiplication	1pp	2pp	4pp	6pp	LV
7	OpenCL Events	1pp	2pp	4pp	6pp	LV
8	OpenCL Array Multiplication	TXT
8	OpenCL Reduction	1pp	2pp	4pp	6pp	LV
8	Looking At OpenCL Assembly language	1pp	2pp	4pp	6pp	LV
8	OpenCL / OpenGL Vertex Buffer Interoperability	1pp	2pp	4pp	6pp	LV-1	LV-2
9	Message Passing Interface (MPI)	1pp	2pp	4pp	6pp	LV-1	LV-2	LV-3
9	Compute:Communicate Ratio	1pp	2pp	4pp	6pp	LV
10	Parallelism Jeopardy	1pp	2pp	4pp	6pp	LV
10	Finding More Information	PDF
10	Test #2 Review	HTML				LV

Downloadable Files

Projects

Project Turn-In Procedures

• Your project turnins will be electronic.

• Your electronic turnin will be done at http://teach.engr.oregonstate.edu and will consist of:
  
  1. A PDF report. The PDF file should include:
     • A cover page (name, email address, project name)
     • Key snippets of code, if applicable
     • Tables of data
     • Graphs of data
     • Your conclusions: how did things turn out? Why?
  2. Source files of everything (.cpp, .cu, .cl).

• You can zip up all the source files you are turning in if you want. But, do not put the PDF file in a .zip. Please turn it in separately so I can run my gather-up-all-the-PDFs script.

• Submissions are due at 23:59:59 on the listed due date.

• Your project will be graded and the score posted to your Canvas Grades. If you lost any points, there will be a Canvas comment about it. If you want to reply to the comment, please do it via email, not the Canvas Comments -- I am much more reachable on email.

Bonus Days and Late Assignments

Projects are due at 23:59:59 on the listed due date, with the following exception:

Each of you has been granted 5 Bonus Days, which are no-questions-asked one-day extensions which may be applied to any project, subject to the following rules:

1. No more than 2 Bonus Days may be applied to any one project
2. Bonus Days cannot be applied to quizzes
3. Bonus Days cannot be applied to tests
4. Bonus Days cannot be applied to Project #7 or the 575 Paper Project

If you turn in a project three or more days late, the score is a zero.

If you turn in a project late and don't have enough Bonus Days left to spend on it, the score is a zero.

You don't need to ask me, or even tell me. that you are using Bonus Days. Just turn your project in late. I have a script that will check your turn-in date and deduct the proper number of Bonus Days.

It is your job to track your Bonus Day usage. But, if you lose track, feel free to email me and ask me to look it up.

Grading

All programming projects will be turned in on Teach, http://teach.engr.oregonstate.edu .

The quizzes will be done on Canvas. They will open each Friday at 12:01 PM and will close Sunday night at 23:59 (=11:59 PM). Canvas is very unforgiving about due times -- don't push it.

Your final grade will be based on your overall class point total. Based on an available point total of 1110, grade cutoffs will be no higher than:

Points	Grade
1060	A
1040	A-
1020	B+
1000	B
980	B-
960	C+
940	C
920	C-
900	D+
880	D
860	D-

Notice that this grade scale is not 90%-80%-70%-60%. That is because I do such a soft grade on the projects.

Class Rules

Other Notes You Might Enjoy!

Other Useful Parallel Programming Information

• OpenMP:
  • Industry-wide OpenMP information: http://www.openmp.org
  • Current OpenMP Specification (version 5.2): https://www.openmp.org/wp-content/uploads/OpenMP-API-Specification-5-2.pdf
  • OpenMP 4.5 Quick Reference Card: https://www.openmp.org/wp-content/uploads/OpenMP-4.5-1115-CPP-web.pdf
  • OpenMP Quick Reference Sheet: http://www.plutospin.com/files/OpenMP_reference.pdf

• Single-Instruction-Multiple-Data (SIMD):
  • Good explanation of SSE arithmetic: http://www.tommesani.com/SSEArithmetic.html

• CUDA:
  • About CUDA: https://developer.nvidia.com/about-cuda
  • CUDA Quick Start Guide: https://docs.nvidia.com/cuda/pdf/CUDA_Quick_Start_Guide.pdf
  • CUDA C Programming Guide: https://docs.nvidia.com/cuda/pdf/CUDA_C_Programming_Guide.pdf
  • cuSolver Library: https://docs.nvidia.com/cuda/pdf/CUSOLVER_Library.pdf

• OpenCL:
  • Industry-wide OpenCL Information: http://www.khronos.org/opencl
  • OpenCL Reference (~man) Pages http://www.khronos.org/registry/cl/sdk/1.0/docs/man/xhtml
  • OpenCL 2.0 Quick Reference Card: https://www.khronos.org/files/opencl20-quick-reference-card.pdf
  • NVIDIA's OpenCL example code: http://developer.nvidia.com/opencl-sdk-code-samples
  • Prof. Bailey's OpenCL status error code printer: errorcodes.cpp
  • NVIDIA OpenCL Code Examples https://developer.nvidia.com/opencl