CS 475 / 575 -- Parallel Programming

Spring Quarter 2025

http://cs.oregonstate.edu/~mjb/cs575

Class Resources Page

Projects	Handouts	Office Hours

This page was last updated: April 19, 2025

Attention Ecampusers!

A special welcome to all you Ecampusers! It is good to have a cohort here that comes from all over the world. That makes you very special!

You will experience the same course that the on-campus students experience. Same material, same notes, same projects, same quizzes, same tests, and same online Office Hours. There are recorded videos that go along with the notes (they are called "LV" for "Lecture Videos" and you will see them in the same table that has links to the notes).

Like I do whenever I have Ecampus students, I will setup a time every week to do a Live Lecture so that you can hear a discussion of the topics with the chance to ask live questions. For this course, it will be Wednesdays at 3:00 PDT. They are on Zoom at: https://oregonstate.zoom.us/j/8340727662?pwd=b01tZ0hJUzdHNUtrdTRqSkdwbG4zdz09 During the LL, you can also ask questions in the Zoom Chat, and I will answer them during the LL and in a document that I will post in the same place I post the recorded LL videos. On-campus Parallelers are welcome to come as well.

These Live Lectures will be recorded so that if you miss them, you can catch them later.

Thanks, Ecampusers, for being here!

Stuff You Should Be Interested In:

• EECS is conducting a survey to find out more about your attitudes towards using Generative AI tools. This would be really, really useful to us as we (like everyoine else) struggle with how best to integrate AI intro our teaching enterprise. Take the survey here: https://tinyurl.com/reflect-ai
  And, there is a raffle for a $50 Amazon gift card for taking the survey.

• All of you have been added to the cs475-575 account to use the COE DGX system (Projects 5 and 6) and the COE cluster (Project 7). A number of you, however, have not activated an ENGR account. This is understandable, especially if you are not an engineering major, but it might prevent you from accessing these resources. To activate your ENGR account, log into TEACH using your ONID credentials. Then logout. After that, you are good to go!
  If you aren't sure if you have ENGR credentials, just login and logout anyway.

• Do not turn on the optimizer in any of your parallel projects! When you do performance benchmarks, like we are doing, it looks to the compiler like we are doing computing for the sake of computing. That is, it appears we are computing numbers but not doing anything with them. A good optimizer will decide to do you a big fat favor and eliminate the code that it thinks is accomplishing nothing. If you see that your performance is quite high, or you see that multicore is making no difference, that is probably what is happening.

• Alert student Vincent Wong has been writing a handbook that is a guide for CS Ecampus students. Find it here!

• A former TA created a guide to getting onto the flip server. Check it out here.

• The OSU Academic Success Center has a simple-but-effective term-at-a-glance-calendar to fill with events and assignments. If you struggle with keeping track of everything (and I am still trying to find the best ways to make that work for me...), check it out here.

• There is an interesting article called The New Golden Age for Computer Architecure written by two of the biggest names in computer architecture. Get it here. (Reading this isn't needed for this class, but it is an interesting article.)

• So that you can plan for the tests:
  • There will be two tests. Each will be taken on Canvas, will be multiple choice, and will be open-notes (no proctoring -- I'm counting on your honesty). You will have 60 minutes.
  • Test #1 will open in Week #5 on Wednesday, April 30 at 12:01 AM PDT (one minute after midnight). It will close on Saturday, May 3, at 11:59 PM PDT (one minute before midnight).
  • Test #2 will open in Finals Week on Wednesday, June 11 at 12:01 AM PDT (one minute after midnight). It will close on Saturday, June 14, at 11:59 PM PDT (one minute before midnight).

• Want to see if your Windows system supports OpenCL? Grab these two files:
  1. PrintInfo.exe
  2. printinfo.cl

  (You might have to change permissions on PrintInfo.exe to make it executable.) Then double-click on PrintInfo.exe -- it will print out the OpenCL characteristics of your system into a file called printinfo.out -- if it does all of that, you have OpenCL.

• The source is in printinfo.cpp so that you can compile it yourself. (There are compilation instructions in the OpenCL noteset.) You will also need printinfo.cl in the same folder as the executable.

• Want to see if your Linux system supports OpenCL? Grab these two files:
  1. printinfo
  2. printinfo.cl

  Then type: chmod 0755 printinfo
  Then type: ./printinfo -- it will print out the OpenCL characteristics of your system into a file called printinfo.out -- if it does all of that, you have OpenCL.

• If you want to run Project 6 on your ARM M1 Macs, here are the compiler flags:
  clang -framework OpenCL first.cpp -Xpreprocessor -fopenmp -I/usr/local/include -L/usr/local/lib -lomp -lc++ -o first
  You needed to change the include to the Apple OpenCL framework as well:
  #include <OpenCL/opencl.h>
  #include <OpenCL/cl_platform.h>

• Here is the tragic video of a train hitting one of the OSU food robots. (Poor little guy.)

• It has been reported that you can get Microsoft Office 365 for free here.

• It has been reported that this is how you compile OpenMP on a Mac:

   g++  -Xpreprocessor  -fopenmp   -I/opt/homebrew/Cellar/libomp/14.0.0/include  -L/opt/homebrew/Cellar/libomp/14.0.0/lib  -lomp   *.cpp  -o  main

• But, Stackoverflow reports that this is how to compile OpenMP on a Mac:

  clang  -Xpreprocessor  -fopenmp  -I/usr/local/include  -L/usr/local/lib  -lomp    main.c  -o  main

  and

  clang++  -Xpreprocessor  -fopenmp  -I/usr/local/include  -L/usr/local/lib  -lomp    main.cpp  -o  main

• The error code-printing C/C++ function from the OpenCL notes is here.

• You can get some of the Microsoft packages for free. This includes Visual Studio. To do this, go to: https://azureforeducation.microsoft.com/devtools. Click the blue Sign In button on the right. Login using your ONID@oregonstate.edu username and password.

  The package you really want is Visual Studio Enterprise 2022.

• A former TA prepared a document covering common Troubleshooting Problems with Visual Studio. Check it out here.

• Click here to see events happening in EECS.

Graduation! Graduation! Graduation!

1. On Friday, June 13 (the Friday of Finals Week), EECS will hold its annual Graduation Celebration. The event runs from 3:00-5:30 at Gill Coliseum and there is a reception afterwards in the plaza just outside. Invite your friends and families to join us to celebrate your extraordinary achievement! Ecampusers: this includes you too! Go here for more information. Registration has opened!

   BTW, this will all be livestreamed for your far-flung friends and relatives! The link will be posted here.

2. The all-university graduation is the next day, Saturday, June 14. Ecampusers: this includes you too! Go here for more information.

3. And, as if that isn't enough, Ecampus is holding a special reception for all online graduates, Saturday morning, June 14, 7:30-9:00 AM, leaving you plenty of time to lineup for the all-university graduation. Go here to learn all about it.

Job-Hunting Information:

• I have heard from a number of students that they have been contacted by companies they've never heard of because of their LinkedIn page. If you don't have one, get one! If you have one, be sure it has all the keywords that describe what you are interested in. And be sure to check the "Looking For Work" box.

• If you'd like your resume proofread and commented on, email a PDF of it to me.

• The Engineering Expo is Friday, June 6. It is held at the CH2M-Hill Alumni Center and the La Sells Stewart Center. (It's big enough that it takes 2 buildings to hold it.)
  If you are a member of a capstone team now, be sure to bring a stack of resumes with you. The Expo is always full of visiting industry people. (We invite them to campus that day for the annual Industry Advisory Board meeting.)
  If you are doing capstone next year, be sure to go so you know what it's like!

• Even though it is the start of the quarter, I have always believed that you can list, on your resume, the skills that we will be covering throughout the term. This is because you will actually have acquired those skills by the time you are on the job. If you would like to do this, here are the skills to list: OpenMP, SIMD, CUDA, OpenCL, and MPI.

• The Spring All-Industries Career and Internship Fair is April 16, 11:00-3:00 PDT at the CH2M-Hill Alumni Center. Additional information is likely to be posted here.

• The student Career and Networking Events Calendar can be found here.

• Interviewing this year?
  There is a great book called: Programming Interviews Exposed: Secrets to Landing Your Next Job
  It prepares you for CS interview questions by revealing strategies and giving a nice review of data structures, recursion, and other programming information you might be asked about.
  Best of all, the OSU library has it in electronic form. Click here and then click on "Read Online".
  (I have a paper copy of it if you'd like to see it. Also, it's around $30 on Amazon

My Resume Advice:

I moved the resume advice to another web page. Click here to see it.

What We Will Be Doing

The goals of this course are to leave you "career-ready" (i.e., both work-ready and research-ready) for tasks that require you to speed the execution of programs using parallelism.

CS 475/575 topics include:

• Moore's Law and Multicore
• Multicore programming using OpenMP
• Speedups and Amdahl's Law
• Hyperthreading
• Caching issues and False Sharing
• Data Decomposition
• Functional Decomposition
• Single Instruction Multiple Data (SIMD)
• GPU Computing, including CUDA and OpenCL
• OpenCL / OpenGL Interoperability
• Message Passing Interface (MPI)

Prerequisites and Course Incoming Expectations

Professor

Office:	Kelley 2117 (2nd floor, south side)
Email:	mjb@cs.oregonstate.edu
Phone:	541-737-2542

Email is, above everything else, the best way to reach me. Don't send me messages on Canvas -- I will probably never see them.

TAs

The TAs for this course are: Pavlo Havrylyuk, Aakash Kulkarni, Manoj Kumar Gummadi, Shubhangi Pandey, Uma Kanth Pendyala, Ryan Roberts, Suriyakrishnan Sathish, Suyash Sreekumar, and Jing Wang.

Office Hours

Live Lectures

The Live Lectures are on Zoom at: https://oregonstate.zoom.us/j/8340727662?pwd=b01tZ0hJUzdHNUtrdTRqSkdwbG4zdz09

During the LL, you can ask questions just by speaking or raising your Zoom Hand. You can also type in the Zoom Chat, and I will answer those questions during the LL and in a document that I will post in the same place as I post the recorded LL videos.

Attendance at the Live Lectures is optional, but the material we cover there could be on quizzes and tests. The material will definitely help you on the projects. These sessions will be recorded (see below) in case you missed them.

Lurking is welcome and encouraged. Come, turn of your microphone and camera, and listen in. You could even be working on something else at the same time. It's much better to be there and catch bits and pieces than to not be there at all!

LLV = Live Lecture Video

LL #	Chat	Video
1	PDF	LLV
2	PDF	LLV
3	PDF	LLV

Notes in place of a textbook

On-campus Section of the Class

Our on-campus class time is: Monday and Wednesday, 12:00 - 1:50. All of our classes will be in the Learning Innovation Center (LiNC), room 128.

Online Section of the Class

The lectures for the Ecampus version of the class will be asynchronous, that is, you are free to review the notes and watch the videos whenever it suits you. But, be careful not to fall behind! The programming projects, and their due dates, will assume that you have been keeping up with the material.

Both Sections of the Class

Everyone learns in different ways. As such, feel free to attend/use resources that have been setup for the other class modality.
Ecampus students, if you are in the area, feel free to attend the on-campus class.
On-campus students, feel free to watch the videos and attend the Live Lectures.
Everyone should be studying the notes!

Note Handouts and Recorded Videos

Week	Topic
1	Class Resources Page	HTML				LV
1	Course Introduction	1pp	2pp	4pp	6pp	LV
1	Project Notes	1pp	2pp	4pp	6pp	LV
1	Scripting	1pp	2pp	4pp	6pp	LV
1	Graphing	1pp	2pp	4pp	6pp	LV
1	Pivot Tables	1pp	2pp	4pp	6pp	LV
1	Windows Powershell Scripting	1pp	2pp	4pp	6pp	LV
1	Simple OpenMP	1pp	2pp	4pp	6pp	LV
1	Parallel Programming: Background Information	1pp	2pp	4pp	6pp	LV
2	OpenMP	1pp	2pp	4pp	6pp	LV-1	LV-2
2	OpenMP 4.0 Quick Reference Card	PDF
2	Reduction with OpenMP by showing Trapezoid Integration	1pp	2pp	4pp	6pp	LV
2	Speedups and Amdahl's Law	1pp	2pp	4pp	6pp	LV
2	Moore's Law and Multicore	1pp	2pp	4pp	6pp	LV
2	Functional Decomposition	1pp	2pp	4pp	6pp	LV
2	A Tale of Two Assembly Codes	1pp	2pp	4pp	6pp	LV
3	Hyperthreading and "Almost Amdahl"	1pp	2pp	4pp	6pp	LV
3	Caching Issues in Multicore Performance	1pp	2pp	4pp	6pp	LV-1	LV-2
3	OpenMP Tasks	1pp	2pp	4pp	6pp	LV
4	Data Decomposition	1pp	2pp	4pp	6pp	LV
4	SIMD Vector Parallel Programming	1pp	2pp	4pp	6pp	LV
4	Prefetching	1pp	2pp	4pp	6pp	LV
5	GPU 101	1pp	2pp	4pp	6pp	LV
5	Test #1 Review	HTML
6	CUDA	1pp	2pp	4pp	6pp	LV
6	CUDA Quick Reference Card	PDF
6	DGX System	1pp	2pp	4pp	6pp	LV
6	CUDA Array Multiplication	1pp	2pp	4pp	6pp	LV
6	CUDA Matrix Multiplication	1pp	2pp	4pp	6pp	LV
7	Transition CUDA ↔ OpenCL	1pp	2pp	4pp	6pp	LV
7	OpenCL	1pp	2pp	4pp	6pp	LV-1	LV-2	LV-3
7	OpenCL 2.0 Quick Reference Card:	PDF
7	OpenCL Events	1pp	2pp	4pp	6pp	LV
8	OpenCL Array Multiplication	TXT
8	OpenCL Reduction	1pp	2pp	4pp	6pp	LV
8	Looking At OpenCL Assembly language	1pp	2pp	4pp	6pp	LV
8	OpenCL / OpenGL Vertex Buffer Interoperability	1pp	2pp	4pp	6pp	LV-1	LV-2
9	Message Passing Interface (MPI)	1pp	2pp	4pp	6pp	LV-1	LV-2	LV-3
9	Compute:Communicate Ratio	1pp	2pp	4pp	6pp	LV
10	Combined Parallelism	1pp	2pp	4pp	6pp	LV
10	Finding More Information	PDF
10	Test #2 Review	HTML

Downloadable Files

Projects

Project Turn-In Procedures

• Your project turnins will be electronic.

• Your project turn-ins will all be done on Canvas. You will turn in your source and PDF files:
  1. Your source files: .cpp, .cu, .cl
  2. A short PDF report containing:
     • A cover page (name, email address, project name)
     • Key snippets of code, if applicable
     • Tables of data
     • Graphs of data
     • Your conclusions: How did things turn out? Why do you think they turned out that way?

• To see how to turn these files in to Canvas, go to our Project Notes noteset, and go the the slide labeled How to Turn In a Project on Canvas.

• Do not zip, tar, rar, etc. any of the files. Just upload them all individually.

• Electronic submissions are due at 23:59 PT on the listed due date.

• Your project will be graded and the score posted to your Canvas Grades. If you lost any points, there will be a Canvas comment about it. If you want to reply to the comment, please do it via email, not the Canvas Comments -- I am much more reachable on email.

Bonus Days and Late Assignments

Projects are due at 23:59:59 on the listed due date, with the following exception:

Each of you has been granted 5 Bonus Days, which are no-questions-asked one-day extensions which may be applied to any project, subject to the following rules:

1. No more than 2 Bonus Days may be applied to any one project
2. Bonus Days cannot be applied to quizzes
3. Bonus Days cannot be applied to tests
4. Bonus Days cannot be applied to Project #7 or the 575 Paper Project

Bonus Days are to help enable your success!
They are not there to enable your procrastination.
Save them for when you really need them.

If you turn in a project three or more days late, the score is a zero.

If you turn in a project late and don't have enough Bonus Days left to spend on it, the score is a zero.

You don't need to ask me, or even tell me. that you are using Bonus Days. Just turn your project in late. Canvas will check your turn-in date and tell me how to deduct the proper number of Bonus Days.

It is your job to track your Bonus Day usage. But, if you lose track, feel free to email me and ask me to look it up.

Grading

Class Rules

Other Notes You Might Enjoy!

Other Useful Parallel Programming Information

• OpenMP:
  • Industry-wide OpenMP information: http://www.openmp.org
  • Current OpenMP Specification (version 5.2): https://www.openmp.org/wp-content/uploads/OpenMP-API-Specification-5-2.pdf
  • OpenMP 4.5 Quick Reference Card: https://www.openmp.org/wp-content/uploads/OpenMP-4.5-1115-CPP-web.pdf
  • OpenMP Quick Reference Sheet: http://www.plutospin.com/files/OpenMP_reference.pdf

• Single-Instruction-Multiple-Data (SIMD):
  • Good explanation of SSE arithmetic: http://www.tommesani.com/SSEArithmetic.html

• CUDA:
  • About CUDA: https://developer.nvidia.com/about-cuda
  • CUDA Quick Start Guide: https://docs.nvidia.com/cuda/pdf/CUDA_Quick_Start_Guide.pdf
  • CUDA C Programming Guide: https://docs.nvidia.com/cuda/pdf/CUDA_C_Programming_Guide.pdf
  • cuSolver Library: https://docs.nvidia.com/cuda/pdf/CUSOLVER_Library.pdf

• OpenCL:
  • Industry-wide OpenCL Information: http://www.khronos.org/opencl
  • OpenCL Reference (~man) Pages http://www.khronos.org/registry/cl/sdk/1.0/docs/man/xhtml
  • OpenCL 2.0 Quick Reference Card: https://www.khronos.org/files/opencl20-quick-reference-card.pdf
  • NVIDIA's OpenCL example code: http://developer.nvidia.com/opencl-sdk-code-samples
  • Prof. Bailey's OpenCL status error code printer: errorcodes.cpp
  • NVIDIA OpenCL Code Examples https://developer.nvidia.com/opencl