CS 475 / 575 -- Parallel Programming

Spring Quarter 2021

http://cs.oregonstate.edu/~mjb/cs575

Resources Page

Projects	Handouts	Ed Discussion	PB Zoom

This page was last updated: May 12, 2021

Coronavirus Information

• If you are in the Corvallis area, click here to schedule your vaccination. Do it now! Please.
  (I know that some people are squeamish about getting shots. I get that. If it would help, I'd be happy to go with you. Send me an email.)
• OSU's Planning for Fall Quarter 2021
• OSU Updates
• OSU Graduate School updates
• Benton County Covid Information (including vaccines)
• US State Department
• US Centers for Disease Control and Prevention
• More communications from the US Centers for Disease Control and Prevention
• Johns Hopkins World Map
• ESRI Oregon and World Map
• Benny Beaver's advice.

Stuff You Should Be Interested In:

• This just in as of May 4: To contribute to the health and safety of our students, employees and broader community, we are announcing today that Oregon State University intends to implement a COVID-19 vaccination requirement beginning fall term 2021 for those students and employees who learn, work or engage onsite at OSU locations throughout Oregon. While students enrolled exclusively in online courses through Ecampus — and those who do not come onsite for university activities — are not required to be vaccinated, we strongly encourage everyone to obtain the vaccine as soon as possible.

• I originally wrote the SIMD code in assembly because I wanted to get at that power. And it worked great. But, I hate the long-term sustainability of assembly code so I switched to the Intel intrinsics, and they worked great … until this year, for some reason. Using the Intel intrinsics, you are all seeing around 2.5-3.0 speedup, even though 4.0 should be possible. It was for me when I first tried them. So, the assembly language code and .h file are here: simd.asm.cpp and simd.p5.h .

  If you have already run P4 and don’t want to do it again, go ahead and submit your numbers but point out to us that you have used the Intel intrinsics. If you want to really see what SIMD can do, use the assembly language. Compile something like this:
  g++ -o arraymult -DNUMK=$n arraymult.cpp simd.asm.cpp -lm -fopenmp
  (I used the NUMK in my looping script to specify how many K (x1024) there were to be in the array.) Do not turn on any optimization. -Oanything jumbles up the use of the registers and gives wrong values at best, and segmentation faults at worst.
  Sorry about this. I have no idea why this is happening now and not before. But, I do want you to know that you really can get x4 speedup (and sometimes more) with SSE.

• We are now at the halfway point in the quarter! I would appreciate it if you would take this anonymous Mid-Quarter Check-in survey. It's really short, but it helps me to think about mid-course corrections that would help you. Thanks.

• The OSU Game Development Club's Beaver Game Jam results are here.

• We stand in solidarity with members of Asian, Asian American, Native Hawaiian and Pacific Islander communities as the rise in anti-Asian racism continues in our country. We affirm that members of the OSU community who are of Asian descent belong at Oregon State, are valued members of our community and deserve to feel safe and respected. See the OSU video on this topic.

• The lectures have been recorded, but I will also be doing Live Lectures Mondays and Wednesdays 12:00 (noon) – 2:00. Unlike the recorded lectures, these will allow you to ask live questions.
  My Zoom-room is: https://oregonstate.zoom.us/j/8340727662?pwd=b01tZ0hJUzdHNUtrdTRqSkdwbG4zdz09

  If it asks for a passcode, use: ZeldaDog

  This week, the Live Lectures will cover:
  Monday, May 3: DGX (including rabbit), CUDA, CUDA Array Multiplication
  Wednesday, May 5: Go over Test #1 answers, CUDA Matrix Multiplication, Project #5

  The Live lectures will be recorded and posted where you can get at them.

• We are switching from Piazza to Ed-Discussion, which has many more features to help us help you. Read more about it here.

• So that you can plan for the tests:
  • There will be two tests. Each will be taken on Canvas, will be multiple choice, and will be open-notes (no proctoring -- I'm counting on your honesty), and you will have 60 minutes.
  • Test #1 will open in Week #5 on Thursday, April 29 at 12:01 AM PDT (one minute after midnight). It will close on Sunday, May 2, at 11:59 PM PDT (one minute before midnight).
  • Test #2 will open in Finals Week on Thursday, June 10 at 12:01 AM PDT (one minute after midnight). It will close on Sunday, June 13, at 11:59 PM PDT (one minute before midnight).

• The error code-printing C/C++ function from the OpenCL notes is here.

• Want to see if your Windows system supports OpenCL? Grab these two files:
  1. PrintInfo.exe
  2. printinfo.cl

  (You might have to change permissions on PrintInfo.exe to make it executable.) Then double-click on PrintInfo.exe -- it will print out the OpenCL characteristics of your system. If it does all of that, you have OpenCL.

• The source is in printinfo.cpp.

• Want to see if your Linux system supports OpenCL? Grab these two files:
  1. printinfo
  2. printinfo.cl

  Then execute printinfo -- it will print out the OpenCL characteristics of your system. If it does all of that, you have OpenCL.

• Interviewing this year?
  There is a great book called: Programming Interviews Exposed: Secrets to Landing Your Next Job
  It prepares you for CS interview questions by revealing strategies and giving a nice review of data structures, recursion, and other programming information you might be asked about.
  Best of all, the OSU library has it in electronic form. Click here and then click on "Read Online".
  (I have a paper copy of it if you'd like to see it. Also, it's around $21 on Amazon

• You can get some of the Microsoft packages for free. This includes Visual Studio. To do this, go to: https://azureforeducation.microsoft.com/devtools. Click the blue Sign In button on the right. Login using your ONID@oregonstate.edu username and password.

• TA Braxton Cuneo prepared a document covering common Troubleshooting Problems with Visual Studio. Check it out here.

• The most recent gcc/g++ is version 10.2.0 . On OSU Linux machines, it is hiding at /usr/local/common/gcc-10.2.0/bin/gcc and /usr/local/common/gcc-10.2.0/bin/g++
  Just using "gcc" or "g++" gets you version 4.8.5 instead.
  Because OpenMP is compiler-based, I always assume that more recent versions of compilers do a better job optimizing OpenMP. So, the newer version of gcc/g++ might do a better job, it might make no difference. Using it is optional.

• Click here to see the calendar of student events.

What We Will Be Doing

The goals of this course are to leave you "career-ready" (i.e., both work-ready and research-ready) for tasks that require you to speed the execution of programs using parallelism.

CS 475/575 topics include:

• Moore's Law and Multicore
• Multicore programming using OpenMP
• Speedups and Amdahl's Law
• Hyperthreading
• Caching issues and False Sharing
• Data Decomposition
• Functional Decomposition
• Single Instruction Multiple Data (SIMD)
• GPU Computing, including CUDA and OpenCL
• OpenCL / OpenGL Interoperability
• Message Passing Interface (MPI)

Prerequisites and Course Incoming Expectations

Professor

Office:	Kelley 2117 (2nd floor, south side)
Phone:	541-737-2542
Email:	mjb@cs.oregonstate.edu

TAs

The TAs for this course are: Braxton Cuneo, Hongyu Nie, Botong Qu, Jian Tang, Mikhail Tokarev, and Mian Xie. We are lucky to have them!

Total Office Hours

	Braxton Cuneo	Hongyu Nie	Botong Qu	Jian Tang	Mikhail Tokarev	Mian Xie	Prof. Bailey
Email @oregonstate.edu	cuneob	nieh	qub	tangjian	tokarevm	xiemia	mjb
Zoom-room	Zoom	Zoom	Zoom	Zoom	Zoom	Zoom	Zoom
Mondays		1:00-7:00	9:00-1:00	8:00-9:00 , 10:00-12:00	4:00-6:00		2:00-4:00
Tuesdays	11:00-12:00 , 2:00-4:00			8:00-9:00	4:00-6:00	8:00-11:00 , 12:00-4:00	3:00-5:00
Wednesdays	11:00-12:00	9:00-1:00		8:00-12:00 , 2:00-6:00	2:00-6:00
Thursdays	11:00-12:00 , 2:00-4:00			8:00-9:00	4:00-6:00	8:00-11:00 , 12:00-4:00	12:00-2:00
Fridays		9:00-1:00	1:00-4:00	8:00-9:00	12:00-4:00		5:00-6:00

	8:00	9:00	10:00	11:00	12:00	1:00	2:00	3:00	4:00	5:00	6:00
Mondays	JT	BQ	BQ JT	BQ JT	BQ	HN	HN	HN	HN MT	HN MT	HN
Tuesdays	MX JT	MX	MX	BC	MX	MX	MX BC	MX PB BC	MT PB	MT
Wednesdays	JT	HN JT	HN JT	HN JT BC	HN		JT MT	JT MT	JT MT	JT MT
Thursdays	MX JT	MX	MX	BC	MX PB	MX PB	MX BC	MX BC	MT	MT
Fridays	JT	HN	HN	HN	HN MT	BQ MT	BQ MT	BQ MT		PB

PB = Prof. Bailey
BC = Braxton Cuneo
HN = Hongyu Nie
BQ = Botong Qu
JT = Jian Tang
MT = Mikhail Tokarev
MX = Mian Xie
The links point to the Zoom-Rooms.

Live Lectures

Every Monday and Wednesday, starting at 12:00 noon, I will be holding Live Lectures. We will go over some current materials, some upcoming materials, and maybe some extra materials. It will be a time for you to also ask live questions. Attendance at the live session is optional, but the material we cover there is eligible to be on quizzes and tests. These sessions will be recorded (see below) in case you missed them.

Here is my Zoom-room.
If you need a passcode, use ZeldaDog

Week #	Day	Chat	Video
1	Monday	PDF	LV
1	Wednesday	PDF	LV
2	Monday	PDF	LV
2	Wednesday	PDF	LV
3	Monday	PDF	LV
3	Wednesday	PDF	LV
4	Monday	PDF	LV
4	Wednesday	PDF	LV
5	Monday	PDF	LV
5	Wednesday	PDF	LV
6	Monday	PDF	LV
6	Wednesday	PDF	LV
7	Monday	PDF	LV
7	Wednesday	PDF	LV

Notes in place of a textbook

Downloadable Files

Quizzes

Projects

Project Turn-In Procedures

• Your project turnins will be electronic.

• Your electronic turnin will be done at http://engr.oregonstate.edu/teach and will consist of:
  
  1. A PDF report. The PDF file should include:
     • A cover page (name, email address, project name)
     • Key snippets of code, if applicable
     • Tables of data
     • Graphs of data
     • Your conclusions: how did things turn out and why
  2. Source files of everything (.cpp, .cu, .cl).

• You can zip up all the files you are turning in except the PDF. Please turn it in separately so I can run my gather-up-all-the-PDFs script.

• Submissions are due at 23:59:59 on the listed due date.

• Your project will be graded and the score posted to your Canvas Grades. If you lost any points, there will be a Canvas comment about it.

Bonus Days and Late Assignments

Projects are due at 23:59:59 on the listed due date, with the following exception:

Each of you has been granted 5 Bonus Days, which are no-questions-asked one-day extensions which may be applied to any project, subject to the following rules:

1. No more than 2 Bonus Days may be applied to any one project
2. Bonus Days cannot be applied to tests
3. Bonus Days cannot be applied such that they extend a project due date past the start of Test #2.

If you turn in a project 3 or more days late, the score is a zero.

If you turn in a project late and don't have enough Bonus Days left to spend on it, the score is a zero.

You don't need to ask me or even tell me that you are using Bonus Days. Just turn your project in late. I have a script that will check your turn-in date and deduct the Bonus Days.

Grading

Class Rules

https://beav.es/codeofconduct

Students With Disabilities

Accommodations for students with disabilities are determined and approved by Disability Access Services (DAS). If you, as a student, believe you are eligible for accommodations but have not obtained approval please contact DAS immediately at 541-737-4098 or at http://ds.oregonstate.edu. DAS notifies students and faculty members of approved academic accommodations and coordinates implementation of those accommodations. While not required, students and faculty members are encouraged to discuss details of the implementation of individual accommodations.

Religious Holidays

Oregon State University strives to respect all religious practices. If you have religious holidays that are in conflict with any of the requirements of this class, please see me immediately so that we can make alternative arrangements.

Life Events

As {John Lennon? Allen Saunders?} has said: "Life is what happens to you while you're busy making other plans". I care about you as a person. When life happens to you, send me an email and come talk with me. I might be able to help, I might not. But I surely can listen. You are not alone.

CAPS

Oregon State University's Counseling and Psychological Services (CAPS) provides OSU students with individual, relationship, and group counseling. I have met some of these people, and they are really good. If you are in any type of emotional difficulty, don't hesitate to contact to them.

Reach Out for Success

University students encounter setbacks from time to time. If you encounter difficulties and need assistance, it's important to reach out. Consider discussing the situation with me or an academic advisor. Learn about resources that assist with wellness and academic success at http://oregonstate.edu/ReachOut. If you are in immediate crisis, please contact the Crisis Text Line by texting OREGON to 741-741 or call the National Suicide Prevention Lifeline at 1-800-273-TALK (8255)

Basic Needs

Any student who has difficulty affording groceries or accessing sufficient food to eat every day, or who lacks a safe and stable place to live, is urged to contact the Human Services Resource Center (HSRC) for support: hsrc@oregonstate.edu, 541-737-3747. The HSRC has a food pantry, a textbook lending program, and other resources to help. Furthermore, if you are comfortable doing so, please come talk with me. I will do everything I can do to help you.

University Housing & Dining Services has been piloting a meal program called Food for Thought. Using the extra food generated by campus dining locations, they create pre-packaged, balanced meal options available at no cost to Oregon State students who meet eligibility criteria determined by the Human Services Resource Center. For others, these packages are available for purchase at a reduced price.

Other Notes You Might Enjoy!

Grades 2-12
	Blender
	Processing
	Scratch
	SketchUp
	TinkerCad

Other Useful Parallel Programming Information

• OpenMP:
  • Industry-wide OpenMP information: http://www.openmp.org
  • OpenMP 4.5 Quick Reference Card: http://www.openmp.org/mp-documents/OpenMP-4.5-1115-CPP-web.pdf
  • OpenMP Quick Reference Sheet: http://www.plutospin.com/files/OpenMP_reference.pdf

• Single-Instruction-Multiple-Data (SIMD):
  • Good explanation of SSE arithmetic: http://www.tommesani.com/SSEArithmetic.html

• CUDA:
  • About CUDA: https://developer.nvidia.com/about-cuda
  • CUDA Quick Start Guide: https://docs.nvidia.com/cuda/pdf/CUDA_Quick_Start_Guide.pdf
  • CUDA C Programming Guide: https://docs.nvidia.com/cuda/pdf/CUDA_C_Programming_Guide.pdf
  • cuSolver Library: https://docs.nvidia.com/cuda/pdf/CUSOLVER_Library.pdf

• OpenCL:
  • Industry-wide OpenCL Information: http://www.khronos.org/opencl
  • OpenCL Reference (~man) Pages http://www.khronos.org/registry/cl/sdk/1.0/docs/man/xhtml
  • OpenCL 2.0 Quick Reference Card: https://www.khronos.org/files/opencl20-quick-reference-card.pdf
  • NVIDIA's OpenCL example code: http://developer.nvidia.com/opencl-sdk-code-samples
  • Prof. Bailey's OpenCL status error code printer: errorcodes.cpp
  • NVIDIA OpenCL Code Examples https://developer.nvidia.com/opencl

• Parallel Computing tutorials from Lawrence Livermore National Labs:
  • Parallel computing: https://computing.llnl.gov/tutorials/parallel_comp/

• vi Quick Reference Card
  • http://tnerual.eriogerg.free.fr/vimqrc.pdf

• Association of Computing Machinery http://www.acm.org
  • Special Interest Group on High Performance Computing http://www.sighpc.org
  • Special Interest Group on Computer Graphics http://www.siggraph.org

• C++ Style Guide used by Google: https://google.github.io/styleguide/cppguide.html

• Interesting collection of information about the history of the CPU: CPU-DB.pdf