CS 475 / 575 -- Parallel Programming

Spring Quarter 2017

http://cs.oregonstate.edu/~mjb/cs575


Lectures Projects Handouts Grades VHR


This page was last updated: March 14, 2017



Under Construction:
This content is not final until you see this message go away!


Announcements:


Here are Some Background Resources for You


What We Will Be Doing This Quarter

The goals of this course are to leave you "career-ready" (i.e., both work-ready and research-ready) for tasks that require desktop parallelism, both on a CPU and on a GPU.

CS 475/575 topics include:

Prerequisites

This course will use C/C++ for its programming. Already being comfortable with function calls, arrays, for loops, structures, arrays of structures, structures of arrays, pointers, stacks, queues, trees, and linked lists is important. It is strongly suggested that you not use this class as an opportunity to learn C for the first time.

Many of the assignments can be done on Linux systems. It would be good if you already know how to use Linux and know at least one Linux-based editor (vim is good.

Beyond that, understanding the analysis of algorithms and Big-O notation is highly recommended.

Some knowledge about computer architecture (e.g., cores, cache) would be a plus, but not critical.

Learning Outcomes

On completion of the course, students will have demonstrated the ability to:

  1. Explain the clockspeed limitations of computing using physics and Moore's Law
  2. Explain the limitations of parallel computing using Amdahl's Law
  3. Demonstrate "parallel thinking" in program design
  4. Explain the difference between TLP, DLP, and SIMD
  5. Demonstrate the ability to program parallel algorithms in TLP, DLP, and SIMD.
  6. Characterize what types of problems are best able to be parallelized
  7. Characterize different parallel programming patterns and what types of problems they best address
  8. Chacacterize how cache issues affect parallel performance
  9. Demonstrate the proper use of synchronization to avoid race conditions and deadlock
  10. Characterize the benefits of using a CPU versus using a GPU for parallel programming
  11. Characterize the benefits of using a GPU versus using a CPU for parallel programming

In addition, those taking this course as CS 575 will also have deminstrated the ability to:

  1. Read a parallel-programming-related research paper and write a 5-page analysis paper of it. (I will make some of these available for you, or you can propose your own. It has to be a real research paper, though.)

Professor

The class is being taught by Professor Mike Bailey.

Office: Kelley 2117
Phone: 541-737-2542
E-mail: mjb@cs.oregonstate.edu

Prof. Bailey Office Hours:

TBD.    
    or, anytime my office door is open
    or, by appointment -- send email

Note that anytime my office door is open, it becomes an implied Office Hour time. However, this also means that when my office door is not open, I have made a deliberate choice not to make this an impled Office Hour time. Please respect this.

TA

??????'s Office Hours:

TBD.    
    or, by appointment -- send email

The Virtual Hand Raise (VHR)

I recognize that it sometimes takes a certain amount of courage to ask a question in class. But, the worst thing of all is to not ask! So, this class also uses something called the Virtual Hand Raise. Click here to get into it. It will allow you to send me a question or comment, completely anonymously. I will answer questions submitted this way by email to the class or in class.

Textbook

There is no required textbook for this course. We will use notes that will be made available on the class web site.

There is a free Parallel Programming book available. Click here to get to it. I'm not sure how much it helps for this class, but it is certainly interesting to look through.

Lecture Schedule

To see an Academic Year calendar, click here.

Class lecture time is: Monday, Wednesday, and Friday, 1:00 - 1:50. Unless otherwise specified, all classes will be in Kidder 364.

  Date Topics
1 April 3 Introduction. Syllabus. What this course is ... and isn't.
Project notes: timing, graphing. (This is really important.)
2 April 5 Simple OpenMP. Project #0.
Parallel programming background information. The three things we care about Parallel Programming for. Von Neumann architecture.
3 April 7 Multithreading
OpenMP: fork-join model, pragmas, what it does for you, what it doesn't do for you,
OpenMP: parallelizing for-loops
OpenMP: variable sharing
4 April 10 OpenMP: Reduction. Not doing anything special vs. critical vs. atomic vs. reduction.
Trapezoid integration.
Project #1.
5 April 12 Moore's Law. What holds, what doesn't.
Multicore. Hyperthreading.
Timing. Speedup. Amdahl's Law. Parallel efficiency. Gustafson's observation.
6 April 14 OpenMP: dynamic vs. static thread assignment.
Chunksize.
Project #2.
Mutexes.
Barriers.
Sections, tasks.
7 April 17 OpenMP: bubble sort (red-black), tree traversal using tasks.
8 April 19 Caches. Architecture. Hits. Misses.
9 April 21 Caches. False sharing.
Project #3.
10 April 24 Designing parallel programs.
11 April 26 Designing parallel programs.
12 April 28 Functional Decomposition
Project #4
13 May 1 Project #4.
Test #1 review
Vectorization: what it is, how to use it.
14 May 3 More vectorization
Prefetching
Xeon Phi
Project #5
15 May 5 Test #1
16 May 8 Go over the test answers.
GPU 101.
Architecture.
17 May 10 GPUs -- what they are good at.
GPUs -- what they are not good at.
Why?
18 May 12 Prof. Bailey out of town -- No Class Today
Remember that Sunday is Mother's Day!
19 May 15 OpenCL: What is it? Diagram. Mapping onto GPU architecture.
OpenCL library. Querying configurations.
20 May 17 OpenCl Events
21 May 19 Engineering Expo -- No Class Today
22 May 21 OpenCL Reduction.
Project #6.
23 May 24 OpenCL / OpenGL Interoperability
24 May 26 More OpenCL / OpenGL Interoperability
Project #7.
25 May 29 Memorial Day -- OSU Holiday -- No Class Today
26 May 31 Looking at OpenCL Assembly Language.
OpenGL Compute Shaders
27 June 2 Prof. Bailey out of the office -- no class today.
Guest Speaker: Chris Schultz, NVIDIA: "GPU Architectures" Not confirmed.
28 June 5
29 June 7 Class Evaluations.
Compute Shaders.
Test #2 review.
More Information.
30 June 9
T2 June 8 Test #2 Wednesday, June 8, 9:30 - 11:00 AM.
(You can confirm this for yourself by going here.
G June 16, 3:30 EECS Graduation Celebration
G June 17 OSU Graduation

Projects

Project # Points Title Due Date
0 30 Simple OpenMP Experiment April 10
1 100 OpenMP: Numeric integration April 25
2 100 OpenMP: N-body problem May 1
3 70 False sharing. May 9
4 100 Functional Decomposition May 16
5 60 Vectorized Array Multiplication May 23
6 120 OpenCL Array Multiplication and Reduction June 2
7 120 OpenCL/OpenGL Particle System June 13
575-only paper 100 Paper analysis June 13

Project Notes

Refer back to the Project Notes handout.

Grading

Grades will be posted through this web page. To protect your privacy, they will be posted by your alias that you give me in Project #1.

Click here to see the current grade posting.

CS 475/575 will be graded on a fill-the-bucket basis. There will be 8 projects, 10 quizzes, and two tests. You get to keep all the points you earn.

In addition, the CS 575 people have an extra 100-point project.
The CS 475 people automatically get 100 points for this project, even though they didn't have to do it.

Your final grade will be based on your overall class point total. Based on an available point total of 1100, grade cutoffs will be no higher than:

Points Grade
1070
1050 A- 
1030 B+
1010
990 B- 
970 C+
950
930 C- 
910 D+
890
870 D- 


Project Turn-In Procedures

Bonus Days and Late Assignments

Projects are due at 23:59:59 on the listed due date, with the following exception:

Each of you has been granted five Bonus Days, which are no-questions-asked one-day extensions which may be applied to any project, subject to the following rules:

  1. No more than 2 Bonus Days may be applied to any one project
  2. Bonus Days cannot be applied to tests
  3. Bonus Days cannot be applied such that they extend a project due date past the start of Test #2.

Click here to get a copy of the Bonus Day Submission Form. Fill this out and turn it in the next class period after turning in your project.

Downloadable Files

None yet.

Handouts

Don't print these until you are told to do so in the Announcements section.
Sometimes I will put notes out here that are not quite complete, just to show you where we are headed.

  1. Project Notes 1pp, 2pp, 6pp
  2. Simple OpenMP 1pp, 2pp, 6pp
  3. Parallel programming: background Information 1pp, 2pp, 6pp
  4. Moore's Law and Multicore: 1pp, 2pp, 6pp
  5. OpenMP 1pp, 2pp, 6pp
  6. OpenMP 4.0 Quick Reference Card: http://openmp.org/mp-documents/OpenMP-4.0-C.pdf
  7. Trapezoid Intergration with OpenMP 1pp, 2pp, 6pp
  8. Speedups and Amdahl's Law 1pp, 2pp, 6pp
  9. rabbit.engr.oregonstate.edu 1pp, 2pp, 6pp
  10. Bubble Sort Case Study 1pp, 2pp, 6pp
  11. Tree Traversal using Tasks 1pp, 2pp, 6pp
  12. Caching Issues in Multicore Performance 1pp, 2pp, 6pp
  13. Mallocing on a cache line 1pp, 2pp, 6pp
  14. How Do Barriers Work When They Come From Different Parts of the Code? Click here
  15. Prefetching 1pp, 2pp, 6pp
  16. Parallel Program Design Patterns and Strategies 1pp, 2pp, 6pp
  17. Vector Parallel Programming (SIMD) 1pp, 2pp, 6pp
  18. Xeon Phi 1pp, 2pp, 6pp
  19. GPU 101 1pp, 2pp, 6pp
  20. OpenCL 1pp, 2pp, 6pp
  21. first.cpp C++ source
  22. first.cl OpenCL source
  23. OpenCL 1.1 Quick Reference Card: PDF.
  24. OpenCL 1.2 Quick Reference Card: PDF.
  25. OpenCL 2.0 Quick Reference Card: PDF.
  26. OpenCL Events 1pp, 2pp, 6pp
  27. OpenCL Reduction 1pp, 2pp, 6pp
  28. OpenCL / OpenGL Vertex Buffer Interoperability 1pp, 2pp, 6pp
  29. Looking At OpenCL Assembly language 1pp, 2pp, 6pp
  30. OpenCL / OpenGL Image Interoperability 1pp, 2pp, 6pp
  31. Compute Shaders 1pp, 2pp, 6pp
  32. Finding More Information on Parallel Programming PDF

Lecture Recordings

I am simultaneously teaching this class on Ecampus. I will put links to those classes here so that you can relive them if that would help. Warning: We often talk about other things in class, and those are eligible to be on the test, even if they didn't happen in the Ecampus recordings!

None yet.

Class Rules

Students With Disabilities

Accommodations for students with disabilities are determined and approved by Disability Access Services (DAS). If you, as a student, believe you are eligible for accommodations, but have not obtained approval, please contact DAS immediately at 541-737-4098 or at http://ds.oregonstate.edu. DAS notifies students and faculty members of approved academic accommodations and coordinates implementation of those accommodations. While not required, students and faculty members are encouraged to discuss details of the implementation of individual accommodations.

Other Useful Online Parallel Programming Information