Multiprocessing

  1. Write a script that uses the multiprocessing module from the Python standard library to apply a function to each number in a large list of numbers and then print out the result. (Hint: think map.)

  2. Install the IPython parallel processing support. Then use its parallel version of map to perform the same task as the previous exercise, above. Note that you will have to start up an IPython cluster.

  3. Install asynctools. Use it to write a Python function in the "async style" (think JavaScript/Node.js). For example: (in a second thread) open a file and read it with file.readlines() and print out each line. Meanwhile in the main (calling ?) thread, print a few lines then wait for the second thread to finish and then print a few more lines.

    If you prefer, use the threading module from the Python standard library instead of asynctools. The asynctools library simply provides some convenience wrappers around functionality from the threading module.

    For information on asynctools, see:

    Question: Effectively, your solution is creating and reading the file in a separate thread. How could you return a value (e.g. the number of characters in the file) from that function in that separate thread back to the main thread. For a hint or suggestion, look in the multiprocessing module in the Python standard library.

    Also see: Exercises_python/bonus_exercises.html

  4. Install asynctools. Use it to retrieve and report the length of each page in a list of Web pages. Do this both serially and in parallel. Here is a list of URLs that you can use (or use your own):

    Urls = [
        'http://www.python.org',
        'http://www.davekuhlman.org',
        'https://www.ruby-lang.org/en/',
        'http://www.ruby-doc.org/',
        'https://www.raspberrypi.org/',
        'http://www.ubuntu.com/',
        'https://nodejs.org/en/',
        'https://nodejs.org/en/docs/',
    ]
    
  5. Install pyzmq. Use it to write a load balancing application. Implement (1) a worker, (2) a broker, and (3) a client.

    • The worker receives a request and returns a response. Possible tasks are (1) retrieve a Web page and return its length or (2) parse an XML document and return the number of elements.
    • The broker forwards request from the client to the workers and balances requests across workers.
    • The client sends requests to the broker, for example, sending a URL for a worker that retrieves a Web page or sending a file name for a worker that parses an XML file.

References -- Additional information on concurrency and parallelism and multiprocessing in Python can be found here: