disco.schemes – Default input streams for URL schemes

By default, Disco looks at an input URL and extracts its scheme in order to figure out which input stream to use.

When Disco determines the URL scheme, it tries to import the name input_stream from disco.schemes.scheme_[SCHEME], where [SCHEME] is replaced by the scheme identified. For instance, an input URL of http://discoproject.org would import and use disco.schemes.scheme_http.input_stream().

disco.schemes.scheme_dfs.input_stream(fd, size, url, params)

Opens the url path locally, relative to the path [Task.root]/input.

disco.schemes.scheme_disco.input_stream(fd, size, url, params)

Opens the path on host using an http client and the setting DISCO_PORT.

disco.schemes.scheme_file.input_stream(fd, size, url, params)

Opens the url locally on the node.

disco.schemes.scheme_http.input_stream(fd, sze, url, params)

Opens the specified url using an http client.

disco.schemes.scheme_raw.input_stream(fd, sze, url, params)

Opens a StringIO whose data is everything after the url scheme.

For example, raw://hello_world would return hello_world when read by the task.