Read Range Of Files In Pyspark

I need to read contiguous files in pySpark. The following works for me. from pyspark.sql import SQLContext file = 'events.parquet/exportDay=2015090[1-7]' df = sqlContext.read.

Solution 1:

Use curly braces.

file = "events.parquet/exportDay=201509{08,09,10,11,12,13,14}"

Here's a similar question on stack overflow: Pyspark select subset of files using regex glob. They suggest either using curly braces, OR performing multiple reads and then unioning the objects (whether they are RDDs or data frames or whatever, there should be some way).

Solution 2:

It uses shell globbing, I believe.

The post: How to read multiple text files into a single RDD?

Seems to suggest the below should work.

"events.parquet/exportDay=2015090[89],events.parquet/exportDay=2015091[0-4]"