Val rdd4 = ("src/main/resources/csv/text01.txt," + Println("#read multiple text files into a RDD") ![]() This method also takes the path as an argument and optionally takes a number of partitions as the second argument. SparkContext.wholeTextFiles() reads a text file into PairedRDD of type RDD with the key being the file path and value being contents of the file. #spark read text files from a directory into RDDĬlass .MapPartitionsRDDġ.2 wholeTextFiles() – Read text files into RDD of Tuple. Here, it reads every line in a "text01.txt" file as an element into RDD and prints below output. ![]() Val rddFromFile = ("src/main/resources/csv/text01.txt") Println("#spark read text files from a directory into RDD") SparkContext.textFile() method is used to read a text file from HDFS, S3 and any Hadoop supported file system, this method takes the path as an argument and optionally takes a number of partitions as the second argument.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |