Parsing CSV Rows into Separate Text Files

The #100DaysofCode challenge is an initiative by Alexander Kallaway designed to help people learning how to code to be more consistent. The goal of this challenge was to code for at least an hour everyday for 100 days.

Going off this premise, I decided to work at least an hour every day on just research and coding for image classification. Today marks the end of the first week, in which I researched and worked on the project Analyzing Deforestation and Urbanization Using Intel AI Technologies. During this project, I have been working with Intel Optimized TensorFlow for image classification of satellite imagery. In this first week I have learned many interesting things about image classification and labeling. The one I found most interesting to share would be how to parse labels from a CSV file and transform the labels into the format that I wanted.

For this challenge I faced, let's look at the following example CSV file. This table has two columns, the first column being image name, and the second column consisting of an array of labels. This was the design of the original CSV file that I was working with.

The desired format I needed for labels was to take the image name and create a text file based on that name. For the first row, the file would be called Image_001.jpg.txt and contain all the labels for that image with each label on a separate line. Coding in Python 3, I worked to create a function that would transform the original CSV into the desired text files. As can be seen below, the file Image_001.jpg.txt would be populated with data like so:

haze 
city 
farm

This function created to make this transformation first reads in the data path and filename. Data path refers to the path in which the original CSV is stored and filename refers to the name of the CSV file. This data is stored in a dataframe that can be then traversed to gather data. Each row in the CSV is read as the variable row. Once in row, the first column row[0] is used to create the name of the new text file. Then, the second column, row[1] is split based on spaces. This list of labels, called splitrow, is then iterated over to append each label on a new line of the text file. Once completed, the file is closed and a new file can be created for the next image. This will create one label text file for each image.

def _parse_labels(self):
df_train_tags = pd.read_csv(self.data_path + self.filename)
i = 0
for index, row in df_train_tags.iterrows():
    if i > len(df_train_tags):
        print("Error: Cannot read images, file length too short...")
        break
    else:
        print("Processing: Reading image " + str(row[0]) + "...")
        f = open(self.labels_path + str(row[0])+'.jpg.txt', 'w')
        splitrow = row[1].split(" ")
        for item in splitrow: 
            f.write(item + "\n")
        f.close()
        i+=1
print("Completed: Finished reading labels dataset...")

An interesting note to make here that I didn't realize at first was why the index was required in the for loop. When thinking of a for loop, I refer to for each item in dataset then loop where each item is a row of the dataframe. This logic makes sense but what is happening here is iterrows gives a tuple value rather than just the rows. This means we have to call (index, row) for the tuple in order to access the columns.

After working this past week on research and coding, this was one of the more interesting problems I faced as it involved splitting and transforming one singular column into a text file of many lines. This function also helped give a good overview of basic dataframe and array manipulation in Python.

What interesting problems have you faced this week?

References:

Cover image sourced from Alpha Coders
Intel Optimized TensorFlow
TensorBoard
TensorFlow Guide
Intel DevMesh
Intel AI Academy
pandas.DataFrame.iterrows