Python 中文件比较和合并的几种有效策略

在日常编程或数据分析任务中，处理比较和合并多个文件是很常见的。Python 具有强大的文件处理能力和广泛的库支持，是处理此类任务的理想选择。

下面，我们将探讨几种有效的文件比较和合并策略，每种策略都附有详细的代码示例和解释。

1. 基本文件读写

首先，了解如何读取和写入文件是基础。

# Open and read content from the input file
with open('input_file.txt', 'r') as input_file:  
    data = input_file.readlines()  # Read all lines from the input file

# Open the output file and write the content into it
with open('output_file.txt', 'w') as output_file:  
    for line in data:  
        output_file.write(line)  # Write each line to the output file

2. 文件内容比较

使用 difflib 库来比较两个文件之间的差异。

# Import the difflib module for file comparison
import difflib  

# Open and read the first input file
with open('input_file1.txt', 'r') as input_file1, open('input_file2.txt', 'r') as input_file2:
    
    # Compare the content of the two files using unified_diff
    diff = difflib.unified_diff(input_file1.readlines(), input_file2.readlines())
    
    # Print the differences line by line
    print('\n'.join(diff))

3. 合并 CSV 文件

对于 CSV 文件，pandas 库可用于合并操作。

# Import pandas library for data manipulation
import pandas as pd  

# Read the first CSV file into a DataFrame
df1 = pd.read_csv('data_file1.csv')  

# Read the second CSV file into a DataFrame
df2 = pd.read_csv('data_file2.csv')  

# Merge the two DataFrames by concatenating them, assuming matching column names
merged_df = pd.concat([df1, df2], ignore_index=True)  

# Save the merged DataFrame to a new CSV file
merged_df.to_csv('output_merged.csv', index=False)

4. 逐列 CSV 合并

合并特定列，例如基于公共键联接文件。

# Import pandas library for data manipulation
import pandas as pd  

# Read the first CSV file into a DataFrame
df1 = pd.read_csv('data_file1.csv')  

# Read the second CSV file into a DataFrame
df2 = pd.read_csv('data_file2.csv')

# Merge the two DataFrames based on a common column named 'common_key'
# 'how="outer"' ensures that all rows from both DataFrames are included, 
# with missing values filled as NaN where data does not match
merged_df = pd.merge(df1, df2, on='common_key', how='outer')  

# Save the merged DataFrame to a new CSV file
merged_df.to_csv('output_merged_by_key.csv', index=False)  

5. 基于行的合并

当基于相似行结构合并文件时，直接迭代和追加行。

# Initialize an empty list to store the content from all input files
data = []  

# List of input text files to be read and merged
for filename in ['input_file1.txt', 'input_file2.txt']:  
    # Open each file in read mode
    with open(filename, 'r') as file:  
        # Read all lines from the current file and add them to the data list
        data.extend(file.readlines())  

# Open the output file in write mode
with open('output_merged_file.txt', 'w') as merged_file:  
    # Write each line from the data list into the output file
    for line in data:  
        merged_file.write(line)

6. 去重合并

使用 sets 在合并之前删除重复的行。

# Initialize a set to store unique lines from all input files
unique_lines = set()  

# List of input text files to be read and merged
for filename in ['input_file1.txt', 'input_file2.txt']:  
    # Open each file in read mode
    with open(filename, 'r') as file:  
        # Add all lines from the current file to the set (duplicates are automatically removed)
        unique_lines.update(file.readlines())  

# Open the output file in write mode
with open('output_merged_unique.txt', 'w') as merged_file:  
    # Sort the unique lines to ensure consistent output order
    for line in sorted(unique_lines):  
        # Write each unique line into the output file
        merged_file.write(line)

7. 文本文件的二进制比较

使用 filecmp 模块比较文件的二进制内容。

# Import the filecmp module for file comparison
import filecmp  

# Compare the binary contents of 'input_file1.txt' and 'input_file2.txt'
if filecmp.cmp('input_file1.txt', 'input_file2.txt'):  
    print("Files are identical.")  # Output message if files are identical
else:
    print("Files differ.")  # Output message if files differ

8. 大文件高效比对

对于大型文件，请逐行读取和比较它们以节省内存。

# Open the first large file ('input_large_file1.txt') and second large file ('input_large_file2.txt') for reading
with open('input_large_file1.txt', 'r') as f1, open('input_large_file2.txt', 'r') as f2:  
    
# Read lines from both files simultaneously and compare them
    for line1, line2 in zip(f1, f2):  
        # If a difference is found between the two lines, print a message and stop the comparison
        if line1 != line2:  
            print("Difference found!")  
            break  # Exit the loop as the first difference has been found

9. 多个文件的动态合并

使用循环动态合并文件路径列表中的文件。

# Generate a list of file paths for input files ('input_file1.txt' to 'input_file3.txt')
file_paths = ['input_file{}.txt'.format(i) for i in range(1, 4)]  

# Open the output file ('output_merged_all.txt') in write mode
with open('output_merged_all.txt', 'w') as merged:  
    # Iterate through the list of input file paths
    for path in file_paths:  
        # Open each file in read mode
        with open(path, 'r') as file:  
            # Write the content of the current file to the merged output file
            # Add a newline character to separate the content of different files
            merged.write(file.read() + '\n')

10. 高级合并策略：智能合并

对于更复杂的合并标准，例如按日期或 ID 合并，请在合并之前对数据进行排序。

# Import pandas library for data manipulation
import pandas as pd  

# Read CSV files ('input_file1.csv' and 'input_file2.csv') into DataFrames
dfs = [pd.read_csv(f) for f in ['input_file1.csv', 'input_file2.csv']]  

# Concatenate the DataFrames and sort by the 'date_column', assuming it's the column holding the date data
sorted_df = pd.concat(dfs).sort_values(by='date_column')  

# Save the merged and sorted DataFrame to a new CSV file
# Import pandas library for data manipulation
sorted_df.to_csv('output_smart_merged.csv', index=False)