Python Basics: input and output (I/O)

To do an NLP task

The steps are:
1. Read the file;
2. Remove all punctuation and line breaks, and change all uppercase to lowercase;
3. Merge the same words, count the frequency of each word, and rank them according to the word frequency;
4. Output the result to the file out.txt by line.
Code:
import re
import os,sys
# You don't have to worry about this function
def parse(text):
  # Using regular expressions to remove punctuation and line breaks
  text = re.sub(r'[^\w ]', '', text)

  # Turn to lowercase
  text = text.lower()
  # Generate a list of all words
  word_list = text.split(' ')
  # Remove blank words
  word_list = filter(None, word_list)

  # Generate Dictionary of words and frequency
  word_cnt = {}
  for word in word_list:
    if word not in word_cnt:
      word_cnt[word] = 0
      word_cnt[word] += 1
      print(word_cnt.items())
  # Sort by word frequency
  sorted_word_cnt = sorted(word_cnt.items(), key=lambda kv: kv[1], reverse=True)
  return sorted_word_cnt

inFile = 'in.txt'
if not os.path.exists(inFile):
  print(f'file {inFile} not exist')
  sys.exit()
with open(inFile, 'r') as fin:
  text = fin.read()

word_and_freq = parse(text)

outFile = 'out.txt'
with open(outFile, 'w') as fout:
  for word, freq in word_and_freq:
    try:
      fout.write('{} {}\n'.format(word, freq))
    except Exception as ex:
      print(f"error in wirte {outFile},error msg:{ex}")
If the file is very large, and one-time reading may cause memory crash, it can be realized by reading one line at a time:
from collections import defaultdict
import re,sys,os

inFile = 'in.txt'
if not os.path.exists(inFile):
  print(f'file {inFile} not exist')
  sys.exit()
f = open(inFile, mode="r", encoding="utf-8")
word_cnt = defaultdict(int) #defaultdict The initialization function of class takes a type as a parameter. When the accessed key does not exist, a value can be instantiated as the default value

for line in f: #Read by line
  line =re.sub(r'[^\w ]', '', line) #Using regular expressions to remove punctuation and line breaks
  for word in filter(None, line.split(' ')): #Group the words by spaces and remove the blank words
    word_cnt[word] += 1

outFile = 'out.txt'
with open(outFile,'w') as fout:
  for word, freq in sorted(word_cnt.items(), key=lambda kv: kv[1], reverse=True):
    try:
      fout.write(f'{word} {freq}\n')
    except Exception as ex:
      print(f"error in wirte {outFile},error msg:{ex}")

I/O should be careful. All I/O operations should be handled with errors to prevent coding vulnerabilities.

Json serialization and deserialization

json.dumps(), which takes Python's basic data type and serializes it into string;

json.loads(), which takes a legal string and deserializes it to Python's basic data type.

Similarly, when serializing and deserializing Json, you should also pay attention to error handling. For example, json.loads('123.2 ') will return a float type. Therefore, after deserialization, you need to determine whether you want the type:
original_params = json.loads(params_str)
t = type(original_params)
if t is not dict:
    print(f'is {t} not dict')

Examples of json.dumps() and json.loads():

import json,sys
params = {
'symbol': '123456',
'type': 'limit',
'price': 123.4,
'amount': 23
}
try:
    params_str = json.dumps(params)
except Exception as ex:
    print(f'error on dumps error msg:{ex}')
    sys.exit()

print('after json serialization') 
print('type of params_str = {}, params_str = {}'.format(type(params_str), params)) 
#after json serialization
#type of params_str = <class 'str'>, params_str = {'symbol': '123456', 'type': 'limit', 'price': 123.4, 'amount': 23}

original_params = json.loads(params_str)
t = type(original_params)
if t is not dict:
    print(f'is {t} not dict')
print('after json deserialization')
print('type of original_params = {}, original_params = {}'.format(type(original_params), original_params))
#after json deserialization 
#type of original_params = <class 'dict'>, original_params = {'symbol': '123456', 'type': 'limit', 'price': 123.4, 'amount': 23}

 

 

reference material:

Geek time "Python core technology and actual combat"

Tags: Python JSON Lambda encoding

Posted on Tue, 05 Nov 2019 15:05:30 -0500 by bubblocity