{"id":127598,"date":"2022-08-27T08:22:58","date_gmt":"2022-08-27T08:22:58","guid":{"rendered":"https:\/\/blog.finxter.com\/?p=618493"},"modified":"2022-08-27T08:22:58","modified_gmt":"2022-08-27T08:22:58","slug":"how-to-convert-a-log-to-a-csv-file-in-python","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2022\/08\/27\/how-to-convert-a-log-to-a-csv-file-in-python\/","title":{"rendered":"How to Convert a Log to a CSV File in Python?"},"content":{"rendered":"\n
\n
\n
\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n<\/p><\/div>\n
\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n
\n
<\/div>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n
5\/5 – (1 vote) <\/div>\n<\/div>\n

A not-so-fictious problem<\/strong>: Say, you’ve created a web application that runs on a dedicated Linux server in the cloud<\/a>. Thousands of users visit your web app<\/a> and suddenly … it crashes. Your users start complaining, and you lose revenue. More importantly, you bleed credibility by the hour. Your server is down, so what do you do?<\/em> \"\ud83e\udd2f\"<\/p>\n

First<\/strong>, don’t panic. \"\ud83d\udef8\"<\/p>\n

Let’s analyze your server logs!<\/p>\n

This article shows you how to convert your log file to a CSV file<\/strong> in Python, that you can use for further processing (e.g., in Pandas or Excel). <\/p>\n

Problem Formulation by Example<\/h2>\n

Given a file my_file.log<\/code> like this one I pulled from a real IBM server log<\/a> example:<\/p>\n

03\/22 08:51:01 INFO :.main: *************** RSVP Agent started ***************\n03\/22 08:51:01 INFO :...locate_configFile: Specified configuration file: \/u\/user10\/rsvpd1.conf\n03\/22 08:51:01 INFO :.main: Using log level 511\n03\/22 08:51:01 INFO :..settcpimage: Get TCP images rc - EDC8112I Operation not supported on socket.\n03\/22 08:51:01 INFO :..settcpimage: Associate with TCP\/IP image name = TCPCS<\/pre>\n
How to convert this log file to a CSV file of the following standard comma-separated values format:<\/p>\n
03\/22,08:51:01,INFO,:.main: *************** RSVP Agent started ***************\n03\/22,08:51:01,INFO,:...locate_configFile: Specified configuration file: \/u\/user10\/rsvpd1.conf\n03\/22,08:51:01,INFO,:.main: Using log level 511\n03\/22,08:51:01,INFO,:..settcpimage: Get TCP images rc - EDC8112I Operation not supported on socket.\n03\/22,08:51:01,INFO,:..settcpimage: Associate with TCP\/IP image name = TCPCS\n<\/pre>\n
Or, here’s how that would look if you opened it with Excel:<\/p>\n
\n
\"\"<\/figure>\n<\/div>\n
Prettier, isn’t it? Unlike the first representation (log file), this CSV representation is easier to read for (most) human beings. \"\ud83e\udd16\"<\/p>\n
Convert Server Log to CSV with Pandas<\/h2>\n
You can convert a .log<\/code> file to a CSV file in Python in four simple steps: (1) Install the Pandas <\/a>library, (2) import the Pandas library, (3) read the log file<\/a> as DataFrame, and (4) write<\/a> the DataFrame to the CSV file<\/strong>.<\/p>\n
    \n
  1. (Optional in shell) pip install pandas<\/code><\/li>\n
  2. import pandas as pd<\/code><\/li>\n
  3. df = pd.read_csv('my_file.log', sep='\\s\\s+', engine='python')<\/code><\/li>\n
  4. df.to_csv('my_file.csv', index=None)<\/code><\/li>\n<\/ol>\n
    Here’s a minimal example:<\/p>\n
    import pandas as pd\ndf = pd.read_csv('my_file.log', sep='\\s\\s+', engine='python')\ndf.to_csv('my_file.csv', index=None)<\/pre>\n
    \"\u2139\" Note<\/strong>: The regular expression<\/a> sep='\\s\\s+'<\/code> specifies more than one single whitespace<\/a> as a separator between two CSV values. If you have a different separator string, you can define it here. <\/p>\n
    You specify the engine='python'<\/code> to tell Pandas that we want the Python regular expression engine to process the separator regular expression.<\/p>\n
    The result of the code is the following CSV file:<\/p>\n
    \n
    \"\"<\/figure>\n<\/div>\n
    You can use this CSV file as input for, say, an Excel sheet or Google Spreadsheet for further processing and analysis.<\/p>\n
    This is what your log file looks converted to a CSV and imported to Excel:<\/p>\n
    \"\"<\/figure>\n
    And this is how your log file looks as a Pandas DataFrame<\/a>:<\/p>\n
     03\/22 ... :.main: *************** RSVP Agent started ***************\n0 03\/22 ... :...locate_configFile: Specified configuration... 1 03\/22 ... :.main: Using log level 511 2 03\/22 ... :..settcpimage: Get TCP images rc - EDC8112I O... 3 03\/22 ... :..settcpimage: Associate with TCP\/IP image na... [4 rows x 4 columns]<\/pre>\n
    \"\ud83c\udf0d\" Related Tutorial<\/strong>: Python Pandas DataFrame to_csv()<\/code><\/a><\/p><\/p>\n","protected":false},"excerpt":{"rendered":"
    5\/5 – (1 vote) A not-so-fictious problem: Say, you’ve created a web application that runs on a dedicated Linux server in the cloud. Thousands of users visit your web app and suddenly … it crashes. Your users start complaining, and you lose revenue. More importantly, you bleed credibility by the hour. Your server is down, […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[857],"tags":[73,468,528],"class_list":["post-127598","post","type-post","status-publish","format-standard","hentry","category-python-tut","tag-programming","tag-python","tag-tutorial"],"_links":{"self":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/127598","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/comments?post=127598"}],"version-history":[{"count":0,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/127598\/revisions"}],"wp:attachment":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media?parent=127598"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/categories?post=127598"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/tags?post=127598"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}