<!– –>

<!–

Calendar

View All     Submit Event

---

–>

If you enjoy reading this site, you might also want to check out these UBM Tech sites:

---

---

  

---

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

---

During the ongoing development of Torchlight III, we want players to get up-to-the-moment information about what’s going on with the game & dev team. This month’s Developer Update comes from Guy Somberg, Lead Programmer of Echtra, with help from Jill Sullivan, Senior Community Manager.

Introduction

In game development, not every problem that you solve is performance, features, or functionality that the players get to see.  Sometimes, you have something messy that needs to be fixed, and you just need to dive in and fix it so that work can get done.

There is a phrase that we use to talk about this sort of work: “yak shaving”.  Originally from a reference to the TV program “Ren and Stimpy”, it now refers to work that appears to be completely unrelated to the end goal, but which you have to accomplish in order to reach it.  For example – “I am trying to build a stone bridge over this creek.  I am shaving this yak so that I can trade the fur to a yarn maker, who in exchange will let me borrow the cart so that I can take it to the quarry to pick up some stones.”  Shaving the yak isn’t necessarily an important part of building that bridge, but you won’t be able to make any progress while the yak still has its coat!

This is a collection of just some of the times that we’ve had yak-shaving problems that needed to be “just fixed”.

0D0D0A

Source control systems are one of the fundamental tools that game developers (and, in fact, pretty much all developers) use.  It is a database containing the entire history of every file that makes up our game – source code, assets, sounds, you name it.

About three years ago, we switched our source control from one system to another.  It doesn’t matter from what to what.  The system we were using was breaking down under our load, so we needed a new one.  We did our homework, examined the alternatives, and made a call.

Now, when you transition source control systems, there are broadly speaking two ways to go about it.  The simpler way is to lock everybody out of source control, take a snapshot of the latest stuff, import it into the new system, tweak it to conform to the new system’s idea of how the universe should work, and then turn the new system on for people.  This has the advantage that it “just works”, but it loses all of the source history from before the changeover.  In these environments there is often a single moment in source history that says “Imported everything.  If you want history before this, go look in the other source control system.”  That’s fine, so long as the other system stays around or you otherwise have access to it, but often the commits from earlier are lost forever.

The more complex way of doing this is to actually import the history from the old system into the new one.  Most source control systems allow you to do this, but it is time-consuming, error prone, and still requires some manual intervention to conform to the idiosyncrasies of the new system.  Although it is more work up-front, it is invaluable in the end to have your entire source history available.

We opted for the history import, which – at least on the surface – seemed to go just fine.  We saw the history, we saw the files, and we were able to poke around and verify that everything looked right.  Some files had weird spacing issues, which didn’t seem like that big a deal.

But then we tried to compile, and it all crumbled down.  The Visual Studio compiler complained about “Mac line endings” and refused to compile anything.

What?!  Why would that be?

A bit of background here: when a computer wants to represent a character, it has to select an encoding.  The most common encoding in use today is called UTF-8, which can encode all English-language characters, common punctuation, or a number of control codes into a single byte of data.  (Using multiple bytes, you can encode data in just about any language, but that’s another discussion.)

Two of these control codes are the Carriage Return (CR) and the Line Feed (LF) characters, which hearken back to the days where computers were hooked up to automated typewriters rather than screens.  In those days, you would tell the printer carriage to go back to its home column by sending it a CR code, and you would have the paper roll to the next line by sending it an LF code.  Thus, if you wanted to start typing at the beginning of a new line, you would send the sequence CR LF.

When the switch to fancy graphical displays occurred, this CR LF convention remained for backward compatibility.  However, developers of new systems – like the sparkly new Apple Macintosh computer and the Unix system at Bell Labs – weren’t hindered by backward compatibility and were free to make different choices.

It turns out that the three most common systems in the world today all made different choices: DOS used CR LF, Macintosh used CR, and Unix used LF.  Windows inherited its line endings from DOS, and MacOS now uses LF (the same as Unix).

Over time, the differences sorted themselves out.  Software is generally able to operate in “text mode” and provide the user with whatever line endings they need for their system to render it correctly.  The details of these differences leak through every so often, but usually aren’t a big deal.

All of this background flashed through our minds when we saw the error about Mac line endings.  What was it talking about?  We develop on Windows, so all of the line endings should have been Windows (CR LF) line endings – or, at the very least a combination of Windows and Unix (LF) line endings.  Where were these lone CR characters coming from?

And then we remembered the weird spacing issues – all of our source code appeared to be double-spaced.  Where did those extra spaces come from?

This is where somebody had the idea to look at the file in a hex editor – a tool which allows us to see the binary representation of the text files by displaying each byte’s value in hexadecimal.  Ordinarily, on a file with Windows line endings, you expect to see a line of text, then a CR (13, or 0D in hexadecimal representation) and an LF (10, or 0A).  For some reason, on the broken lines, we saw a CR (0D), then another CR (0D), and then an LF (0A), giving us 0D0D0A.

Somehow, during the conversion process from one source control program to another, the conversion program decided that the file had Unix line endings, then went through and did a blind search/replace every LF with CR LF, even if it already had a CR!  That explained everything.  Our editor was perfectly happy to render the CR as a blank line, and knew how to convert the CRLF into a blank line, which was why our code appeared to be double-spaced.  Contrariwise, the Visual Studio compiler was happy to interpret the CR LF combo as a newline, but errored out on the preceding CR.

I fixed this by writing a little program in C++.  It would iterate over our source code directory, open every text file, find patterns of 0D0D0A and replace them with 0D0A.  We don’t expect to change source control systems again, so the code for this tool is lost to the sands of time.  (Ed. – Or, so we thought!  A drive containing the source for this program was discovered after this article was written, so we have uploaded the code to our repository for posterity.)

There were only two or three of us who worked on this particular issue, but you can get any of us to twitch a little just by saying “oh doh doa”.

Octothorpe Fixer

A couple of years ago our sound designer and our composer took a trip out to Bratislava, Slovakia to record a live orchestra for some of our music.  It was an awesome trip (so I’m told), and they got a lot done over the few days that they were there.

One of the outputs of this trip was a suite of content that we call “vzory” – Slovakian for “pattern”.  These are small orchestral chunks of music that can be combined in myriad ways to create new music, and are recorded in various combinations of keys and notes.  The end result is that we have a particular pattern in G, in G#, in F, in F#, etc.

Our composer did the natural thing – he spent a bunch of time cutting and organizing all of this content, doled it out into folders and files matching the note that they were recorded at, and then imported the whole suite into the audio tool that we use, FMOD Studio.  FMOD is hooked up to our source control system, and it happily added all of the new files and then checked them in.

So far so good.  Until people started to get mysterious warnings about filenames when they got the latest code and data through our source control system.  They were just warnings – they weren’t preventing anybody from working – but it was definitely something that we didn’t want to stick around.

These vzory tracks were tracked down as the culprit.  It turns out that our source control system doesn’t like it if you check in files with an octothorpe (‘#’, sometimes called a pound sign, hash mark, hash tag, number sign, or various other things) in the filename.  It will accept them, but complain loudly.  It turns out that our composer named the directory and matching files for the vzory tracks in the key of A sharp with the name “A#” – naturally!  (The other sharp keys were set up this way as well.)

The source control system was most displeased with this choice.

Renaming the files wasn’t enough, because FMOD keeps track of file and directory metadata in XML files – each one with a GUID (a sequence of letters, numbers, and dashes) as the filename.

Once again, code to the rescue.  This time it was a program written in C# (ironically) that would iterate over all of the files and subdirectories in the given path, find ones with an octothorpe in the name, and rename them to replace the ‘#’ with the word ‘sharp’.  So, ‘A#’ became ‘Asharp’.  Then it would iterate over the XML files in the path, find any that had an octothorpe in the file contents (which were therefore metadata about the files or directories that had been renamed), and replace the ‘#’ in that line with the word ‘sharp’.

Other than telling our folks “don’t do that”, there’s not much that we had to do to prevent this from happening again.  This time we kept the source code, so if we make that particular mistake again then the tool to fix it is ready at hand.

POFixer

Localization and internationalization are important parts of any game project.  We use the Unreal engine, which has a suite of localization tools built-in.  By using a particular data structure in our data files, Unreal can find all of the localized lines in the game.  We can then export them into a standardized format called a “portable object” (.po) file, used by the GNU gettext tools, among others.

This is a format that our translators have tools to deal with.  They grab the files, translate the lines, and send them back.  We can then import them to a particular locale and then Unreal will render the text.  All very neat, so long as you color within the lines and follow the way that Unreal expects you to work.

Naturally, we have built some of our own stuff which lives inside of Unreal’s systems and plays nicely with them, but is sufficiently “off to the side” that it is invisible to some of Unreal’s other systems.  One of those parts is the localized string system, which didn’t see any of our fancy assets.

We wrote a tool that makes them visible, and called it a day.  Our first big batch of localization went out to the translators.  We went to import it…only to find that none of our strings got imported!

What happened?

Unreal allows you to identify each localized string by a pair of text strings: a category and an entry within that category.  If you don’t provide either of those entries, it will generate them for you.  It turns out that the tool that we had written to make our assets visible to the translators generated a new category and entry for every localized text string every time it was run, which meant that every text line would get a different code every time we ran the import or the export.

Oh, dear.  We fixed the underlying problem and made the category/entry pairs be consistent across runs, but we had this massive drop of strings in all of the languages that was incompatible with the fixed-up data!  We had to figure out how to run a one-time fixup on these strings to make them match.

Fortunately, each string came with a lot of metadata about its context and provenance.  Much of this metadata did not change, or at least changed in a predictable fashion.  This metadata turned out to be enough that we could compare an imported line and a newly-exported line and match up the strings.

As before, writing some code was the answer here.  We wrote a program (C++ again) to read in the translated file (containing the old, incorrect category/entry pairs) and a newly-exported English-language file (containing the new, correct category/entry pairs), match up the metadata, and then write out a fixed version of the file containing the translated text with the corrected category/entry pairs.

Here is one situation where simply fixing up the data was insufficient.  We needed to solve the underlying problem first before we could write the tool to fix the data.

Conclusion

These problems all had a common theme: through some sequence of events – human error or machine error – a bunch of important files appeared that were all broken in some way.  Ultimately, to the people who are working with the data it doesn’t really matter why any of these things happened.  They just want to take their broken files and fix them.  It is always a worthwhile endeavour to figure out a root cause and prevent an issue from occurring again, but sometimes you just need to get to work.  All of the postmortem analysis and preventative work in the world won’t help people get their jobs done with the broken files that they already have.

The examples I have talked about here were all important work that needed to get done, but all three of those programs got run exactly once.  It turns out that many of the systems that we build are complex, and these sorts of issues crop up as a normal part or development as we discover some of the edge cases.

Sometimes, you need to write tools that you run exactly once, and that is not a problem – you just need to grit your teeth and shave that yak.

– Guy Somberg

---

Related Jobs

Gameloft Australia Pty Ltd — Brisbane, Queensland, Australia
[12.22.20]
UI Developer

---

<!–

Extra Div –>

Wondering how to get Coin Master free spins? You’ve come to the right place. This is an addictive mobile game by design. It combines the thrill of playing slots with the social battling of Clash of Clans to create something that you just can’t put down; in a good way. The problem is, you so often have to put it down if you’re not willing to fork out the cash for regular spins. That’s understandable when as little as 30 spins will set you back £1.99 in the UK or $1.99 in the US.

Fortunately for you though, there are a wide number of means of getting Coin Master free spins, reducing the need for you to spend and increasing the speed at which you can progress throughout this addictive experience. Many of them are easy to pull off too, so you don’t have to worry about going through complicated maneuvres to carry on playing your favourite game.

In this guide, we’re going to provide you with all of the ways you can get your hands on a few free spins here and there. This will allow you to continue playing long after your daily free spins run out, and provide you with the means of getting extra, without the need to spend your hard-earned money on premium spins in the game’s store. We’d also recommend checking out our Coin Master free cards and Coin Master free coins guides to get even more rewards.

Coin master free spins – daily links:

Below are daily links that you can follow to get a bunch of free spins in Coin Master. We update these every day with new links so it’s worth bookmarking this page and checking back each day to get even more.

It’s worth noting that the links expire after three days, so those from the two prior are still good to go.

December 26

December 25

December 24

How can I get free spins in Coin Master?

Here are a bunch of tips to help you get even more free spins in Coin Master.

Follow coin master on social media

Each day, Moon Active, Coin Master’s developer, provides a bunch of links that you can follow to get your hands on Coin Master free spins. If you keep on top of this, you can get a steady stream of free stuff for very little effort. You can follow Coin Master on Facebook or Twitter.

Sign up for email gifts

If you sign up for email gifts, you can get yourself a handful of Coin Master free spins every single day just be following a link on your phone. We haven’t encountered any spam from signing up so far either, so it’s a quick and easy method of getting yourself some tasty free spins.

Invite friends

Each time you invite a friend who successfully joins Coin Master through Facebook, you’ll get 40 Coin Master free spins, which is considerable. They don’t even have to actually play the game; they simply have to download it and login via their Facebook account to get you the free spins. Of course, it’s in both your interests to actually play it, which brings us nicely to our next point.

Request spins as gifts

You can get up to 100 Coin Master free spins per day from friends, though to get to those heights you’ll need 100 active friends who are kind enough to send you a gift each day. Each gift consists of a single free spin.

Unless you’re incredibly popular, it’s highly unlikely that you’ll have 100 friends; let alone 100 that will actually deign to play a game with you. We recommend heading on over to the official Reddit community or Facebook communities to try and find people willing to play with you.

Watch video ads

You can get a limited number of Coin Master free spins per day by watching a video ad. Simply scroll to the slot machine and tap on the spin energy button on the bottom right. If it’s not there, you’ve run out of free spins you can get through this method for the day, but if it is, simply tap on it and you’ll watch an ad.

Spin

Ironically, you can actually get a ton of Coin Master free spins by, well, spinning. If you get three spin energy symbols in a row, you’ll get a bunch of free spins. Pick up a chain of them and you can spin for ages before you run out.

Level up your village

Each time you level up your village, you’ll get a bunch of Coin Master free spins. It’s not easy though, as it costs a considerable amount of gold to purchase new buildings and improve them, and you have to purchase every single one of them, including improvements, to level up. That’s going to cost a lot of spins, as it is.

Participate in events

There’s almost always at least one event happening in Coin Master, and it can absolutely shower you with free spins. While viewing the slot machine, look at the top right of the screen. Any virtual buttons that you can see beneath the menu (which is displayed as three lines) is an event. Tap on one and you’ll see what each event involves.

Take advantage of these events and you can get yourself a lot more Coin Master free spins than usual.

Wait

This is an obvious suggestion, but it’s actually worth taking into consideration. You get five free spins every single hour, and you can only hold a maximum of 50 spins at any one time. That means every ten hours you’ll hit the maximum number of spins, and any Coin Master free spins you would have earned after that will cease to exist.

So, we recommend setting a reminder to visit Coin Master every ten hours at least to spend your spins so you are always earning more. You’ll actually end up earning a huge number of extra spins if you’re dedicated, so it’s totally worth doing.

Coin Master free spins FAQ:

Now, we’ll answer a bunch of questions you may have regarding getting Coin Master free spins.

Do Coin Master spins links expire?

Yes, the daily links that we include at the top of this page expire after three days, which is why we only include those from today and the two days prior.

Can I get Coin Master 50 spin rewards?

Coin Master 50 spin rewards most commonly appear during in-game events, like those that reward you for raiding or battling other players. There’s also a small chance to get this number from daily links, so bookmark this page and check back often.

How do I get Coin Master 60 spins?

Yes, though it doesn’t appear to happen often from daily links. We’d recommend playing often and participating in events, and following the social media channels to find out what’s happening soon.

How do I get Coin Master 70 spins?

We’ve never seen a Coin Master 70 spin reward appear as part of the daily links, but it has been known to appear as part of special events.

To get your hands on this rare reward, we would recommend playing on a daily basis and following the social media channels to get an indication of when the next big event will take place.

How do I get Coin Master 100 spins?

Yes, though not from the daily links. We’ve seen this number of free spins appear often during in-game events, most notably for those that reward you for raiding and participating in PvP battles.

Play often and follow the social media channels for events to keep an eye out for this.

How do I get Coin Master 400 spins?

Again, 400 spin rewards don’t seem to appear as part of the daily rewards cycle, but have been known to crop up during events. Follow the socials and play regularly to get the best chance at this reward.

2020 is nearly over, we’ve looked back on the best Switch games of the year, but it’s time to leave the past behind and look to the future. We’ve seen some fantastic games come to Switch in the last few months — several of them released within a matter of weeks of their surprise announcement — but we’ve also seen some big names slip into 2021 for understandable reasons.

Below we’ve rounded up thirty of the biggest Switch games we’re looking forward to in 2021. In no particular order (although vaguely chronological), they run the gamut from AAA first-party offerings to promising looking indies, plus a few big names that have been waiting in the wings for a while and stand a chance of breaking free this year (fingers crossed!).

Who can say exactly what 2021 holds for Nintendo and the Switch? A new hardware revision certainly isn’t out of the question, but whatever happens the schedule is already looking tidy.

Publisher: Ubisoft

Release Date: 2021 (USA) / 2021 (UK/EU)

Originally scheduled for the final weeks of 2020, Ubisoft will be bringing Scott Pilgrim vs. The World: The Game to Switch on 14th January 2021. It’s been a long time coming for this celebrated 360-era beat-em-up which was delisted from online stores back in 2014 and hasn’t been available since. Fortunately, this Complete Edition will give old and new players alike the chance to get stuck in against the world.

Publisher: Koei Tecmo / Developer: Gust

Release Date: 26th Jan 2021 (USA) / 29th Jan 2021 (UK/EU)

Koei Tecmo Europe and developer Gust Studios’ direct sequel to Atelier Ryza: Ever Darkness & The Secret Hideout is Switch-bound at the start of 2021 in the West. Atelier Ryza 2: Lost Legends & the Secret Fairy sees Ryza return as the protagonist, the first character in the history of the long-running RPG Atelier franchise to take on the hero role in two successive titles. You’ll be able to get your hands on this on 26th January 2021.

Publisher: Obsidian Entertainment / Developer: Obsidian Entertainment

Release Date: 2020 (USA) / 2020 (UK/EU)

Obsidian’s Pillars of Eternity picked up the torch put down by Baldur’s Gate many moons ago, and both of those games are available to play on your favourite Nintendo hybrid handheld. Pillars Of Eternity II: Deadfire is coming soon and is set five years after the events of the previous game. It’s been available on PS4 and Xbox One since January, although it had its own performance issues on thise consoles. Hopefully the extra time taken with the Switch port will iron out any significant issues. We expect this to arrive in Early 2021.

Publisher: Nintendo

Release Date: 12th Feb 2021 (USA)

The Wii U port we’ve all been waiting for, Super Mario 3D World + Bowser’s Fury will offer a tweaked experience that expands on the original game with new content. There’s a dual pack of Cat Mario and Cat Peach amiibo incoming on the same day, so expect an overflow of cuteness when this arrives on 12th February 2021.

Please note that some external links on this page are affiliate links, which means if you click them and make a purchase we may receive a small percentage of the sale. Please read our FTC Disclosure for more information.

Publisher: Modus Games / Developer: Dreams Uncorporated

Release Date: Q1 2021 (USA) / Q1 2021 (UK/EU)

Delayed from 17th November 2020 to ‘Early 2021’, Cris Tales is an indie RPG from Dreams Uncorporated that looks to pay homage to traditional JRPGs of old. The game’s been in development since 2014, so a little longer in the oven isn’t going to make much difference. “A delayed game is eventually good, but a bad game is always a rotter, unless it’s patched up the wazoo.” Some famous developer said something like that, we’re sure.

Publisher: Atlus / Developer: Atlus

Release Date: 23rd Feb 2021 (USA)

When Joker was announced for Super Smash Bros. Ultimate, we all assumed that Persona 5 would be coming to Switch. After all, it released on the PlayStation 3, so Switch wouldn’t struggle to run the game. However, while P5 got an updated release with Persona 5 Royal on PS4, there’s still no sign of that acclaimed game heading Switchwards. Instead, Japan got a hack-and-slash Dynasty Warriors crossover in the form of Persona 5 Scramble: The Phantom Strikers back in February 2020, and it’s now finally scheduled for a western release on 23rd February 2021.

While it might not be the game we were hoping for, we’ve seen some excellent Warriors crossovers in the form of Fire Emblem Warriors and most recently Hyrule Warriors: Age of Calamity. This has the potential to be an equally tasty mash up, so while we’ll keep hoping for Atlus to bring Persona 5 proper to Nintendo’s console, we’re intrigued to see what Omega Force has in store. Love that Joker.

Publisher: Square Enix / Developer: Team Asano

Release Date: 26th Feb 2021 (USA) / 26th Feb 2021 (UK/EU)

Unfortunately, this slipped from its original release date into 2021, but Square Enix’s Bravely Default II is one to watch for fans of the 3DS originals, or anyone with a taste for traditional-style JRPG gameplay and turn-based combat. There’s a demo available on the Switch eShop to whet your appetite for when the final game appears on 26th February 2021. Fingers crossed it doesn’t slip.

Publisher: Marvelous (XSEED)

Release Date: 23rd Mar 2021 (USA)

Story Of Seasons: Pioneers Of Olive Town will put you to work cultivating a farm just next to the titular port town and promises more customisation options and more freedom than ever before to live your best life, plus plenty more marriage candidates of course. You’ll have your work cut out for you clearing space in the forest for your new abode and cultivating all the bountiful natural wonders nearby. Marvelous’ next instalment in the Story of Seasons life/farm sim series is coming to Switch on 23rd March 2021.

Publisher: Capcom

Release Date: 26th Mar 2021 (USA)

Monster Hunter Rise has been described as “a brand new take on the Monster Hunter experience”. Up to four players can team up and save Kamura village from the impending ‘Rampage’, harnessing the new Wirebug hunting tool and enlisting the help of Palicoes, and Palamutes, along the way. The ‘Rise’ in the title is apparently a reference to this entry’s verticality, with the ability to traverse and climb rock faces and other terrain which will make the hunt all the more interesting. The game launches on Switch on 26th March 2021.

Publisher: Square Enix

Release Date: 26th Mar 2021 (USA) / 26th Mar 2021 (UK/EU)

A 3D action platforming game for 1-2 players, Balan Wonderworld brings back warm memories of 3D SEGA games like Nights Into Dreams on the Saturn and Billy Hatcher and the Giant Egg on GameCube. That’s probably because it’s a new game from Yuji Naka and Naoto Ohshima, perhaps best known as the creators of one Sonic the Hedgehog and involved in both those other titles we just mentioned. This one comes to Switch (and other consoles) on 26th March 2021.

Heatmaps are a specific type of plot which exploits the combination of color schemes and numerical values for representing complex and articulated datasets. They are largely used in data science application that involves large numbers, like biology, economics and medicine.

In this video we will see how to create a heatmap for representing the total number of COVID-19 cases in the different USA countries, in different days. For achieving this result, we will exploit Seaborn, a Python package that provides lots of fancy and powerful functions for plotting data.

Here’s the code to be discussed:

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns #url of the .csv file

url = r"path of the .csv file" # import the .csv file into a pandas DataFrame

df = pd.read_csv(url, sep = ';', thousands = ',') # defining the array containing the states present in the study

states = np.array(df['state'].drop_duplicates())[:40] #extracting the total cases for each day and each country

overall_cases = []

for state in states: tot_cases = [] for i in range(len(df['state'])): if df['state'][i] == state: tot_cases.append(df['tot_cases'][i]) overall_cases.append(tot_cases[:30]) data = pd.DataFrame(overall_cases).T

data.columns = states #Plotting

fig = plt.figure()

ax = fig.subplots()

ax = sns.heatmap(data, annot = True, fmt="d", linewidths=0, cmap = 'viridis', xticklabels = True)

ax.invert_yaxis()

ax.set_xlabel('States')

ax.set_ylabel('Day n°')

plt.show()

Let’s dive into the code to learn Seaborn’s heatmap functionality in a step-by-step manner.

Importing the required libraries for this example

We start our script by importing the libraries requested for running this example; namely Numpy, Pandas, Matplotlib and Seaborn.

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

What’s in the data?

As mentioned in the introduction part, we will use the COVID-19 data that were also used in the article about Scipy.curve_fit() function. Data have been downloaded from the official website of the “Centers for Disease Control and Prevention” as a .csv file.

The file reports multiple information regarding the COVID-19 pandemic in the different US countries, such as the total number of cases, the number of new cases, the number of deaths etc…; all of them have been recorded every day, for multiple US countries.

We will generate a heatmap that displays in each slot the number of total cases recorded for a particular day in a particular US country. To do that, the first thing that should be done is to import the .csv file and to store it in a Pandas DataFrame.

Importing the data with Pandas

The data are stored in a .csv file; the different values are separated by a semi-colon while the thousands symbol is denoted with a comma. In order to import the .csv file within our python script, we exploit the Pandas function .read_csv() which accepts as input the path of the file and converts it into a Pandas DataFrame.

It is important to note that, when calling .read_csv(), we specify the separator, which in our case is “;” by saying “sep = ‘;’” and the symbol used for denoting the thousands, by writing “thousands = ‘,’”. All these things are contained in the following code lines:

#url of the .csv file

url = r"path of the file" # import the .csv file into a pandas DataFrame

df = pd.read_csv(url, sep = ';', thousands = ',')

Creating the arrays that will be used in the heatmap

At this point, we have to edit the created DataFrame in order to extract just the information that will be used for the creation of the heatmap.

The first values that we extract are the ones that describe the name of the countries in which the data have been recorded. To better identify all the categories that make up the DataFrame, we can type “df.columns” to print out the header of the file. Among the different categories present in the header, the one that we are interested in is “state”, in which we can find the name of all the states involved in this chart.

Since the data are recorded on daily basis, each line corresponds to the data collected for a single day in a specific state; as a result, the names of the states are repeated along this column. Since we do not want any repetition in our heatmap, we also have to remove the duplicates from the array.

We proceed further by defining a Numpy array called “states” in which we store all the values present under the column “state” of the DataFrame; in the same code line, we also apply the method .drop_duplicates() to remove any duplicate of that array. Since there are 60 states in the DataFrame, we limit our analysis to the first 40, in order not to create graphical problems in the labels of the heatmap x-axis, due to the limited window space.

#defining the array containing the states present in the study

states = np.array(df['state'].drop_duplicates())[:40] 

The next step is to extract the number of total cases, recorded for each day in each country. To do that, we exploit two nested for loops which allow us creating a list containing the n° of total cases (an integer number for each day) for every country present in the “states” array and appending them into another list called “overall_cases” which needs to be defined before calling the for loop.

#extracting the total cases for each day and each country

overall_cases = []

As you can see in the following code, in the first for loop we iterate over the different states that were previously stored into the “states” array; for each state, we define an empty list called “tot_cases” in which we will append the values referred to the total cases recorded at each day.

for state in states: tot_cases = []

Once we are within the first for loop (meaning that we are dealing with a single state), we initialize another for loop which iterates through all the total cases values stored for that particular state. This second for loop will start from the element 0 and iterate through all the values of the “state” column of our DataFrame. We achieve this by exploiting the functions range and len.

 for i in range(len(df['state'])):

Once we are within this second for loop, we want to append to the list “tot_cases” only the values that are referred to the state we are currently interested in (i.e the one defined in the first for loop, identified by the value of the variable “state”); we do this by using the following if statement:

 if df['state'][i] == state: tot_cases.append(df['tot_cases'][i])

When we are finished with appending the values of total cases for each day of a particular country to the “tot_cases” list, we exit from the inner for loop and store this list into the “overall_cases” one, which will then become a list of lists. Also in this case, we limit our analysis to the first 30 days, otherwise we would not have enough space in our heatmap for all the 286 values present in the DataFrame.

 overall_cases.append(tot_cases[:30])

In the next iteration, the code will start to analyze the second element of the “states” array, i.e. another country, will initialize an empty list called “tot_cases” and enter in the second for loop for appending all the values referred to that country in the different days and eventually, once finished, append the entire list to the list “overall_cases”; this procedure will be iterated for all the countries stored in the “states” array. At the end, we will have extracted all the values needed for generating our heatmap.

Creating the DataFrame for the heatmap

As already introduced in the first part, we exploit the Seaborn function .heatmap() to generate our heatmap.

This function can take as input a pandas DataFrame that contains the rows, the columns and all the values for each cell that we want to display in our plot. We hence generate a new pandas DataFrame (we call it “data”) that contains the values stored in the list “overall_cases”; in this way, each row of this new DataFrame is referred to a specific state and each column to a specific day.

We then transpose this DataFrame by adding “.T” at the end of the code line, since in this way we can then insert the name of the states as the header of our Dataframe.

data = pd.DataFrame(overall_cases).T

The names of the states were previously stored in the array “states”, we can modify the header of the DataFrame using the following code:

data.columns = states

The DataFrame that will be used for generating the heatmap will have the following shape:

   CO  FL  AZ  SC  CT  NE  KY  WY  IA  ...  LA  ID  NV  GA  IN  AR  MD  NY  OR 0   0   0   0   0   0   0   0   0   0  ...   0   0   0   0   0   0   0   0   0 1   0   0   0   0   0   0   0   0   0  ...   0   0   0   0   0   0   0   0   0 2   0   0   0   0   0   0   0   0   0  ...   0   0   0   0   0   0   0   0   0 3   0   0   0   0   0   0   0   0   0  ...   0   0   0   0   0   0   0   0   0 4   0   0   1   0   0   0   0   0   0  ...   0   0   0   0   0   0   0   0   0 

The row indexes represent the n° of the day in which the data are recorded while the columns of the header are the name of the states.

Generating the heatmap

After generating the usual plot window with the typical matplotlib functions, we call the Seaborn function .heatmap() to generate the heatmap.

The mandatory input of this function is the pandas DataFrame that we created in the previous section. There are then multiple optional input parameters that can improve our heatmap:

• linewidths allows adding a white contour to each cell to better separate them, we just have to specify the width;
• xticklabels modify the notation along the x-axis, if it’s equal to True, all the values of the array plotted as the x-axis will be displayed.
• We can also chose the colormap of the heatmap by using cmap and specifying the name of an available heatmap (“viridis” or “magma” are very fancy but also the Seaborn default one is really cool);
• finally, it is possible to display the numerical value of each cell by using the option annot = True; the numerical value will be displayed at the center of each cell.

The following lines contain the code for plotting the heatmap. One final observation regards the command .invert_yaxis(); since we plot the heatmap directly from a pandas DataFrame, the row index will be the “day n°”; hence it will start from 0 and increase as we go down along the rows. By adding .invert_yaxis() we reverse the y-axis, having day 0 at the bottom part of the heatmap.

#Plotting

fig = plt.figure()

ax = fig.subplots()

ax = sns.heatmap(data, annot = True, fmt="d", linewidths=0, cmap = 'viridis', xticklabels = True)

ax.invert_yaxis()

ax.set_xlabel('States')

ax.set_ylabel('Day n°')

plt.show() 

Figure 1 displays the heatmap obtained by this code snippet.

Figure 1: Heatmap representing the number of COVID-19 total cases for the first 30 days of measurement (y-axis) in the different USA countries (x-axis).

As you can see in Figure 1, there are a lot of zeroes, this is because we decided to plot the data related to the first 30 days of measurement, in which the n° of recorded cases were very low. If we decided to plot the results from all the days of measurement (from day 0 to 286), we would obtain the result displayed in Figure 2 (in this latter case, we placed annot equal to False since the numbers would have been too large for the cell size):

Figure 2: Heatmap representing the number of COVID-19 total cases for the first 286 days of measurement (y-axis) in the different USA countries (x-axis); this time annot = False, since the cells are too small for accommodating the n° of total cases (which becomes very large towards the upper part of the heatmap).

The post Creating Beautiful Heatmaps with Seaborn first appeared on Finxter.