Skip to main content

Converting an Excel Worksheet into a JSON document with C# and .NET Core and ExcelDataReader

Excel isn't a database, except when it isI've been working on a little idea where I'd have an app (maybe a mobile app with Xamarin or maybe a SPA, I haven't decided yet) for the easily accessing and searching across the 500+ videos from http://friday.azure.com.

HOWEVER. I don't have access to the database that hosts the metadata and while I'm trying to get at least read-only access to it (long story) the best I can do is a giant Excel spreadsheet dump that I was given that has all the video details.

This, of course, is sub-optimal, but regardless of how you feel about it, it's a database. Or, a data source at the very least! Additionally, since it was always going to end up as JSON in a cached in-memory database regardless, it doesn't matter much to me.

In real-world business scenarios, sometimes the authoritative source is an Excel sheet, sometimes it's a SQL database, and sometimes it's a flat file. Who knows?

What's most important (after clean data) is that the process one builds around that authoritative source is reliable and repeatable. For example, if I want to build a little app or one page website, yes, ideally I'd have a direct connection to the SQL back end. Other alternative sources could be a JSON file sitting on a simple storage endpoint accessible with a single HTTP GET. If the Excel sheet is on OneDrive/SharePoint/DropBox/whatever, I could have a small serverless function run when the files changes (or on a daily schedule) that would convert the Excel sheet into a JSON file and drop that file onto storage. Hopefully you get the idea. The goal here is clean, reliable pragmatism. I'll deal with the larger business process issue and/or system architecture and/or permissions issue later. For now the "interface" for my app is JSON.

So I need some JSON and I have this Excel sheet.

Turns out there's a lovely open source project and NuGet package called ExcelDataReader. There's been ways to get data out of Excel for decades. Literally decades. One of my first jobs was automating Microsoft Excel with Visual Basic 3.0 with COM Automation. I even blogged about getting data out of Excel into ASP.NET 16 years ago!

Today I'll use ExcelDataReader. It's really nice and it took less than an hour to get exactly what I wanted. I haven't gone and made it super clean and generic, refactored out a bunch of helper functions, so I'm interested in your thoughts. After I get this tight and reliable I'll drop it into an Azure Function and then focus on getting the JSON directly from the source.

A few gotchas that surprised me. I got a "System.NotSupportedException: No data is available for encoding 1252." Windows-1252 or CP-1252 (code page) is an old school text encoding (it's effectively ISO 8859-1). Turns out newer .NETs like .NET Core need the System.Text.Encoding.CodePages package as well as a call to System.Text.Encoding.RegisterProvider(System.Text.CodePagesEncodingProvider.Instance); to set it up for success. Also, that extra call to reader.Read at the start to skip over the Title row had me pause a moment.

using System;

using System.IO;

using ExcelDataReader;

using System.Text;

using Newtonsoft.Json;


namespace AzureFridayToJson

{

    class Program

    {

        static void Main(string[] args)

        {

            Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);


            var inFilePath = args[0];

            var outFilePath = args[1];


            using (var inFile = File.Open(inFilePath, FileMode.Open, FileAccess.Read))

            using (var outFile = File.CreateText(outFilePath))

            {

                using (var reader = ExcelReaderFactory.CreateReader(inFile, new ExcelReaderConfiguration()

                { FallbackEncoding = Encoding.GetEncoding(1252) }))

                using (var writer = new JsonTextWriter(outFile))

                {

                    writer.Formatting = Formatting.Indented; //I likes it tidy

                    writer.WriteStartArray();

                    reader.Read(); //SKIP FIRST ROW, it's TITLES.

                    do

                    {

                        while (reader.Read())

                        {

                            //peek ahead? Bail before we start anything so we don't get an empty object

                            var status = reader.GetString(0);

                            if (string.IsNullOrEmpty(status)) break;


                            writer.WriteStartObject();

                            writer.WritePropertyName("Status");

                            writer.WriteValue(status);


                            writer.WritePropertyName("Title");

                            writer.WriteValue(reader.GetString(1));


                            writer.WritePropertyName("Host");

                            writer.WriteValue(reader.GetString(6));


                            writer.WritePropertyName("Guest");

                            writer.WriteValue(reader.GetString(7));


                            writer.WritePropertyName("Episode");

                            writer.WriteValue(Convert.ToInt32(reader.GetDouble(2)));


                            writer.WritePropertyName("Live");

                            writer.WriteValue(reader.GetDateTime(5));


                            writer.WritePropertyName("Url");

                            writer.WriteValue(reader.GetString(11));


                            writer.WritePropertyName("EmbedUrl");

                            writer.WriteValue($"{reader.GetString(11)}player");

                            /*

                            <iframe src="https://channel9.msdn.com/Shows/Azure-Friday/Erich-Gamma-introduces-us-to-Visual-Studio-Online-integrated-with-the-Windows-Azure-Portal-Part-1/player" width="960" height="540" allowFullScreen frameBorder="0"></iframe>

                             */


                            writer.WriteEndObject();

                        }

                    } while (reader.NextResult());

                    writer.WriteEndArray();

                }

            }

        }

    }

}

The first pass is on GitHub at https://github.com/shanselman/AzureFridayToJson and the resulting JSON looks like this:

[

  {

    "Status": "Live",

    "Title": "Introduction to Azure Integration Service Environment for Logic Apps",

    "Host": "Scott Hanselman",

    "Guest": "Kevin Lam",

    "Episode": 528,

    "Live": "2019-02-26T00:00:00",

    "Url": "https://azure.microsoft.com/en-us/resources/videos/azure-friday-introduction-to-azure-integration-service-environment-for-logic-apps",

    "embedUrl": "https://azure.microsoft.com/en-us/resources/videos/azure-friday-introduction-to-azure-integration-service-environment-for-logic-appsplayer"

  },

  {

    "Status": "Live",

    "Title": "An overview of Azure Integration Services",

    "Host": "Lara Rubbelke",

    "Guest": "Matthew Farmer",

    "Episode": 527,

    "Live": "2019-02-22T00:00:00",

    "Url": "https://azure.microsoft.com/en-us/resources/videos/azure-friday-an-overview-of-azure-integration-services",

    "embedUrl": "https://azure.microsoft.com/en-us/resources/videos/azure-friday-an-overview-of-azure-integration-servicesplayer"

  },

...SNIP...

Thoughts? There's a dozen ways to have done this. How would you do this? Dump it into a DataSet and serialize objects to JSON, make an array and do the same, automate Excel itself (please don't do this), and on and on.

Certainly this would be easier if I could get a CSV file or something from the business person, but the issue is that I'm regularly getting new drops of this same sheet with new records added. Getting the suit to Save As | CSV reliably and regularly isn't sustainable.

Sponsor: Stop wasting time trying to track down the cause of bugs. Sentry.io provides full stack error tracking that lets you monitor and fix problems in real time. If you can program it, we can make it far easier to fix any errors you encounter with it.


© 2018 Scott Hanselman. All rights reserved.
     


from Scott Hanselman's Blog http://feeds.hanselman.com/~/599260678/0/scotthanselman~Converting-an-Excel-Worksheet-into-a-JSON-document-with-C-and-NET-Core-and-ExcelDataReader.aspx
Labels:

Comments

Post a Comment

Popular posts from this blog

dotnet sdk list and dotnet sdk latest

Can someone make .NET Core better with a simple global command? Fanie Reynders did and he did it in a simple and elegant way. I'm envious, in fact, because I spec'ed this exact thing out in a meeting a few months ago but I could have just done it like he did and I would have used fewer keystrokes! Last year when .NET Core was just getting started, there was a "DNVM" helper command that you could use to simplify dealing with multiple versions of the .NET SDK on one machine. Later, rather than 'switching global SDK versions,' switching was simplified to be handled on a folder by folder basis. That meant that if you had a project in a folder with no global.json that pinned the SDK version, your project would use the latest installed version. If you liked, you could create a global.json file and pin your project's folder to a specific version. Great, but I would constantly have to google to remember the format for the global.json file, and I'd constan
Post a Comment
Read more

R vs Python for Machine Learning

There are so many things to learn before to choose which language is good for Machine Learning. We will discuss each and everything about R as well as Python and the situation or problem in which situation we have to use which language. Let’s start Python and R are the two most Commonly used Programming Languages for Machine Learning and because of the popularity of both the languages Novice or you can say fresher are getting confused, whether they should choose R or Python language to commence their career in the Machine learning domain. Don’t worry guys through this article we will discuss R vs Python for Machine Learning. So, without exaggerating this article let’s get started. We will start it from the very Basics things or definitions. R vs Python for Machine Learning Introduction R is a programming language made by statisticians and data miners for statistical analysis and graphics supported by R foundation for statistical computing. R also provides high-quality graphics and
1 comment
Read more

Top Tips For PCB Design Layout

Are you thinking about designing a printed circuit board? PCBs are quite complicated, and you need to make sure that the layout that you choose is going to operate as well as you want it to. For this reason, we have put together some top tips for PCB design layout. Keep reading if you would like to find out more about this. Leave Enough Space One of the most important design tips for PCB layout is that you need to make sure that you are leaving enough space between the components. While many people might think that packing components closely is the best route to take, this can cause problems further down the line. This is why we suggest leaving extra space for the wires that will spread. This way, you’ll have the perfect PCB design layout. Print Out Your Layout Struggling to find out if your components sizes match? Our next tip is to print out your layout and compare the printed version to your actual components. Datasheets can sometimes come with errors, so it doesn’t hurt to do
1 comment
Read more