I had to face a log file in a JSON format, which managed to grow over 700MB with 1 million plus entries in it, and my regular code wasn’t able to process it any more. So I was in trouble.
Solution Link to heading
I turned to a search engine and started looking for a solution, and I run into this awesome article from 2014 from the Happy SysAdmin Reading large text files with PowerShell, and the [System.IO.File]::ReadAllLines() code look really promising.
Using this standard .NET method it was easy, and really fast to get the content of the file into the memory. But I still had to convert it into [PSCustomObject[]] and I knew that I cannot simply pass over a 1o million plus lines of text to ConvertFrom-JSON because PowerShell would probably just laugh into my face. And it did.
How to deal with 10 millions of lines of JSON code? Link to heading
I was lucky, I knew that all entries in my JSON structure look the same, 9 properties, always. And it was a well formatted file, so I could iterate over the lines and parse it by the number of lines. And this Get-JsonContentX was born.