I’ve moved my blog over to WordPress because of the nasty SharePoint exploit that my beloved SharePoint 2010 is vulnerable to. While SharePoint is a fine blogging platform, I thought I’d go with something a little more conventional since I have to move. I’ll be spending the next few weeks figuring out how to get this all set up and getting my content moved over. Thanks for your patience.
tk
I know Mondays can be rough. Fortunately I have something to help you through this particular Monday. I was recently interviewed on the “Mastering Mondays” podcast from the No More Bad Mondays guys. I had a blast catching up with Dave and Matt. We talk about our favorite tech and I recount an experience in my childhood that nearly scarred me for life. The time flew by. Give the episode a listen on your favorite platform and let me know what you think.
tk
ShortURL: https://www.toddklindt.com/HappyMonday
I know I have a checkered past with the Developer community. Back in the old days they could do some pretty awful things to my beloved on-prem SharePoint servers with their incessant BINing and GACing things. Fortunately for all of us (including those defenseless servers) those days are behind us.
So, I just got off this Microsoft 365 & Power Platform Development Community call, and boy, do I have some cool stuff to share with you. I got the chance to show off this neat trick I’ve been working on – using GitHub Copilot and ChatGPT (or any AI you’re into) to give your PowerShell scripting a serious boost.
You’re probably thinking, “AI and PowerShell, really?” But stick with me here. It’s like having a co-pilot for your coding. Like someone smarter than you, looking over your shoulder, and there if you have questions. It’s there to help you out, make things smoother, and let’s be honest, who doesn’t want to feel a bit like Tony Stark talking to JARVIS while coding?
During the demo, I took everyone on a little tour of how you can get GitHub and Copilot and ChatGPT into the mix with your PowerShell development routine. It’s all about using AI to help with the heavy lifting – coding, debugging, repetitive tasks, and even the dreaded documentation. I’m telling you, this is next-level stuff, and we’re just scratching the surface of what AI can do for us.
But hey, don’t just take my word for it. Give it a whirl! Play around with adding some AI into your PowerShell development and see how it goes. I used ChatGPT and GitHub Copilot for the demo, but you can pick any AI you’re comfortable with.
That’s it from me for now. Keep an eye out for more cool AI posts.
Microsoft 365 & Power Platform Development Community call
tk
ShortURL: https://www.toddklindt.com/PoshwithAI
In my last masterpiece, Tackling SharePoint’s 5000 Item Limit with PowerShell and Search, I show how to use PowerShell with Search to return more than 5000 items from a SharePoint list or library, even if SharePoint refuses to return more than 5000 items. Pretty impressive, right? When demonstrating something like that you need a list or library with more than 5000 items. There are are a lot of scripts out there that can create test data, but I needed something specific. So, I opened up a new windows in VS Code and got to coding.
The PowerShell function I wrote is Add-AttorneyFiles, which is designed to streamline the creation of attorney files and case folders. Lots and lots of them.
This function has a few parameters to tailor the files and folders it creates. It accepts two mandatory parameters, AttorneyCount and CaseCount, specifying the number of attorneys and case folders to create for each attorney.
There are also a few switch parameters that allow you to customize the function’s behavior further. You can choose to create a file in each case folder using the CreateStaticFile switch, or specify that only closed or client case folders should be created with the OnlyClosedCases and OnlyClientCases switches.
You can also specify the name of the static file created using the StaticFileName parameter. If no name is provided, the function will default to creating a file named “readme.txt“.
The function starts by checking if a connection to a SharePoint site exists. If it does, the function creates attorney files and case folders in a SharePoint directory. The names for these attorney files are generated randomly from a list of common first and last names. This randomness helps to create a more realistic environment for testing.
Once the list of attorney names is created, the function will loop through each attorney, creating the appropriate case folders, either client case folders, closed case folders, or both, based on the parameters passed in.
The function gives the folders and files it creates a randomly generated last and first name, along with a random case number, for a more realistic setup.
If the CreateStaticFile switch is present, a static file will be created in each case folder. The content of these static files is a random selection of words, downloaded from a free online dictionary. This randomization also contributes to a more realistic testing environment.
Here’s an example of how you might use this function:
Add-AttorneyFiles -AttorneyCount 10 -CaseCount 5 –CreateStaticFile
This will create 10 attorney folders, each with 5 case folders. A static file named “readme.txt” will be created in each case folder. For my blog a created a whole lot of autorun.inf files.
By automating the setup of this testing environment, I was able to generate a high volume of test data, with a format that accurately represented my client’s data, without exposing any of it.
I’ve uploaded the code to GitHub. Check it out.
Happy PowerShelling, and as always, feel free to drop any questions or comments below!
tk
In the world of SharePoint, the 5000 item per view limit is a well-known challenge, the stuff of legend. To recap, while a SharePoint list or library can have up to 30 million items in it, SharePoint refuses to show you more than 5000 of them at a time. That’s an API level control to protect the backend, so it won’t do this in a view in a web page, an API call through PowerShell or CLI, nothing. While there are several ways to navigate around this limit, such as using CAML queries, these methods often fall short when dealing with really large datasets. This blog post will explore a unique solution to this problem using PowerShell, specifically focusing on the use of Search to retrieve data.
Recently, I was working with a client, a law firm, who had a whopping 7.8 million items in a SharePoint document library. Of course not a great Information Architecture. We were helping them fix that. Among other horrors, over the years, attorneys had copied the contents of CDs and DVDs to various places in SharePoint, creating a massive and complex data structure. The challenge was to find all these so we could migrate them out or delete them. However, due to the sheer volume of data, there was no way to slice the data using the normal tools to get back fewer than 5000 results. This is where PowerShell swoops in and saves the day.
The solution came in two parts. We were looking for the DVDs by looking for the autorun.inf file in the root. I discovered I couldn’t using something like Get-PnPListItem to find all of the autorun.inf files because I couldn’t find a way with CAML, or anything else to pare the result set down below 5000. There were just too many files. However, I did discover that I could find them in the Search Center, which gave me the first idea. Get them from Search in PowerShell. I used the Submit-PnPSearchQuery cmdlet to send a search query to SharePoint and it retrieves all the results. It gets them in batches of 500 (the maximum for a single search request), with the option of making multiple requests if necessary to retrieve all results. This worked pretty well, but was tedious because I could only get 500 at a time and there were thousands. I had to modify the search, run it again, and append those results to the results from the previous searches. That was too much work, and led to the second part. I wrote a PowerShell function called Submit-PnPSearchQueryAll. This function uses the Submit-PnPSearchQuery cmdlet to send a search query to SharePoint and retrieves all results, paging through them and running multiple queries as needed. If the -ShowProgress switch is provided, the function will display the total number of results and a progress bar.
Before we look at the function itself, let’s see how it works. My usage looked like this:
$AllResults = Submit-PnPSearchQueryAll -query “autorun.inf”
That stored every file named autorun.inf in the variable $AllResults. There are 5046 of them:
$AllResults.Count
Since that’s a collection of objects I can treat them like any old object:
$AllResults[1000]
and
$AllResults | Where-Object {$_.ParentLink -like “*AttorneyFiles/Johnson, Michael*” }
or
$AllResults | Where-Object {$_.ParentLink -like “*AttorneyFiles/Johnson, Michael*” } | select Path,ParentLink
and
$AllResults | Where-Object {$_.ParentLink -like “*AttorneyFiles/Johnson, Michael*” } | Export-Csv .\mj.csv
See all the fun you can have? Since the object we’re getting back is a PnPResultTable object, it doesn’t have all the same properties as a PnPListItem. When I wrote the function I had to decide which ones I needed. If you use this, you might need something different.
The function begins by initializing variables for the starting row and page size, set to 500. It then enters a loop where it performs the search query with Submit-PnPSearchQuery and retrieves the results. If the –ShowProgress switch is provided, it will display the total number of results on the first run and a progress bar for each subsequent run.
For each result, the function outputs a custom object with the desired properties. It then increments the $startRow by the $pageSize and continues the loop while the $startRow is less than the Total Rows.
Before I got this working I tried a couple of other approaches, but this one worked the best.
Grab the PSM1 file with Submit-PnPSearchQueryAll here.
This PowerShell function proved to be an effective solution to the SharePoint 5000 item limit, allowing us to retrieve all items from a massive SharePoint document library. It demonstrates the power and flexibility of PowerShell and SharePoint’s Search functionality when dealing with large datasets. Whether you’re dealing with millions of items or just want a more efficient way to retrieve data from SharePoint, consider giving this function a try.
tk
Like mixing chocolate and peanut butter I’m mixing two of my favorite things, SysKit and Sympraxis, in one magnificent webinar. Toni, Marc, Derek, and I will be talking about how technology can improve business and how we work with customers. And then, we’ll spend a lot of time with the mics open metaphorically and taking questions from the audience. So if you want to stump Marc or Derek, now is your chance. 
The magic happens Wednesday, Sep 22 at 10:00 am CDT. Please sign up free and join us. We’d love to have you.
tk
I don’t know if I could pick my favorite blog crash. There have been so many over the years. It would be like picking my favorite kid, or favorite tablet. I just can’t.
This most recent crash was caused by the drive the VM my blog is running on failing. “But what about backups??” I hear you all shouting in unison. I had VM backups in place, but I had moved the VM to a different server and the backup job didn’t follow along. I also had database backups. These were going directly to my NAS device. Unfortunately when I replaced my NAS in December I forgot to point the SQL backup job to the new location. My only backup was a lonely database backup from December. So, with my SharePoint 2010 and SQL 2012 CDs in hand, I reinstalled everything, and here we are.
I was able to republish the blog posts I’ve written since the backup, so there was no loss there. I did lose the comments you’ve left and I’ve responded to. Sorry about that. I’m in the process of putting it all back together, but there are a few things I haven’t finished yet. I’m putting the list here, so we can all keep track of it. If you see something wrong and it’s not on this list, send me a tweet or leave me a comment (I’ll try not to lose it) and I’ll add it to the list.
To Do List:Get Search working
Update SharePoint 2010 Builds list(9/10/21)Update SharePoint 2013 Builds list(9/13/21)
Go through IIS Logs and look for failed requests for ShortURLs I’ve missed
Verify all the http to https redirects work properlyCreate SQL backup job (9/13/21)
Create VM backup job
Fix broken images in blog posts
Thanks for sticking around,
tk
Since the PnP team announced that the venerable SharePointPnPPowerShellOnline module was going to be replaced by the shiny, new PnP.PowerShell module there has been some confusion on which module to use and which PowerShell to use. I have good news, the answer is “D. All of the above.” In this blog post I’ll show you how you can have both modules installed and use them interchangeably as well as use either of them on whatever version of PowerShell tickles your fancy.
Before we can get into how to do this I want to spend some pixels on why we have to do it the way we do. I intended to publish this blog post a couple of weeks (okay, months) ago but as I was doing my research I kept learning more. Most of learning was around the “Module” cmdlets and what each of them does. In order to get the two modules to cooperate we need to use the right Module cmdlets at the right time. Here are the Cliff’s Notes:
Several Module cmdlets deal with what PowerShell Modules are installed on your system with PowerShellGet. There are a lot of ways to install Modules into PowerShell and PowerShellGet is one of the most popular and it’s built into PowerShell. Here are the pertinent cmdlets and what they do:
The key here is that the scope of those commands, and the other PowerShellGet cmdlets, is the whole computer
“Host” is a fancy word for PowerShell window or console. There are a few common Module cmdlets that deal only with the host they’re run in. Here are some of my favorites:
These cmdlets are all part of Microsoft.PowerShell.Core as opposed to PowerShellGet.
Now we know how to get a module installed onto our computer, and how to manually coax it into our host. But most of us have never done all this Import-Module business but everything seems to work. How’s that? PowerShell has the ability to automatically load modules when they’re needed. The entire, exciting, story of how PowerShell imports modules is chronicled here, but I’ll give you the highlights. If you try to use (or reference it with something like Get-Command) a cmdlet that isn’t in a module already loaded PowerShell will walk through the PSModulePath locations looking into each module for the cmdlet you’re trying to run. If it finds it, it implicitly imports that module and Bob’s your uncle. You can use $env:PSModulePath to see where PowerShell will look. It’s important to note that Windows PowerShell 5 and PowerShell 7 have similar, but different PSModulePaths. Here’s what it looks like in both. I added -split “’;” to put each path on its own line, and I piped it through Sort-Object to make them easy to keep track of.
To make either Module, SharePointPnPPowerShellOnline or PnP.PowerShell, available to both Windows PowerShell 5 and PowerShell 7 it has to be installed in one of the paths that both versions of PowerShell will look in. For backwards compatibility PowerShell 7 looks in the PowerShell 5 paths, so I do the Install-Module bit in PowerShell 5 and PowerShell 7 gets it for free. You could install it in both, that works. But this way keeps you from having to update it both places every time a new version comes out. The PnP.PowerShell module is aimed for PowerShell 7, but also works fine in PowerShell 5. Erwin told me they may remove that support in future, but for now it’s safe.
You can see from the screenshots that I have my Known Folders redirected to OneDrive. When installing these modules PowerShell was installing them to my personal folders and it was causing problems. A couple of the Module cmdlets don’t handle that well. To get around some of that chicanery I had to install the modules in a different path. The easiest way I found to do that was the specify the scope Allusers, like this:
Install-Module PnP.PowerShell -Scope AllUsers
In PowerShell 5 that installed into C:\Program Files\WindowsPowerShell\Modules\PnP.PowerShell, which PowerShell 7 can see so it can be imported into a host of either version and it’s not in OneDrive.
We know both PowerShells can run both modules if we install it right. To get everything playing nicely I uninstalled both modules from both versions of PowerShell using Uninstall-Module -Force –AllVersions. In one case Uninstall couldn’t clean it all up so I had to go into the file system and delete the folder manually. I also had to close the host various times as it had imported the module I was trying to uninstall and I couldn’t get it unloaded in that host.
After all the uninstalling was done I closed all of the PowerShell windows I had open and I opened a PowerShell 5 window in Admin mode. I installed the old module with this:
Install-Module -Name SharePointPnPPowerShellOnline -Scope Allusers
That made it so it could be imported into both PowerShell 5 and PowerShell 7. Next I installed the PnP.PowerShell module. This one took a bit of extra coaxing. It’s currently in prerelease so Install-Module requires the –AllowPrerelease parameter. The version of PowerShellGet in PowerShell 5 does have that. I had to upgrade PowerShellGet first with this line:
Update-Module -Name PowerShellGet
I close the window and opened a new one for good measure. This put me at version 2.2.5 of PowerShellGet. One problem solved. The second problem is that PnP.PowerShell and SharePointPnPPowerShellOnline share most of their cmdlet names, so a regular Install-Module is going to fail. To fix that we need to use the –AllowClobber parameter. The whole thing looks like this:
Install-Module -Name PnP.PowerShell -AllowPrerelease –AllowClobber
At first this sounds scary, but remember, Install is just dropping the bits onto your computer. You can still control which module gets loaded in a script or host. To do that using Import-Module. The key is to use Import-Module before you do anything that will trigger PowerShell implicitly loading the module for you. If you want to force your script to use a specific module include one of these lines at the top:
Import-Module PnP.PowerShell
or
Import-Module SharePointPnPPowerShellOnline
You would do the same thing in a PowerShell window when you open it to run some cmdlets. I’ve been doing this for a few weeks and it works well.
Now I’m just going to show off a bit. We know we can load either module into a host, but what if we need both modules into the same host or script at the same time? It came be done! It sounds like magic, but it works. The key is using the –Prefix parameter of Import-Module. If you want both modules loaded and available in the same window you need to import one with a prefix. It looks like this:
Import-Module SharePointPnPPowerShellOnline -Prefix Old
You can see from the screenshot that both sets of cmdlets are available, with the SharePointPnPPowerShellOnline versions having the additional prefix of “Old.” I wouldn’t recommend doing this as a normal course of action, but it’s good to know it’s there. PowerShell 7 is a little fussy when doing this. I had issues importing the old version if I had already imported the PnP.PowerShell module explicitly or implicitly, which is why I didn’t in this screenshot. Get-Command implicitly loaded it for me, so that wasn’t an issue. PowerShell 5 handles it more gracefully.
The End
There it is, the culmination of weeks (maybe months) of me fiddling around to understand the inner workings of PowerShell module installing and importing. Thanks to Jeff Hicks for holding my hand and answering all my dumb questions. Hopefully my pain will make your transitions from PowerShell 5 to PowerShell 7 and from SharePointPnPPowerShellOnline to PnP.PowerShell easier.
tk
As the new year approaches I find myself making changes. Like everyone on the planet I plan to eat less and exercise more. I also plan on making another change, I’ll be moving away from my friend the venerable old SharePointPnPPowerShellOnline module to the new, sexier PnP.PowerShell. I thought I’d blog the experience so you all can follow along.
The SharePointPnPPowerShellOnline module has been good to me over the last few years. It has helped me do amazing things both in PowerShell and in SharePoint. Why would I ever give that all up? Because the times, they are a-changing. The SharePointPnPPowerShellOnline module is being retired and all of that effort is being channeled to the PnP.PowerShell module. The SharePointPnPPowerShellOnline module will continue to be around for the on-prem versions of SharePoint Server (though not updated), but for SharePoint Online/Office 365/Microsoft 365 you’ll need to use the PnP.PowerShell module.
Why did the PnP team decide this? You can get the whole story at the official PnP PowerShell page, but there are two main reasons. The first is wider platform support. SharePointPnPPowerShellOnline had .NET dependencies that meant it would only run on Windows PowerShell. Emphasis on Windows. When that module was first created 6 years ago that made perfect sense, but since then PowerShell has gone open source and now runs on a variety of platforms like Mac, Linux, a variety toasters, etc. Since SharePointPnPPowerShellOnline required Windows it couldn’t follow PowerShell onto those other platforms. The PnP.PowerShell module was written without those Windows dependencies so it can run anywhere PowerShell can run. You may not use a Mac, but this also means that you can use PnP.PowerShell more easily in cloud environments like Azure Functions.
The second reason was to give wider application support. The SharePointPnPPowerShellOnline mostly supported SharePoint, as the name would suggest. These days you can’t manage SharePoint Online without also needing to touch Microsoft Groups, Teams, etc. The PnP added more and more cmdlets to handle that, but there were some big authentication changes that needed to be made to fully support all of the other applications. They started adding some of that to SharePointPnPPowerShellOnline, but it was cobbled together a bit. While they were refactoring the module for .NET Standard they went ahead and made some sweeping authentication changes that mean we’ll be able to use it more easily for SharePoint and all of the other applications we know and love.
The first step to moving to the PnP.PowerShell module is to install PowerShell 7, also called PowerShell Core. I’m on Windows, so that’s the process I’m going to walk through. Windows comes with PowerShell, Windows PowerShell 5.1. To use PowerShell 7 we’ll have to install it. Windows PowerShell 5.1 and PowerShell Core 7 can happily coexist on your machine, so you don’t have to worry about breaking any of your other existing PowerShell scripts or tools.
You can see from this screenshot I have both versions of PowerShell installed and running on this machine. You can use the system variable $PSVersionTable to see which version of PowerShell your host is currently using. It’s also pretty easy to pick the version of PowerShell you want when firing it up.
While PowerShell 7.x can coexist with Windows PowerShell 5.x, it will replace PowerShell 6.x if you have that installed.
I installed PowerShell 7 by downloading the installation MSI from the GitHub page. For me that was the Windows x64 platform and I grabbed the stable build. That file was PowerShell-7.1.0-win-x64.msi, but obviously that filename will change as PowerShell 7 advances. Then I popped open a Windows PowerShell 5 (oh, the irony) prompt in Admin mode and ran the MSI.
The friendly wizard walked me through the process. There are few installation choices to make, but I’ve found the defaults are usually fine.
After the installation is finished you’ll have both Windows PowerShell 5.x and PowerShell Core 7.x. Time to install some modules.
Installing the PnP.PowerShell module is pretty easy, but if you have the older SharePointPnPPowerShellOnline module installed you’ll need to uninstall it first. The cmdlet names in both modules are the same, So PnP.PowerShell won’t install all of its cmdlets while there are collisions with the old version. To uninstall the SharePointPnPPowerShellOnline module open up a Windows PowerShell 5 host in Admin mode and enter this:
Uninstall-Module SharePointPnPPowerShellOnline –AllVersions
and wave a fond farewell to our old friend. After that’s finished, and you’re done sobbing, open up a shiny new PowerShell Core 7 host in Admin mode. While you’re in there opening up in Admin mode, go ahead and add PowerShell 7 to your Start Menu and your Taskbar. You’ll thank me later.
Then issue this command:
Install-Module PnP.PowerShell –AllowPrerelease
You’re all done. Since PnP.PowerShell is not an official release yet you’ll need the scary sounding –AllowPrerelease parameter. After January of 2021 you won’t need that anymore.
I mentioned earlier that one of the reasons for changing modules was a change in authentication. I don’t want to get too deep into it in this blog post but I do want to mention that you should run Register-PnPManagementShellAccess to set up a an Azure Application Registration. This is the magic that allows the new PnP PowerShell access to all of the applications in Office 365. You may have seen this referred to as Graph API. I’ll dig into it later, but for now all you need to know is that a tenant admin needs to run the PnP.PowerShell version of Register-PnPManagementShellAccess once in your tenant. After that is in place you should be able to use Connect-PnPOnline and get connected.
tk
Earlier this month, Alex Weinert, the Director of Identity Security at a little company called Microsoft, published a blog post begging us to stop using SMS as the second factor for MFA. I’m an MFA kind of guy, I live the MFA life style, and I’m on board with that. As the majority of the accounts I use can use the Microsoft Authenticator app, that’s what I use the most. Other companies, like Google, also have authenticator apps. I’m sure they’re fine and well loved by their friends and families. 
Over the last few months I’ve moved most, if not all, of the apps and sites I can to MFA using the Microsoft Authenticator app and I’ve picked up a few tricks along the way. I thought I’d blog a few of them in case they help anyone else. Keep in mind this blog post was written in November of 2020 and the version of the app I’m using is 6.2010.7266 on Android.
My first tip for using Microsoft Authenticator is not about using Microsoft Authenticator at all. It’s about using a Windows 10 feature called “Your Phone.” This feature, along with an app running on your phone, allow you to interact with your Android phone on your Windows 10 machine, or machines. I initially started using it to send text messages but it can do so much more. For instance, you can run phone apps on your PC, via screen sharing. One of those apps can be your friend and mine, Microsoft Authenticator. Since web sites (like Microsoft 365) and other services like VPN use codes generated from Microsoft Authenticator it is handy to have quick access to it on your PC. Here’s what it looks like:
That saves you fumbling with the UI on your phone, but you’d still need to look at your phone to get the code.
For this to really be helpful you also need to change a setting in the app to allow its screen to be captured. Go into Settings and enable Screen Capture:
If you don’t, you’ll see this on your PC when you open Microsoft Authenticator:
When I need to log into my GitHub account I fire up Your Phone on my computer, switch to the Microsoft Authenticator app and type the secret 6 characters in. Now I’m logged into GitHub and ready to cause some trouble.
By default, when you open Microsoft Authenticator you’re greeted with a list of all of the accounts you’ve registered and you select the one you want to log in to. Authenticator takes you to a screen with the one-time passcode for that account. But all that clicking is sooo much work. I take advantage of the “Show Codes” option, like below.
That shows me all the codes for the accounts that support it. You can see how it looks in the first screenshot. Combining these two techniques my MFA process went from:
To this:
Wait, copy and paste the passcode? How’s that again?
Once you have that set up you can actually copy the passcode from your phone in Windows and paste it into whatever web page or app is asking for it. To take advantage of this magic you need to enable copy and paste in the Your Phone app on Windows 10.
You might have to close the Authenticator app both on your phone and your PC for that to take effect. When you have it working, it’s a thing of beauty. Just copy the passcode with your mouse like you would any other application. There’s no visual indication that it’s copying, but trust me, it is. Go ahead, paste it into Notepad and see for yourself. Cool, huh?
I recently heard a sad tale of woe from a friend of mine that uses Microsoft Authenticator for all of his MFA needs. Something went wonky on his phone and he lost a bunch of the account settings. He had to go through a lot of work to get it all set back up. That’s when he and I both noticed the handy Backup functionality. Like all the other fun we’ve looked at it’s in the Settings page of the app. You can read all about it on this Docs Page. But the basic idea is, turn it on. Future you will appreciate your consideration and foresight.
There are a few other fun Authenticator tricks, but these are my favorites. Are you using Authenticator and have tips to share? Put them in the comments below.
tk
