Find a specific offer on a webpage using powershell

Question

I need to use powershell to resolve IP addresses through whois. My company is filtering requests for port 43 and WHOIS, so the workaround I should use here is to ask powershell to use a website, for example https://who.is , read the http stream and find the organization name that matches the IP address.

So far I have managed to load the webpage in powershell (an example here from WHOIS at yahoo.com), which is https://who.is/whois-ip/ip-address/206.190.36.45

So here is my snippet:

$url=Invoke-WebRequest https://who.is/whois-ip/ip-address/206.190.36.45 

now if i do:

 $url.gettype() IsPublic IsSerial Name BaseType -------- -------- ---- -------- True False HtmlWebResponseObject Microsoft.PowerShell.Commands.WebResponseObject 

I see that this object has several properties:

 Name MemberType Definition ---- ---------- ---------- Equals Method bool Equals(System.Object obj) GetHashCode Method int GetHashCode() GetType Method type GetType() ToString Method string ToString() AllElements Property Microsoft.PowerShell.Commands.WebCmdletElementCollection AllElements {get;} BaseResponse Property System.Net.WebResponse BaseResponse {get;set;} Content Property string Content {get;} Forms Property Microsoft.PowerShell.Commands.FormObjectCollection Forms {get;} Headers Property System.Collections.Generic.Dictionary[string,string] Headers {get;} Images Property Microsoft.PowerShell.Commands.WebCmdletElementCollection Images {get;} InputFields Property Microsoft.PowerShell.Commands.WebCmdletElementCollection InputFields {get;} Links Property Microsoft.PowerShell.Commands.WebCmdletElementCollection Links {get;} ParsedHtml Property mshtml.IHTMLDocument2 ParsedHtml {get;} RawContent Property string RawContent {get;} RawContentLength Property long RawContentLength {get;} RawContentStream Property System.IO.MemoryStream RawContentStream {get;} Scripts Property Microsoft.PowerShell.Commands.WebCmdletElementCollection Scripts {get;} StatusCode Property int StatusCode {get;} StatusDescription Property string StatusDescription {get;} 

but every time I try to execute commands like

 $url.ToString() | select-string "OrgName" 

Powershell returns all HTML code because it interprets the text string as a whole. I found a workaround that downloads the output to a file and then reads the file through the object (so each line is an array element), but I have hundreds of IP addresses to check that it is not very optimal to create the file all the time.

I would like to know how I can read the contents of the webpage https://who.is/whois-ip/ip-address/206.190.36.45 and get a line that says: OrgName: Yahoo! Broadcast Services, Inc.

and only this line.

Many thanks for your help!:)

Answer 1

There are probably better ways to figure this out, but you were on the right track with your current logic.

 $web = Invoke-WebRequest https://who.is/whois-ip/ip-address/206.190.36.45 $web.tostring() -split "[`r`n]" | select-string "OrgName" 

Select-String returned a match since it used to be one long string. Using -split , we can break it down just to get the expected return.

 OrgName: Yahoo! Broadcast Services, Inc. 

Some string manipulations will then get a cleaner answer. Again, many ways to approach this also

 (($web.tostring() -split "[`r`n]" | select-string "OrgName" | Select -First 1) -split ":")[1].Trim() 

I used Select -First 1 , since Select-String could return multiple objects. This will simply ensure that we work with 1 when manipulating the string. The string is simply split into a colon and trimmed to remove the remaining spaces.

Since you are pulling HTML data, we can also go through these properties to get more specific results. The purpose of this was to get 1RedOne's answer.

 $web = Invoke-WebRequest https://who.is/whois-ip/ip-address/206.190.36.45 $data = $web.AllElements | Where{$_.TagName -eq "Pre"} | Select-Object -Expand InnerText $whois = ($data -split "`r`n`r`n" | select -index 1) -replace ":\s","=" | ConvertFrom-StringData $whois.OrgName 

All this data is stored in the text of the PRE tag in this example. What I'm doing is dividing the data into its sections (sections are defined by empty lines separating them. I'm looking for consecutive lines of a new line). The second data group contains the name org. Store this in a variable and pull OrgName as a property: $whois.OrgName . Here is what $whois looks like

 Name Value ---- ----- Updated 2013-04-02 City Sunnyvale Address 701 First Ave OrgName Yahoo! Broadcast Services, Inc. StateProv CA Country US Ref http://whois.arin.net/rest/org/YAHO PostalCode 94089 RegDate 1999-11-17 OrgId YAHO 

Answer 2

it is very easy to use whois , this is the application for microsoft put app in System32 or windir, and in powershell it uses the whois command and then get-string gets "orgname" like this

 PS C:\> whois.exe -v 206.190.36.45 | Select-String "Registrant Organization" Registrant Organization: Yahoo! Inc. 

I recommend this application because it has more information for your work.

Answer 3

Here you go, the way to do this is to actually do an Invoke-WebRequest . If we look at some properties of the object that we get from Invoke-WebRequest, we can see that PowerShell has already parsed some of the HTML and text for us.

All we need to do is highlight some of the values ​​that we would like to work with. For example, looking at the ParsedText field, we will see these results.

These fields begin approximately on line 30 or so. In my approach to solving this problem, we know that we will find good data like this in the middle of the page, so if we could clear the values ​​from these lines, we would be on the way to working with the data. The code to execute this first part is as follows:

 $url = "https://who.is/whois-ip/ip-address/$ipaddress" $Results = Invoke-WebRequest $url $ParsedResults = $Results.ParsedHtml.body.outerText.Split("`n")[30..50] 

PowerShell now has some very powerful commands for importing and converting data into various formats. For example, if we could replace the colon character “:” with the equal sign “=”, we could send all the clutter to ConverFrom-StringData and have rich PowerShell objects to work with. It turns out that we can easily do this using the universal -Replace operator, like this

 $Results.ParsedHtml.body.outerText.Split("`n")[30..50] -replace ":","=" 

I decided that you would want to do this again in the future, so I took all this and made five functions with five lines for you. Drop it into your profile and enjoy.

So, the finished result is as follows:

 Function Get-WhoIsData { param($ipaddress='206.190.36.45') $url = "https://who.is/whois-ip/ip-address/$ipaddress" $Results = Invoke-WebRequest $url $ParsedResults = $Results.ParsedHtml.body.outerText.Split("`n")[30..50] -replace ":","=" | ConvertFrom-StringData $ParsedResults } 

and its use works as follows:

 PS C:\windows\system32> Get-WhoIsData -ipaddress 206.190.36.45 Name Value ---- ----- NetRange 206.190.32.0 - 206.190.63.255 CIDR 206.190.32.0/19 NetName NETBLK1-YAHOOBS NetHandle NET-206-190-32-0-1 Parent NET206 (NET-206-0-0-0-0) NetType Direct Allocation OriginAS Organization Yahoo! Broadcast Services, Inc. (YAHO) RegDate 1995-12-15 Updated 2012-03-02 Ref http=//whois.arin.net/rest/net/NET-206-190-32-0-1 OrgName Yahoo! Broadcast Services, Inc. OrgId YAHO Address 701 First Ave City Sunnyvale StateProv CA PostalCode 94089 

You can then select any of the properties you want using the usual Select-Object or Where-Object commands. For example, to pull only the orgName property, you should use the following command:

 (Get-WhoIsData).OrgName >Yahoo! Broadcast Services, Inc.