A source code analyzer built for surfacing features of interest and other characteristics to answer the question 'What's in the code?' quickly using static analysis with a json based rules engine. Ideal for scanning components before use or detecting feature level changes.

Overview

Introduction

CodeQL Nuget Nuget

Microsoft Application Inspector is a software source code characterization tool that helps identify coding features of first or third party software components based on well-known library/API calls and is helpful in security and non-security use cases. It uses hundreds of rules and regex patterns to surface interesting characteristics of source code to aid in determining what the software is or what it does from what file operations it uses, encryption, shell operations, cloud API's, frameworks and more and has received industry attention as a new and valuable contribution to OSS on ZDNet, SecurityWeek, CSOOnline, Linux.com/news, HelpNetSecurity, Twitter and more and was first featured on Microsoft.com.

Application Inspector is different from traditional static analysis tools in that it doesn't attempt to identify "good" or "bad" patterns; it simply reports what it finds against a set of over 400 rule patterns for feature detection including features that impact security such as the use of cryptography and more. This can be extremely helpful in reducing the time needed to determine what Open Source or other components do by examining the source directly rather than trusting to limited documentation or recommendations.

The tool supports scanning various programming languages including C, C++, C#, Java, JavaScript, HTML, Python, Objective-C, Go, Ruby, PowerShell and more and can scan projects with mixed language files. It supports generating results in HTML, JSON and text output formats with the default being an HTML report similar to the one shown here.

AppInspector-Features

Be sure to see our complete project wiki page https://Github.com/Microsoft/ApplicationInspector/wiki for additional information and help.

Goals

Microsoft Application Inspector helps you in securing your applications from start to deployment.

Design Choices - Enables you to choose which components meet your needs with a smaller footprint of unnecessary or unknowns features for keeping your application attack surface smaller as well as help to verify expected ones i.e. industry standard crypto only.

Identifying Feature Deltas - Detects changes between component versions which can be critical for detecting injection of backdoors.

Automating Security Compliance Checks - Use to identify components with features that require additional security scrutiny, approval or SDL compliance as part of your build pipeline or create a repository of metadata regarding all of your enterprise application.

Contribute

We have a strong default starting base of Rules for feature detection. But there are many feature identification patterns yet to be defined and we invite you to submit ideas on what you want to see or take a crack at defining a few. This is a chance to literally impact the open source ecosystem helping provide a tool that everyone can use. See the Rules section of the wiki for more.

Official Releases

Application Inspector is in GENERAL AUDIENCE release status. Your feedback is important to us. If you're interested in contributing, please review the CONTRIBUTING.md.

Application Inspector is availble as a command line tool or NuGet package and is supported on Windows, Linux, or MacOS.

Platform specific binaries of the ApplicationInspector CLI are available on our GitHub releases page.

The C# library is available on NuGet as Microsoft.CST.ApplicationInspector.Commands.

The .NET Global Tool is available on NuGet as Microsoft.CST.ApplicationInspector.CLI.

If you use the .NET Core version, you will need to have .NET Core 3.1 or later installed. See the JustRunIt.md or Build.md files for more.

Basic CLI Usage

> dotnet ApplicationInspector.CLI.dll or on *Windows* simply ApplicationInspector.CLI.exe <command> <options>

Microsoft Application Inspector

(c) Microsoft Corporation. All rights reserved

ERROR(S):
  No verb selected.

  analyze        Inspect source directory/file/compressed file (.tgz|zip) against defined characteristics

  tagdiff        Compares unique tag values between two source paths

  tagtest        Test (T/F) for presence of custom rule set in source

  exporttags     Export unique rule tags to view what code features may be detected

  verifyrules    Verify custom rules syntax is valid

  packrules      Combine multiple rule files into one file for ease in distribution

  help           Display more information on a specific command.

  version        Display version information.

Examples:

Command Help

  Usage: dotnet ApplicationInspector.CLI.dll [arguments] [options]

  dotnet ApplicationInspector.CLI.dll <no args> -description of available commands
  dotnet ApplicationInspector.CLI.dll <command> <no args> -arg options description for a given command

For additional help on use of the console interface see CLI Usage.

For help using the NuGet package see NuGet Support

# Build Instructions

See [build.md](https://github.com/microsoft/ApplicationInspector/blob/main/BUILD.md)

Issues
  • Performance Improvements - Parallelize Analysis, Extraction, Compile Regexes

    Performance Improvements - Parallelize Analysis, Extraction, Compile Regexes

    Rewrite AnalyzeCommand to process files in parallel. Made the stats in Metadata threadsafe. Compile regexes for performance. Fix #205 Update multiextractor to work in parallel when possible. * Update multiextractor to catch exceptions. *

    • I also contributed better implementations of these upstream to multiextractor in OSSGadget which we can incorporate once that is published as a nuget. microsoft/OSSGadget#47
    enhancement 
    opened by gfs 24
  • Randomly, Application Inspector analysis never ends.

    Randomly, Application Inspector analysis never ends.

    Describe the bug We have experienced sometimes, apparently random, that applications inspector analysis stucks the machine (we think maybe it is a memory problem) and never ends. We always pass to the cli the argument --process-timeout, but in this cases the execution never ends. What do you think it's going on? Thanks for the help.

    Expected behavior The execution of an analysis always respects the process-timeout param time.

    Additional context It happens when execute the application inspector inside a github or a bamboo workflow.

    bug 
    opened by ismadirolas 19
  • 1.2 dev branch

    1.2 dev branch

    Unit tests Module • 189 passing tests added covering both NuGet and CLI entry points under separate namespaces/classes • Improved overall code quality and reporting of invalid args/combinations

    Overall Code Structure • Refactoring+rework for technical debt getting to 1.0 eliminating unused code, renaming, moving to isolate to right project • *Object results-based support for NuGet callers rather than json/text strings • *New support for First/Best match options for results (best match elevates higher confidence value matches) • Separation of CLI vs Operation argument options • All file output format and writters moved to CLI project • Cleaner isolation of functionality and readability • Text and Json output support added for all commands (previously limited to text for all but analyze) • Better use of inheritance to organize common arguments and override • Better use of polymorphism for checking arguments

    Resolves #191

    opened by guyacosta 15
  • Analyze Error No Compressed File

    Analyze Error No Compressed File

    Describe the bug D:\ApplicationInspector_1.1.5>ApplicationInspector.CLI.exe analyze -s D:\emailba
    Microsoft Application Inspector 1.1.5+ed45aa547b Analyze command running Decompressing files... Problem while decompressing D:\emailba\WebContent\WEB-INF\lib\aws-java-sdk-core-1.11.543.jar: Unzip and try running on uncompressed files. A runtime error has occured. Please see log file for more information.

    To Reproduce D:\ApplicationInspector_1.1.5>ApplicationInspector.CLI.exe analyze -s D:\emailba\

    Expected behavior A clear and concise description of what you expected to happen.

    Screenshots If applicable, add screenshots to help explain your problem.

    Desktop (please complete the following information):

    • OS: Windows
    • Version 10
    opened by imrzegarra 13
  • The analysis process getting stuck at 50%

    The analysis process getting stuck at 50%

    Describe the bug I am running the App Inspector Analysis on a large repository with multiple languages, frameworks and components. The process runs fine till 50% but then it freezes there forever. And also, the appinspector.log.txt file is empty, hence have no idea what is going wrong. I tried with some moderate or small sized repository, and it works fine.

    To Reproduce Steps to reproduce the behavior:

    1. Clone a significantly large repository
    2. Run ApplicationInspector.CLI.exe analyze -s <Repo Local Path>

    Expected behavior Firstly, the analysis should be complete. If not, then some error should be posted on command line where I am running the same, or logs should be written.

    Desktop (please complete the following information):

    • OS: Windows 10
    • Browser: Edge Chromium
    opened by tanmayghosh2507 11
  • Limit the number of matches reported for each tag.

    Limit the number of matches reported for each tag.

    Is your feature request related to a problem? Please describe. With some rules, we can get a large number of matches, but we only need to have two or three examples of that matches, and in big projects, the appinspector generates a very large .json output with all the matches.

    Describe the solution you'd like Add a cli parameter like "--max-matches" that admits a number to configure a maximum number of matches reported for each tag.

    Additional context I see in documentation here that talks about 'match-depth' and 'allow-duplicates' arguments, with a similar idea, but I have seen they were deleted in this release

    enhancement 
    opened by ismadirolas 10
  • Excessively large generated report files

    Excessively large generated report files

    Describe the bug Since you updated to .NET 5, the generated report files (JSON & HTML) became very very large.

    To Reproduce Steps to reproduce the behavior:

    1. Retrieve https://github.com/OrchardCMS/OrchardCore as a test application (git clone)
    2. Analyze the repo dotnet ApplicationInspector.CLI.dll analyze -s "/folder/to/OrchardCore" -v Info -x high -c high,medium -k lib,.vs,.git,.idea
    3. You will obtain one 611MB-large HTML file and one 564MB large JSON file

    Expected behavior With AppInspector 1.2.89 (version I keep using) the same files had a size of around 3 MB.

    Desktop

    • OS: macOS BigSur 11.3.1 (20E241)
    • Version: App Inspector 1.3.17 dotnet_core distribution (leveraging dotnet 5.0.203)
    bug 
    opened by Maarc 10
  • Tagcounters.json no longer being picked up for custom reporting?

    Tagcounters.json no longer being picked up for custom reporting?

    Description Following the documentation here, I wanted to see detail of Classes and Functions in the reports, so I created this tagcounters.json file in the preferences directory:

    [
        {
          "tag": "Metric.Code.Class.Defined",
          "displayName": "Classes",
          "includeAsMatch": true
        },
        {
          "tag": "Metric.Code.Function.Defined",
          "displayName": "Functions",
          "includeAsMatch": true
        },
        {
          "tag": "Metric.Code.Exception.Caught",
          "displayName": "Exceptions",
          "includeAsMatch": true
        },
        {
          "tag": "Metric.Code.HTMLForm.Defined",
          "displayName": "Forms",
          "includeAsMatch": true
        },
        {
          "tag": "Dependency.SourceInclude",
          "displayName": "Dependencies",
          "includeAsMatch": false
        },
        {
          "tag": "Miscellaneous.CodeHygiene.Comment.Todo",
          "displayName": "Comments",
          "includeAsMatch": false
        },
        {
          "tag": "Miscellaneous.CodeHygiene.Comment.Fix",
          "displayName": "Fix-comments",
          "includeAsMatch": true
        },
        {
          "tag": "Miscellaneous.CodeHygiene.Comment.Suspicious",
          "displayName": "Suspicious-comments",
          "includeAsMatch": true
        },
        {
          "tag": "Metric.Code.Logging.Call",
          "displayName": "Logging Calls",
          "includeAsMatch": false
        },
        {
          "tag": "Metric.Code.URL",
          "displayName": "URL's",
          "includeAsMatch": false
        }
      ]
    

    and then added this to tagreportgroups.json:

          {
            "title": "Code Metrics",
            "dataRef": "CodeMetrics",
            "patterns": [
              {
                "searchPattern": "Metric.Code.Logging.Call",
                "displayName": "Code Logging",
                "detectedIcon": "fab fa-dev"
              },
              {
                "searchPattern": "Metric.Code.Class.Defined",
                "displayName": "Class Defined",
                "detectedIcon": "fab fa-dev"
              },
              {
                "searchPattern": "Metric.Code.Exception.Caught",
                "displayName": "Exception Caught",
                "detectedIcon": "fab fa-dev"
              },
              {
                "searchPattern": "Metric.Code.URL",
                "displayName": "Code URL",
                "detectedIcon": "fab fa-dev"
              },
              {
                "searchPattern": "Metric.Code.Function.Defined",
                "displayName": "Function Defined",
                "detectedIcon": "fab fa-dev"
              }                                       
            ]
          } 
    

    Running an analysis results in the custom Code Metrics section being added to the report, but no results for Class Defined or Function Defined are shown.

    Furthermore on the Summary - Tag Counters screen, it displays:

    • Metric.Code.Function.Defined: Count: 119 Include as Match: false
    • Metric.Code.Class.Defined - Count: 18 Include as Match: false

    So it appears to be picking up the customisations in tagreportgroups.json, but not in tagcounters.json.

    Screenshots Screenshot 2020-09-15 at 17 06 17 Screenshot 2020-09-15 at 17 05 56

    Desktop

    • macOS Catalina
    • Google Chrome v85
    • Application Inspector v1.2.63
    bug 
    opened by marc-patterson 10
  • Icons for features that are not present should be grayed out

    Icons for features that are not present should be grayed out

    Describe the bug Icons are not being grayed out when they are not present.

    To Reproduce Get patch from #265 first to ensure resources are loaded.

    Expected behavior Icons for properties that don't apply should be gray not blue.

    Screenshots image

    bug 
    opened by gfs 10
  • NuGet Command Pkg Install Doesn't Copy Default Rules to Local App

    NuGet Command Pkg Install Doesn't Copy Default Rules to Local App

    Describe the bug After installing the Microsoft.CST.ApplicationInspector.Command pkg and running the containing app, the calls fail due to the default rules folder missing from where the dll is copied to i.e. the containing app \bin\debug or \bin\release folders causing an exception. A default or custom rules path is expected. The caller wanting to use the default ruleset would have to copy them manually from the nuget install location locally into their local app bin folder

    To Reproduce Steps to reproduce the behavior:

    1. Download the pkg from Nuget via Visual Studio into an app
    2. Create a AnalyzeCommandOption object
    3. Create an AnalyzeCommand object
    4. Call GetResults method on the command object. See the exception from step 3.

    Expected behavior User does not have to manually copy over the Rules folder into their bin folders for use.

    Screenshots If applicable, add screenshots to help explain your problem.

    Desktop (please complete the following information):

    • OS: [e.g. iOS]
    • Browser [e.g. chrome, safari]
    • Version [e.g. 22]

    Smartphone (please complete the following information):

    • Device: [e.g. iPhone6]
    • OS: [e.g. iOS8.1]
    • Browser [e.g. stock browser, safari]
    • Version [e.g. 22]

    Additional context Add any other context about the problem here.

    opened by guyacosta 10
  • TextContainer GetLocation Throws Exception Occasionally

    TextContainer GetLocation Throws Exception Occasionally

    Describe the bug On occasion there the RulesEngine throws an exception for certain files when trying to determine match locations from the TextContainer.cs file indicated. The issue appears to be a conditional that allows there to be an uneven number of items between the LineEnds and LineStarts lists culminating in a failure in the GetLocation method on line 218. The solution appears to be correcting the boundary limit in the constructor to include '<=' on line 35.

    To Reproduce Source code to reproduce will vary...was primarily python if that helps. The behavior interestingly was seen while using the NuGet model but not the commandLine one -not sure why but time is limited to investigate but it is also not consistent for reasons unknown i.e. sometimes it doesn't throw an exception.

    Expected behavior Complete analysis for source without throwing an exception

    Code Section File: ..\ApplicationInspector-main\RulesEngine\TextContainer.cs Class & Method: TextContainer : TextContainer Line: 35

    while (pos > -1)
    {
               **if (pos > 0 && pos + 1 < FullContent.Length)**
               {
                        LineStarts.Add(pos + 1);
               }
               pos = FullContent.IndexOf('\n', pos + 1);
         
               if (LineEnds.Count < LineStarts.Count)
               {
                    LineEnds.Add(FullContent.Length - 1);
               }
    }
    

    and in method GetLocation line 218 where the indexing exception gets thrown

    result.Column = index - LineStarts[i];
    

    Screenshots appinsbug-txtcontainer

    Additional context Add any other context about the problem here.

    bug 
    opened by guyacosta 9
  • Empty Output.html

    Empty Output.html

    I dont unerstand why sometimes when I scan my python project (as folder with my main.py, all libraries and etc) it gives me an empty output.html and "0 matches, X files skipped out of X files". It seems that it doesnt think my code needs to be analized but I dont know why

    bug 
    opened by ussenkula 1
  • Improve the logging output when hitting an expected error creating Analyze Command

    Improve the logging output when hitting an expected error creating Analyze Command

    When creating an analyze command the "ConfigSourceToScan" and "ConfigRules" methods are called from the constructor. These methods may throw exceptions, and those exceptions are printed to the console with the stack trace. This level of detail isn't needed (perhaps the stack trace should show up on the debug level) but the Error level should be less verbose and a human readable error message. This ctor is called in many places (IDE is indicating over 50), so its a medium effort to refactor all those calls to handle the exception more gracefully.

    Alternately consider a different calling pattern that obviates the need for throwing these exceptions in the first place.

    For example, we currently throw an exception from the AnalyzeCommand if the source to scan contains no files. This seems like a worthwhile check, but likely doesn't need to be in the analyze command, and probably should just be part of the CLI experience not the lib.

    enhancement 
    opened by gfs 0
  • Extension to Rule Object to integrate test cases in the rule file directly

    Extension to Rule Object to integrate test cases in the rule file directly

    Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

    The current rule set doesn't have a test suite.

    Describe the solution you'd like The rule object should have additional fields added for test cases that should match and should not match the rule.

    There should also be a test that runs all of those checks when validating the rules.

    Describe alternatives you've considered Test cases could be distinct from the rule file, but this makes tracking which applies to which more complex.

    enhancement 
    opened by gfs 0
  • AI HTML report requires some CDN assets

    AI HTML report requires some CDN assets

    After Running scan with CLI command , Report created successfully. Report - Feature Group Links are not been displayed , Missing image and link on the list. Reproduced on Edge and Chrome.

    Thanks

    bug 
    opened by Eldadc 13
  • Automatically populate repository information using LibGit2Sharp

    Automatically populate repository information using LibGit2Sharp

    Is your feature request related to a problem? Please describe. When outputting sarif you can specify the repository uri and commit. We should be able to programatically fetch that automatically.

    See LibGit2Sharp

    enhancement 
    opened by gfs 0
Releases(v1.5.13)
Owner
Microsoft
Open source projects and samples from Microsoft
Microsoft
🌘🦊 DalFox(Finder Of XSS) / Parameter Analysis and XSS Scanning tool based on golang

Finder Of XSS, and Dal(달) is the Korean pronunciation of moon. What is DalFox ?? ?? DalFox is a fast, powerful parameter analysis and XSS scanner, bas

HAHWUL 1.9k Jun 30, 2022
Lightweight static analysis for many languages. Find bug variants with patterns that look like source code.

Lightweight static analysis for many languages. Find bugs and enforce code standards. Semgrep is a fast, open-source, static analysis tool that finds

r2c 6.7k Jun 29, 2022
Static binary analysis tool to compute shared strings references between binaries and output in JSON, YAML and YARA

StrTwins StrTwins is a binary analysis tool, powered by radare, that is capable to find shared code string references between executables and output i

Anderson 2 May 3, 2022
PHP functions implementation to Golang. This package is for the Go beginners who have developed PHP code before. You can use PHP like functions in your app, module etc. when you add this module to your project.

PHP Functions for Golang - phpfuncs PHP functions implementation to Golang. This package is for the Go beginners who have developed PHP code before. Y

Serkan Algur 48 Jun 4, 2022
EarlyBird is a sensitive data detection tool capable of scanning source code repositories for clear text password violations, PII, outdated cryptography methods, key files and more.

EarlyBird is a sensitive data detection tool capable of scanning source code repositories for clear text password violations, PII, outdated cryptograp

American Express 492 Jun 16, 2022
Nuclei is a fast tool for configurable targeted vulnerability scanning based on templates offering massive extensibility and ease of use.

Fast and customisable vulnerability scanner based on simple YAML based DSL. How • Install • For Security Engineers • For Developers • Documentation •

ProjectDiscovery 8.7k Jun 26, 2022
Small utility package for stealing tokens from other processes and using them in current threads, or duplicating them and starting other processes

getsystem small utility for impersonating a user in the current thread or starting a new process with a duplicated token. must already be in a high in

Alex Flores 10 May 29, 2022
Monitor your certificates and get notified before they expire.

Sifaka Sifaka is a tool to monitor your x509 certificates or simply websites certificates expirey date. If your organisation is handling some certs re

Jakub 6 Apr 16, 2022
mesh-kridik is an open-source security scanner that performs various security checks on a Kubernetes cluster with istio service mesh and is leveraged by OPA (Open Policy Agent) to enforce security rules.

mesh-kridik Enhance your Kubernetes service mesh security !! mesh-kridik is an open-source security scanner that performs various security checks on a

chenk 22 May 18, 2022
A detector for the Trojan Source and other unicode-based vulnerabilities.

Trojan Source Detector This application detects Trojan Source attacks in source code. It can be used as part of the CI system to make sure there are n

Have you debugged.it? 5 Jan 6, 2022
ARP spoofing tool based on go language, supports LAN host scanning, ARP poisoning, man-in-the-middle attack, sensitive information sniffing, HTTP packet sniffing

[ARP Spoofing] [Usage] Commands: clear clear the screen cut 通过ARP欺骗切断局域网内某台主机的网络 exit exit the program help display help hosts 主机管理功能 loot 查看嗅探到的敏感信息

Re 41 Jun 14, 2022
WIP. Converts Azure Container Scan Action output to SARIF, for an easier integration with GitHub Code Scanning

container-scan-to-sarif container-scan-to-sarif converts Azure Container Scan Action output to Static Analysis Results Interchange Format (SARIF), for

Armel Soro 2 Jan 25, 2022
A FreeSWITCH specific scanning and exploitation toolkit for CVE-2021-37624 and CVE-2021-41157.

PewSWITCH A FreeSWITCH specific scanning and exploitation toolkit for CVE-2021-37624 and CVE-2021-41157. Related blog: https://0xinfection.github.io/p

Pinaki 23 Jun 23, 2022
The dynamic infrastructure framework for everybody! Distribute the workload of many different scanning tools with ease, including nmap, ffuf, masscan, nuclei, meg and many more!

Axiom is a dynamic infrastructure framework to efficiently work with multi-cloud environments, build and deploy repeatable infrastructure focussed on

pry0cc 2.7k Jun 24, 2022
Naabu - a port scanning tool written in Go that allows you to enumerate valid ports for hosts in a fast and reliable manner

Naabu is a port scanning tool written in Go that allows you to enumerate valid ports for hosts in a fast and reliable manner. It is a really simple tool that does fast SYN/CONNECT scans on the host/list of hosts and lists all ports that return a reply.

null 0 Jan 2, 2022
Portmantool - Port scanning and monitoring tool

portmantool Port scanning and monitoring tool Components runner while true do r

Thomann Bits & Beats 0 Feb 14, 2022
Gryffin is a large scale web security scanning platform.

Gryffin (beta) Gryffin is a large scale web security scanning platform. It is not yet another scanner. It was written to solve two specific problems w

Yahoo 2.1k Jun 29, 2022
A fully self-contained Nmap like parallel port scanning module in pure Golang that supports SYN-ACK (Silent Scans)

gomap What is gomap? Gomap is a fully self-contained nmap like module for Golang. Unlike other projects which provide nmap C bindings or rely on other

jtimperio 68 Jun 27, 2022
A Large killer focused on intranet scanning

FscanX 其实FscanX的灵感来源于fscan和LodanGo这两个开源项目,首先不得不说fscan和LadonGo两个都是非常优秀的内网扫描器。并且其独自的特色也让其在内网扫描器领域独占鳌头。其中LadonGo的插件式让其在扫描时更加专注,而fscan的傻瓜式则让其对内网的信息搜集更加高效。

SaiRson 203 Dec 31, 2021