A source code analyzer built for surfacing features of interest and other characteristics to answer the question 'What's in the code?' quickly using static analysis with a json based rules engine. Ideal for scanning components before use or detecting feature level changes.

Overview

Introduction

CodeQL Nuget Nuget

Microsoft Application Inspector is a software source code characterization tool that helps identify coding features of first or third party software components based on well-known library/API calls and is helpful in security and non-security use cases. It uses hundreds of rules and regex patterns to surface interesting characteristics of source code to aid in determining what the software is or what it does from what file operations it uses, encryption, shell operations, cloud API's, frameworks and more and has received industry attention as a new and valuable contribution to OSS on ZDNet, SecurityWeek, CSOOnline, Linux.com/news, HelpNetSecurity, Twitter and more and was first featured on Microsoft.com.

Application Inspector is different from traditional static analysis tools in that it doesn't attempt to identify "good" or "bad" patterns; it simply reports what it finds against a set of over 400 rule patterns for feature detection including features that impact security such as the use of cryptography and more. This can be extremely helpful in reducing the time needed to determine what Open Source or other components do by examining the source directly rather than trusting to limited documentation or recommendations.

The tool supports scanning various programming languages including C, C++, C#, Java, JavaScript, HTML, Python, Objective-C, Go, Ruby, PowerShell and more and can scan projects with mixed language files. It supports generating results in HTML, JSON and text output formats with the default being an HTML report similar to the one shown here.

AppInspector-Features

Be sure to see our complete project wiki page https://Github.com/Microsoft/ApplicationInspector/wiki for additional information and help.

Goals

Microsoft Application Inspector helps you in securing your applications from start to deployment.

Design Choices - Enables you to choose which components meet your needs with a smaller footprint of unnecessary or unknowns features for keeping your application attack surface smaller as well as help to verify expected ones i.e. industry standard crypto only.

Identifying Feature Deltas - Detects changes between component versions which can be critical for detecting injection of backdoors.

Automating Security Compliance Checks - Use to identify components with features that require additional security scrutiny, approval or SDL compliance as part of your build pipeline or create a repository of metadata regarding all of your enterprise application.

Contribute

We have a strong default starting base of Rules for feature detection. But there are many feature identification patterns yet to be defined and we invite you to submit ideas on what you want to see or take a crack at defining a few. This is a chance to literally impact the open source ecosystem helping provide a tool that everyone can use. See the Rules section of the wiki for more.

Official Releases

Application Inspector is in GENERAL AUDIENCE release status. Your feedback is important to us. If you're interested in contributing, please review the CONTRIBUTING.md.

Application Inspector is availble as a command line tool or NuGet package and is supported on Windows, Linux, or MacOS.

Platform specific binaries of the ApplicationInspector CLI are available on our GitHub releases page.

The C# library is available on NuGet as Microsoft.CST.ApplicationInspector.Commands.

The .NET Global Tool is available on NuGet as Microsoft.CST.ApplicationInspector.CLI.

If you use the .NET Core version, you will need to have .NET Core 3.1 or later installed. See the JustRunIt.md or Build.md files for more.

Basic CLI Usage

> dotnet ApplicationInspector.CLI.dll or on *Windows* simply ApplicationInspector.CLI.exe <command> <options>

Microsoft Application Inspector

(c) Microsoft Corporation. All rights reserved

ERROR(S):
  No verb selected.

  analyze        Inspect source directory/file/compressed file (.tgz|zip) against defined characteristics

  tagdiff        Compares unique tag values between two source paths

  tagtest        Test (T/F) for presence of custom rule set in source

  exporttags     Export unique rule tags to view what code features may be detected

  verifyrules    Verify custom rules syntax is valid

  packrules      Combine multiple rule files into one file for ease in distribution

  help           Display more information on a specific command.

  version        Display version information.

Examples:

Command Help

  Usage: dotnet ApplicationInspector.CLI.dll [arguments] [options]

  dotnet ApplicationInspector.CLI.dll <no args> -description of available commands
  dotnet ApplicationInspector.CLI.dll <command> <no args> -arg options description for a given command

For additional help on use of the console interface see CLI Usage.

For help using the NuGet package see NuGet Support

# Build Instructions

See [build.md](https://github.com/microsoft/ApplicationInspector/blob/main/BUILD.md)

Comments
  • Pattern type for structured data files (yaml, json...)

    Pattern type for structured data files (yaml, json...)

    Is your feature request related to a problem? Please describe. In order to be able to detect feature usage in structured configuration files (yaml, json, xml...) it would be very useful to be able to search for occurrences of certain entries regardless of the order of declaration in the file.

    An example would be:

    If we have a config.yaml file and we have the following configuration:

    app:
      myfeature:
         prop1: x
         prop2: x
         propN: x
         enabled: true
    

    If I want to detect with a rule if the 'myfeature' functionality is enabled I would not have a reliable solution at the moment. Detecting if app.my-feature.enabled = true with patterns, modiffiers or conditions does not cover all cases, e.g. if the declaration order is changed or the configuration is put in one line. I.e. a pattern such as:

    "pattern": "app:\s+myfeature:\s+enabled: *true".

    It would not cover the above example since the order of the enabled property can be changed as it is a structured data type.

    Describe the solution you'd like Ability to be able to perform patterns on structured data types.

    Describe alternatives you've considered

    • Implement new pattern types to support operations on structured data types (yaml, json...).
    • Provide an extension mechanism to be able to do file processing with parsing tools.
    enhancement feedback wanted 
    opened by valinha 33
  • Performance Improvements - Parallelize Analysis, Extraction, Compile Regexes

    Performance Improvements - Parallelize Analysis, Extraction, Compile Regexes

    Rewrite AnalyzeCommand to process files in parallel. Made the stats in Metadata threadsafe. Compile regexes for performance. Fix #205 Update multiextractor to work in parallel when possible. * Update multiextractor to catch exceptions. *

    • I also contributed better implementations of these upstream to multiextractor in OSSGadget which we can incorporate once that is published as a nuget. microsoft/OSSGadget#47
    enhancement 
    opened by gfs 24
  • Randomly, Application Inspector analysis never ends.

    Randomly, Application Inspector analysis never ends.

    Describe the bug We have experienced sometimes, apparently random, that applications inspector analysis stucks the machine (we think maybe it is a memory problem) and never ends. We always pass to the cli the argument --process-timeout, but in this cases the execution never ends. What do you think it's going on? Thanks for the help.

    Expected behavior The execution of an analysis always respects the process-timeout param time.

    Additional context It happens when execute the application inspector inside a github or a bamboo workflow.

    bug 
    opened by ismadirolas 19
  • 1.2 dev branch

    1.2 dev branch

    Unit tests Module • 189 passing tests added covering both NuGet and CLI entry points under separate namespaces/classes • Improved overall code quality and reporting of invalid args/combinations

    Overall Code Structure • Refactoring+rework for technical debt getting to 1.0 eliminating unused code, renaming, moving to isolate to right project • *Object results-based support for NuGet callers rather than json/text strings • *New support for First/Best match options for results (best match elevates higher confidence value matches) • Separation of CLI vs Operation argument options • All file output format and writters moved to CLI project • Cleaner isolation of functionality and readability • Text and Json output support added for all commands (previously limited to text for all but analyze) • Better use of inheritance to organize common arguments and override • Better use of polymorphism for checking arguments

    Resolves #191

    opened by guyacosta 15
  • Analyze Error No Compressed File

    Analyze Error No Compressed File

    Describe the bug D:\ApplicationInspector_1.1.5>ApplicationInspector.CLI.exe analyze -s D:\emailba
    Microsoft Application Inspector 1.1.5+ed45aa547b Analyze command running Decompressing files... Problem while decompressing D:\emailba\WebContent\WEB-INF\lib\aws-java-sdk-core-1.11.543.jar: Unzip and try running on uncompressed files. A runtime error has occured. Please see log file for more information.

    To Reproduce D:\ApplicationInspector_1.1.5>ApplicationInspector.CLI.exe analyze -s D:\emailba\

    Expected behavior A clear and concise description of what you expected to happen.

    Screenshots If applicable, add screenshots to help explain your problem.

    Desktop (please complete the following information):

    • OS: Windows
    • Version 10
    opened by imrzegarra 13
  • The analysis process getting stuck at 50%

    The analysis process getting stuck at 50%

    Describe the bug I am running the App Inspector Analysis on a large repository with multiple languages, frameworks and components. The process runs fine till 50% but then it freezes there forever. And also, the appinspector.log.txt file is empty, hence have no idea what is going wrong. I tried with some moderate or small sized repository, and it works fine.

    To Reproduce Steps to reproduce the behavior:

    1. Clone a significantly large repository
    2. Run ApplicationInspector.CLI.exe analyze -s <Repo Local Path>

    Expected behavior Firstly, the analysis should be complete. If not, then some error should be posted on command line where I am running the same, or logs should be written.

    Desktop (please complete the following information):

    • OS: Windows 10
    • Browser: Edge Chromium
    opened by tanmayghosh2507 11
  • Limit the number of matches reported for each tag.

    Limit the number of matches reported for each tag.

    Is your feature request related to a problem? Please describe. With some rules, we can get a large number of matches, but we only need to have two or three examples of that matches, and in big projects, the appinspector generates a very large .json output with all the matches.

    Describe the solution you'd like Add a cli parameter like "--max-matches" that admits a number to configure a maximum number of matches reported for each tag.

    Additional context I see in documentation here that talks about 'match-depth' and 'allow-duplicates' arguments, with a similar idea, but I have seen they were deleted in this release

    enhancement 
    opened by ismadirolas 10
  • Excessively large generated report files

    Excessively large generated report files

    Describe the bug Since you updated to .NET 5, the generated report files (JSON & HTML) became very very large.

    To Reproduce Steps to reproduce the behavior:

    1. Retrieve https://github.com/OrchardCMS/OrchardCore as a test application (git clone)
    2. Analyze the repo dotnet ApplicationInspector.CLI.dll analyze -s "/folder/to/OrchardCore" -v Info -x high -c high,medium -k lib,.vs,.git,.idea
    3. You will obtain one 611MB-large HTML file and one 564MB large JSON file

    Expected behavior With AppInspector 1.2.89 (version I keep using) the same files had a size of around 3 MB.

    Desktop

    • OS: macOS BigSur 11.3.1 (20E241)
    • Version: App Inspector 1.3.17 dotnet_core distribution (leveraging dotnet 5.0.203)
    bug 
    opened by Maarc 10
  • Tagcounters.json no longer being picked up for custom reporting?

    Tagcounters.json no longer being picked up for custom reporting?

    Description Following the documentation here, I wanted to see detail of Classes and Functions in the reports, so I created this tagcounters.json file in the preferences directory:

    [
        {
          "tag": "Metric.Code.Class.Defined",
          "displayName": "Classes",
          "includeAsMatch": true
        },
        {
          "tag": "Metric.Code.Function.Defined",
          "displayName": "Functions",
          "includeAsMatch": true
        },
        {
          "tag": "Metric.Code.Exception.Caught",
          "displayName": "Exceptions",
          "includeAsMatch": true
        },
        {
          "tag": "Metric.Code.HTMLForm.Defined",
          "displayName": "Forms",
          "includeAsMatch": true
        },
        {
          "tag": "Dependency.SourceInclude",
          "displayName": "Dependencies",
          "includeAsMatch": false
        },
        {
          "tag": "Miscellaneous.CodeHygiene.Comment.Todo",
          "displayName": "Comments",
          "includeAsMatch": false
        },
        {
          "tag": "Miscellaneous.CodeHygiene.Comment.Fix",
          "displayName": "Fix-comments",
          "includeAsMatch": true
        },
        {
          "tag": "Miscellaneous.CodeHygiene.Comment.Suspicious",
          "displayName": "Suspicious-comments",
          "includeAsMatch": true
        },
        {
          "tag": "Metric.Code.Logging.Call",
          "displayName": "Logging Calls",
          "includeAsMatch": false
        },
        {
          "tag": "Metric.Code.URL",
          "displayName": "URL's",
          "includeAsMatch": false
        }
      ]
    

    and then added this to tagreportgroups.json:

          {
            "title": "Code Metrics",
            "dataRef": "CodeMetrics",
            "patterns": [
              {
                "searchPattern": "Metric.Code.Logging.Call",
                "displayName": "Code Logging",
                "detectedIcon": "fab fa-dev"
              },
              {
                "searchPattern": "Metric.Code.Class.Defined",
                "displayName": "Class Defined",
                "detectedIcon": "fab fa-dev"
              },
              {
                "searchPattern": "Metric.Code.Exception.Caught",
                "displayName": "Exception Caught",
                "detectedIcon": "fab fa-dev"
              },
              {
                "searchPattern": "Metric.Code.URL",
                "displayName": "Code URL",
                "detectedIcon": "fab fa-dev"
              },
              {
                "searchPattern": "Metric.Code.Function.Defined",
                "displayName": "Function Defined",
                "detectedIcon": "fab fa-dev"
              }                                       
            ]
          } 
    

    Running an analysis results in the custom Code Metrics section being added to the report, but no results for Class Defined or Function Defined are shown.

    Furthermore on the Summary - Tag Counters screen, it displays:

    • Metric.Code.Function.Defined: Count: 119 Include as Match: false
    • Metric.Code.Class.Defined - Count: 18 Include as Match: false

    So it appears to be picking up the customisations in tagreportgroups.json, but not in tagcounters.json.

    Screenshots Screenshot 2020-09-15 at 17 06 17 Screenshot 2020-09-15 at 17 05 56

    Desktop

    • macOS Catalina
    • Google Chrome v85
    • Application Inspector v1.2.63
    bug 
    opened by marc-patterson 10
  • Icons for features that are not present should be grayed out

    Icons for features that are not present should be grayed out

    Describe the bug Icons are not being grayed out when they are not present.

    To Reproduce Get patch from #265 first to ensure resources are loaded.

    Expected behavior Icons for properties that don't apply should be gray not blue.

    Screenshots image

    bug 
    opened by gfs 10
  • NuGet Command Pkg Install Doesn't Copy Default Rules to Local App

    NuGet Command Pkg Install Doesn't Copy Default Rules to Local App

    Describe the bug After installing the Microsoft.CST.ApplicationInspector.Command pkg and running the containing app, the calls fail due to the default rules folder missing from where the dll is copied to i.e. the containing app \bin\debug or \bin\release folders causing an exception. A default or custom rules path is expected. The caller wanting to use the default ruleset would have to copy them manually from the nuget install location locally into their local app bin folder

    To Reproduce Steps to reproduce the behavior:

    1. Download the pkg from Nuget via Visual Studio into an app
    2. Create a AnalyzeCommandOption object
    3. Create an AnalyzeCommand object
    4. Call GetResults method on the command object. See the exception from step 3.

    Expected behavior User does not have to manually copy over the Rules folder into their bin folders for use.

    Screenshots If applicable, add screenshots to help explain your problem.

    Desktop (please complete the following information):

    • OS: [e.g. iOS]
    • Browser [e.g. chrome, safari]
    • Version [e.g. 22]

    Smartphone (please complete the following information):

    • Device: [e.g. iPhone6]
    • OS: [e.g. iOS8.1]
    • Browser [e.g. stock browser, safari]
    • Version [e.g. 22]

    Additional context Add any other context about the problem here.

    opened by guyacosta 10
  • Refactor to use System.Text.Json

    Refactor to use System.Text.Json

    Is your feature request related to a problem? Please describe. App inspector deserializes using Newtonsoft.Json, accordingly it ignores System.Text.Json attributes. If you were to extend an AI rule with more attributes and then call AddString, even if you the T is specified the deserialization may not be correct if your json properties dont match the attribute names or if they have special handling.

    Any downstream users who implement extensions to AI (like DevSkim) are then required to include newtonsoft for the annotations to use the AddString methods.

    Describe the solution you'd like Ideally AppInspector could just use System.Text.Json for serialization in the rules path, using the standard attributes.

    enhancement good first issue 
    opened by gfs 0
  • Support .NET 7 Non-Backtracking Regex as option

    Support .NET 7 Non-Backtracking Regex as option

    Provide an option flag on the RuleProcessor to use non backtracking regex.

    https://devblogs.microsoft.com/dotnet/regular-expression-improvements-in-dotnet-7/#backtracking-and-regexoptions-nonbacktracking

    enhancement good first issue 
    opened by gfs 0
  • More Comprehensive Rules Verifier Self-Test Report

    More Comprehensive Rules Verifier Self-Test Report

    Rule self-tests only confirm that the rule as a whole matched a sample. It may be the case that a rule looks for multiple different things in the same category and you'd like to have self-tests to audit that you have samples for every pattern in every rule. The same perhaps for conditions.

    Other potential enhancements:

    1. Mechanism to indicate where the expected match in the must-match should be
    enhancement good first issue 
    opened by gfs 0
  • ERROR when running the Application Inspector in WSL 2

    ERROR when running the Application Inspector in WSL 2

    Describe the bug A clear and concise description of what the bug is.

    To Reproduce Steps to reproduce the behavior:

    1. Download the OWASP Benchmark by git clone https://github.com/OWASP-Benchmark/BenchmarkJava.
    2. Change the directory to the benchmark: cd BenchmarkJava.
    3. Run ApplicationInspector with appinspector analyze -s ./.
    4. See the error.

    Expected behavior

    Get the result file from the benchmark.

    Screenshots 屏幕截图 2022-07-30 235422

    Operating Environment (please complete the following information):

    • Application Inspector Version: ApplicationInspector.CLI 1.5.20+3790d0a4cb
    • OS: WSL 2 on Windows 10 [version 10.0.19044.1826].
    • .NET version: 6.0.302
    bug needs repro 
    opened by MarkLee131 7
  • Add test cases for CLi parameter parsing

    Add test cases for CLi parameter parsing

    Is your feature request related to a problem? Please describe. See #480 for an example of a regression in parameter parsing.

    Describe the solution you'd like 1.5 had a large refactor of the tests which cover the library well but not the CLI wrapper. There should be some test cases of the CLI specific bits like ensuring arguments are passed correctly.

    enhancement 
    opened by gfs 0
Releases(v1.6.25)
Owner
Microsoft
Open source projects and samples from Microsoft
Microsoft
🌘🦊 DalFox(Finder Of XSS) / Parameter Analysis and XSS Scanning tool based on golang

Finder Of XSS, and Dal(달) is the Korean pronunciation of moon. What is DalFox ?? ?? DalFox is a fast, powerful parameter analysis and XSS scanner, bas

HAHWUL 2.2k Dec 4, 2022
Lightweight static analysis for many languages. Find bug variants with patterns that look like source code.

Lightweight static analysis for many languages. Find bugs and enforce code standards. Semgrep is a fast, open-source, static analysis tool that finds

r2c 7.5k Dec 6, 2022
Static binary analysis tool to compute shared strings references between binaries and output in JSON, YAML and YARA

StrTwins StrTwins is a binary analysis tool, powered by radare, that is capable to find shared code string references between executables and output i

Anderson 2 May 3, 2022
PHP functions implementation to Golang. This package is for the Go beginners who have developed PHP code before. You can use PHP like functions in your app, module etc. when you add this module to your project.

PHP Functions for Golang - phpfuncs PHP functions implementation to Golang. This package is for the Go beginners who have developed PHP code before. Y

Serkan Algur 52 Aug 26, 2022
EarlyBird is a sensitive data detection tool capable of scanning source code repositories for clear text password violations, PII, outdated cryptography methods, key files and more.

EarlyBird is a sensitive data detection tool capable of scanning source code repositories for clear text password violations, PII, outdated cryptograp

American Express 516 Nov 20, 2022
Nuclei is a fast tool for configurable targeted vulnerability scanning based on templates offering massive extensibility and ease of use.

Fast and customisable vulnerability scanner based on simple YAML based DSL. How • Install • For Security Engineers • For Developers • Documentation •

ProjectDiscovery 10.7k Nov 29, 2022
Small utility package for stealing tokens from other processes and using them in current threads, or duplicating them and starting other processes

getsystem small utility for impersonating a user in the current thread or starting a new process with a duplicated token. must already be in a high in

Alex Flores 11 Sep 13, 2022
mesh-kridik is an open-source security scanner that performs various security checks on a Kubernetes cluster with istio service mesh and is leveraged by OPA (Open Policy Agent) to enforce security rules.

mesh-kridik Enhance your Kubernetes service mesh security !! mesh-kridik is an open-source security scanner that performs various security checks on a

chenk 22 Aug 28, 2022
Monitor your certificates and get notified before they expire.

Sifaka Sifaka is a tool to monitor your x509 certificates or simply websites certificates expirey date. If your organisation is handling some certs re

Jakub 6 Apr 16, 2022
A detector for the Trojan Source and other unicode-based vulnerabilities.

Trojan Source Detector This application detects Trojan Source attacks in source code. It can be used as part of the CI system to make sure there are n

Have you debugged.it? 5 Jan 6, 2022
ARP spoofing tool based on go language, supports LAN host scanning, ARP poisoning, man-in-the-middle attack, sensitive information sniffing, HTTP packet sniffing

[ARP Spoofing] [Usage] Commands: clear clear the screen cut 通过ARP欺骗切断局域网内某台主机的网络 exit exit the program help display help hosts 主机管理功能 loot 查看嗅探到的敏感信息

Re 54 Nov 27, 2022
WIP. Converts Azure Container Scan Action output to SARIF, for an easier integration with GitHub Code Scanning

container-scan-to-sarif container-scan-to-sarif converts Azure Container Scan Action output to Static Analysis Results Interchange Format (SARIF), for

Armel Soro 2 Jan 25, 2022
A FreeSWITCH specific scanning and exploitation toolkit for CVE-2021-37624 and CVE-2021-41157.

PewSWITCH A FreeSWITCH specific scanning and exploitation toolkit for CVE-2021-37624 and CVE-2021-41157. Related blog: https://0xinfection.github.io/p

Pinaki 24 Nov 2, 2022
The dynamic infrastructure framework for everybody! Distribute the workload of many different scanning tools with ease, including nmap, ffuf, masscan, nuclei, meg and many more!

Axiom is a dynamic infrastructure framework to efficiently work with multi-cloud environments, build and deploy repeatable infrastructure focussed on

pry0cc 3k Nov 27, 2022
Naabu - a port scanning tool written in Go that allows you to enumerate valid ports for hosts in a fast and reliable manner

Naabu is a port scanning tool written in Go that allows you to enumerate valid ports for hosts in a fast and reliable manner. It is a really simple tool that does fast SYN/CONNECT scans on the host/list of hosts and lists all ports that return a reply.

null 0 Jan 2, 2022
Portmantool - Port scanning and monitoring tool

portmantool Port scanning and monitoring tool Components runner while true do r

Thomann Bits & Beats 0 Feb 14, 2022
Gryffin is a large scale web security scanning platform.

Gryffin (beta) Gryffin is a large scale web security scanning platform. It is not yet another scanner. It was written to solve two specific problems w

Yahoo 2.1k Nov 30, 2022
A fully self-contained Nmap like parallel port scanning module in pure Golang that supports SYN-ACK (Silent Scans)

gomap What is gomap? Gomap is a fully self-contained nmap like module for Golang. Unlike other projects which provide nmap C bindings or rely on other

jtimperio 74 Nov 9, 2022
A Large killer focused on intranet scanning

FscanX 其实FscanX的灵感来源于fscan和LodanGo这两个开源项目,首先不得不说fscan和LadonGo两个都是非常优秀的内网扫描器。并且其独自的特色也让其在内网扫描器领域独占鳌头。其中LadonGo的插件式让其在扫描时更加专注,而fscan的傻瓜式则让其对内网的信息搜集更加高效。

SaiRson 203 Dec 31, 2021