Plugin Development

In this guide we are going to learn how to develop a new plugin for Faraday.

Configure Custom Plugins¶

To add custom plugins in faraday, you first need to run this command:

faraday-manage settings -a update reports

Create Plugin structure¶

Create the folder new_plugin inside the custom_plugins folder and add the following files inside the new_plugin folder:

Plugin Structure:

new_plugin/
    __init__.py # leave this file empty
    plugin.py

Warning

Please respect both file names: __init__.py and plugin.py

Plugin code¶

There are two types of plugins:

File plugins (plugins that process the contents of a file)
Command plugins (plugins that process the execution of a command)

A faraday plugin can be either or both of these types. For example the ping plugin only process the execution of the command, but the nmap command supports both the xml generated by nmap or process the execution of nmap.

In your plugin.py file you must provide a function called createPlugin for the plugin manager to instantiate the plugin

def createPlugin(ignore_info=False, hostname_resolution=True):
    return YourPluginClass(ignore_info=ignore_info)

Plugin classes¶

PluginBase is the base class.

For File plugins we provide all this classes. But you can add other by creating a subclass of PluginByExtension

PluginBase
PluginByExtension
- PluginXMLFormat
- PluginJsonFormat
- PluginMultiLineJsonFormat
- PluginCSVFormat
- PluginZipFormat

For Command plugins only you must inherit from PluginBase

Your plugin class __init__ method must define the id, that will be the identifier of your plugin.

Here's an example of the Ping tool plugin:

class CmdPingPlugin(PluginBase):
    def __init__(self, *arg, **kwargs):
        super().__init__(*arg, **kwargs)
            self.id = "ping"
            self.name = "Ping"
            self.plugin_version = "0.0.1"
            self.version = "1.0.0"
            self._command_regex = re.compile(r'^(sudo ping|ping|sudo ping6|ping6)\s+.*?')

Beware that it must be unique, you can list all the plugins identifiers with this command: faraday-plugins list-plugins

File Plugins¶

Each of the subclasses for File plugins provide a different way for the plugin to detect if it can process the file

PluginByExtension:¶

This class adds the functionality for identify a file by its extension by setting the attribute extension.

It can be one extension (".xml") or a list (['.xml', '.xxx']) if your file can have multiple extensions.

You never inherit directly from this class in a plugin but from some of its subclasses that are more specific.

class ExampleXMLTool(PluginByExtension):

        def __init__(self):
            super().__init__()
            self.extension = ".xml"

PluginXMLFormat:¶

Use this class if the plugin generates vulnerabilities from a xml file.

If your report has xml format but a different extension (like nessus), remember to define the extension attribute

To identify the file set in the identifier_tag attribute the main tag of the xml (ScanGroup in the example), it also can be one tag or a list of tags.

You can also set the identifier_tag_attributes attribute, it is a set of attributes that the main tag must have.

This is used to be more specific for example if the main tag is too generic and the xml of other plugin has the same one.

Example XML

<?xml version="1.0"?>
<ScanGroup ExportedOn="2021-11-05T11:18:57.673843">
    <Scan>
    </Scan>
</ScanGroup>

from urllib.parse import urlparse
from faraday_plugins.plugins.plugin import PluginXMLFormat
import xml.etree.ElementTree as ET

class ExampleToolXmlParser:

    def __init__(self, xml_output):
        self.vulns = self.parse_xml(xml_output)

    def parse_xml(self, xml_output):
        vulns = []
        tree = ET.fromstring(xml_output)
        items = tree.iterfind('details/item')
        for item in items:
            ip = item.get('ip')
            os = item.get('os')
            uri = item.find('uri').text
            url = urlparse(uri)
            hostname = [url.netloc]
            path = url.path
            if url.scheme == 'https':
                port = 443
            else:
                port = 80
            issue = item.find('issue')
            severity = issue.get('severity')
            issue_text = issue.text
            vuln = {'ip': ip, 'uri': uri, 'os': os,
                    'hostname': hostname, 'port': port, 'path': path,
                    'issue_text': issue_text, 'severity': severity}
            vulns.append(vuln)
        return vulns


class ExampleToolPlugin(PluginXMLFormat):
    def __init__(self, *arg, **kwargs):
        super().__init__(*arg, **kwargs)
        self.identifier_tag = "example_tool"
        self.id = "example_tool"
        self.name = "Name of the tool"
        self.plugin_version = "0.0.1"

    def parseOutputString(self, output, debug=False):
        parser = ExampleToolXmlParser(output)
        for vuln in parser.vulns:
            h_id = self.createAndAddHost(vuln['ip'], vuln['os'], hostnames=vuln['hostname'])
            s_id = self.createAndAddServiceToHost(h_id, 'webserver', protocol='tcp', ports=vuln['port'])
            v_id = self.createAndAddVulnWebToService(h_id, s_id, vuln['issue_text'], severity=vuln['severity'],
                                                    path=vuln['path'])

def createPlugin(*args, **kwargs):
    return ExampleToolPlugin(*args, **kwargs)

PluginJsonFormat:¶

Use this class if the plugin generates vulnerabilities from a json file.

If yor report has json format but a different extension (not .json), remember to define the extension attribute

To identify the file set in the json_keys attribute a set with some identifiers keys of the json object

Example JSON

{
  "name": "some name",
  "hosts": ["host1", "host2"],
  "other_field": "some value"
}

class ExampleJsonTool(PluginJsonFormat):
    def __init__(self, ignore_info):
        super().__init__(ignore_info)
        self.json_keys = {'name', 'hosts'}

PluginMultiLineJsonFormat:¶

Is the same as PluginJsonFormat but use it when the file has multiple lines and every line is a json.

Example JSON

{ "name": "some name", "hosts": ["host1", "host2"],"other_field": "some value"}
{ "name": "other name", "hosts": ["host3", "host4"],"other_field": "other value"}

class ExampleMultipleJsonTool(PluginMultiLineJsonFormat):
    def __init__(self, ignore_info):
        super().__init__(ignore_info)
        self.json_keys = {'name', 'hosts'}

PluginCSVFormat:¶

Use this class if the plugin generates vulnerabilities from a csv file.

If yor report has csv format but a different extension (not .csv), remember to define the extension attribute

To identify the file set in the csv_headers attribute a set with some identifiers headers of the csv object

Example CSV

name,ip,os
host1,10.10.10.10,windows
host2,10.10.10.11,linux

class ExampleCSVTool(PluginCSVFormat):
    def __init__(self, ignore_info):
        super().__init__(ignore_info)
        self.json_keys = {'name', 'ip'}

PluginZipFormat:¶

Use this class if the plugin generates vulnerabilities from a zip file.

If yor report has csv format but a different extension (not .zip), remember to define the extension attribute

To identify the file set in the files_list attribute a set with some file names inside the zip file.

class ExampleJsonTool(PluginZipFormat):
    def __init__(self, ignore_info):
        super().__init__(ignore_info)
        self.files_list = {'file1.txt', 'file2.txt'}

Generating data¶

The PluginBase class give you all the main methods to create hosts, services and vulnerabilities

After you parse the data you will need to create the objects with the information to send to faraday. This are the main methods

def createAndAddHost(self, name, os="unknown", hostnames=None, mac=None, scan_template="", site_name="",
                    site_importance="", risk_score="", fingerprints="", fingerprints_software=""):

def createAndAddServiceToHost(self, host_id, name, protocol="tcp?", ports=None, status="open", version="unknown",
                            description=""):

def createAndAddVulnToHost(self, host_id, name, desc="", ref=None, severity="", resolution="", vulnerable_since="", 
                            scan_id="", pci="", data="", external_id=None, run_date=None):

def createAndAddVulnToService(self, host_id, service_id, name, desc="", ref=None, severity="", resolution="", 
                            risk="", data="", external_id=None, run_date=None):

def createAndAddVulnWebToService(self, host_id, service_id, name, desc="", ref=None, severity="", resolution="",
                                website="", path="", request="", response="", method="", pname="",
                                params="", query="", category="", data="", external_id=None, run_date=None):

createAndAddHost and createAndAddServiceToHost will give you a host ID and a service ID. You need to send those to the create vulnerabilities methods

The main method to parse data and create the vulnerabilities objects is parseOutputString, no mather if is a File or a Command plugin.

Command Plugins¶

For command plugins the detection is done via regex.

If the plugin is only for command you inherit from PluginBase if it also will support file detection inherit from the appropriate class for that type of file.

In both cases to detect a command you will need to set the _command_regex attribute with the regex to match the command

Here's an example of the Ping tool plugin:

class CmdPingPlugin(PluginBase):
    def __init__(self, *arg, **kwargs):
        super().__init__(*arg, **kwargs)
            self.id = "ping"
            self.name = "Ping"
            self.plugin_version = "0.0.1"
            self.version = "1.0.0"
            self._command_regex = re.compile(r'^(sudo ping|ping|sudo ping6|ping6)\s+.*?')

    def parseOutputString(self, output):
        reg = re.search(r"PING ([\w\.-:]+)( |)\(([\w\.:]+)\)", output)
        if re.search("0 received|unknown host", output) is None and reg is not None:
            ip_address = reg.group(3)
            hostname = reg.group(1)
            self.createAndAddHost(ip_address, hostnames=[hostname])
        return True

    def _isIPV4(self, ip):
        if len(ip.split(".")) == 4:
            return True
        else:
            return False

In this case the plugin manager will run the command and send the output to the parseOutputString method.

But if the tool generate an ouput file instead of send the data to stdout you will need to check that the command has the required parameters to do that.

To do that you will need to implement the processCommandString method, by default it will return the command without modifications.

To use output files you will need to set 2 attributes in you plugin: _use_temp_file and _temp_file_extension

If _use_temp_file is set the plugin will create a temp file and assign its path to the attribute self._output_file_path of the plugin.

If you need that the temp file has and specific extension use the _temp_file_extension attribute like in the example.

If the _use_temp_file attribute is set the plugin will send te content of that file to parseOutputString()instead of the output of the command.

Here is and extract of the nmap plugin to see how processCommandString will modify the original command and return a new command with the required parameters.

class NmapPlugin(PluginXMLFormat):
    """
    Example plugin to parse nmap output.
    """

    def __init__(self, *arg, **kwargs):
        super().__init__(*arg, **kwargs)
        self.identifier_tag = "nmaprun"
        self.id = "Nmap"
        self.name = "Nmap XML Output Plugin"
        self.plugin_version = "0.0.3"
        self.version = "6.40"
        self.framework_version = "1.0.0"
        self.options = None
        self._current_output = None
        self._command_regex = re.compile(r'^(sudo nmap|nmap|\.\/nmap)\s+.*?')
        self._use_temp_file = True
        self._temp_file_extension = "xml"
        self.xml_arg_re = re.compile(r"^.*(-oX\s*[^\s]+).*$")
        self.addSetting("Scan Technique", str, "-sS")

    def parseOutputString(self, output):
        ...

    def processCommandString(self, username, current_path, command_string):
        """
        Adds the -oX parameter to get xml output to the command string that the
        user has set.
        """
        super().processCommandString(username, current_path, command_string)
        arg_match = self.xml_arg_re.match(command_string)
        if arg_match is None:
            return re.sub(r"(^.*?nmap)",
                          r"\1 -oX %s" % self._output_file_path,
                          command_string)
        else:
            return re.sub(arg_match.group(1),
                          r"-oX %s" % self._output_file_path,
                          command_string)

Full Example¶

File Plugin from XML

This is an example of a Faraday Plugin that process a xml report.

In this example we will create a plugin to analyze this XML provided by the output of a tool.

example.xml

 <?xml version="1.0" ?>
<!DOCTYPE example_tool>
<example_tool scanstart="Thu Nov  9 15:59:13 2017">
    <details>
    <item id="999979" ip="10.23.49.232" os="linux">
    <uri>http://test.com/example.php</uri>
    <issue severity="low">Some vuln text</issue>
    </item>
    <item id="39023023" ip="10.232.62.20" os="linux">
    <uri>http://test.com/login.php</uri>
    <issue severity="low">Some other text</issue>
    </item>
    <item id="8348343" ip="10.12.37.24" os="linux">
    <uri>http://test.com/example.php</uri>
    <issue severity="low">Yet another vuln text</issue>
    </item>
    <statistics elapsed="402" itemsfound="3" itemstested="10" />
    </details>
</example_tool>

plugin.py

from urllib.parse import urlparse
from faraday_plugins.plugins.plugin import PluginXMLFormat
import xml.etree.ElementTree as ET

class ExampleToolXmlParser:

    def __init__(self, xml_output):
        self.vulns = self.parse_xml(xml_output)

    def parse_xml(self, xml_output):
        vulns = []
        tree = ET.fromstring(xml_output)
        items = tree.iterfind('details/item')
        for item in items:
            ip = item.get('ip')
            os = item.get('os')
            uri = item.find('uri').text
            url = urlparse(uri)
            hostname = [url.netloc]
            path = url.path
            if url.scheme == 'https':
                port = 443
            else:
                port = 80
            issue = item.find('issue')
            severity = issue.get('severity')
            issue_text = issue.text
            vuln = {'ip': ip, 'uri': uri, 'os': os,
                    'hostname': hostname, 'port': port, 'path': path,
                    'issue_text': issue_text, 'severity': severity}
            vulns.append(vuln)
        return vulns


class ExampleToolPlugin(PluginXMLFormat):
    def __init__(self):
        super().__init__()
        self.identifier_tag = "example_tool"
        self.id = "example_tool"
        self.name = "Name of the tool"
        self.plugin_version = "0.0.1"

    def parseOutputString(self, output, debug=False):
        parser = ExampleToolXmlParser(output)
        for vuln in parser.vulns:
            h_id = self.createAndAddHost(vuln['ip'], vuln['os'], hostnames=vuln['hostname'])
            s_id = self.createAndAddServiceToHost(h_id, 'webserver', protocol='tcp', ports=vuln['port'])
            v_id = self.createAndAddVulnWebToService(h_id, s_id, vuln['issue_text'], severity=vuln['severity'],
                                                    path=vuln['path'])

def createPlugin(ignore_info=False, hostname_resolution=True):
    return ExampleToolXmlParser(ignore_info=ignore_info)

Test and Debug¶

You can test your plugin by enabling the custom_plugins_folder setting, and try it with faraday.

You can also test your plugin from the command line.

Test Plugins¶

List all available plugins

Verify that the plugins is loaded by the plugin manager

faraday-plugins list-plugins

Available Plugins:
...
...
...
example_tool - Name of the tool
Loaded Plugins: 84

Test plugin report detection

Verify that your file is detected by your plugin

faraday-plugins detect-report /path/to/report.xml

Plugin: example_tool

Test plugin process report

Verify that your plugin parses the file ok and generate the json structure that will be loaded into faraday

faraday-plugins process-report /path/to/report.xml


{"hosts": [{"ip": "10.23.49.232", "os": "linux", "hostnames": ["test.com"], "description": "", "mac": null, "credentials": [], "services": [{"name": "webserver", "protocol": "tcp", "port": 80, "status": "open", "version": "unknown", "description": "", "credentials": [], "vulnerabilities": [{"name": "Some vuln text", "desc": "", "severity": "low", "refs": [], "external_id": null, "type": "VulnerabilityWeb", "resolution": "", "data": "", "website": "", "path": "/example.php", "request": "", "response": "", "method": "", "pname": "", "params": "", "query": "", "category": ""}]}], "vulnerabilities": [], "scan_template": "", "site_name": "", "site_importance": "", "risk_score": "", "fingerprints": "", "fingerprints_software": ""}, {"ip": "10.232.62.20", "os": "linux", "hostnames": ["test.com"], "description": "", "mac": null, "credentials": [], "services": [{"name": "webserver", "protocol": "tcp", "port": 80, "status": "open", "version": "unknown", "description": "", "credentials": [], "vulnerabilities": [{"name": "Some other text", "desc": "", "severity": "low", "refs": [], "external_id": null, "type": "VulnerabilityWeb", "resolution": "", "data": "", "website": "", "path": "/login.php", "request": "", "response": "", "method": "", "pname": "", "params": "", "query": "", "category": ""}]}], "vulnerabilities": [], "scan_template": "", "site_name": "", "site_importance": "", "risk_score": "", "fingerprints": "", "fingerprints_software": ""}, {"ip": "10.12.37.24", "os": "linux", "hostnames": ["test.com"], "description": "", "mac": null, "credentials": [], "services": [{"name": "webserver", "protocol": "tcp", "port": 80, "status": "open", "version": "unknown", "description": "", "credentials": [], "vulnerabilities": [{"name": "Yet another vuln text", "desc": "", "severity": "low", "refs": [], "external_id": null, "type": "VulnerabilityWeb", "resolution": "", "data": "", "website": "", "path": "/example.php", "request": "", "response": "", "method": "", "pname": "", "params": "", "query": "", "category": ""}]}], "vulnerabilities": [], "scan_template": "", "site_name": "", "site_importance": "", "risk_score": "", "fingerprints": "", "fingerprints_software": ""}], "command": {"tool": "example_tool", "command": "example_tool", "params": "/path/to/report.xml", "user": "faraday", "hostname": "", "start_date": "2020-04-01T18:43:34.552623", "duration": 2650, "import_source": "report"}}

You can optionally specify the plugin id to not do the detection step and force to process it with the specific plugin

faraday-plugins process-report --plugin_id YourPluginId /path/to/report.xml


{"hosts": [{"ip": "10.23.49.232", "os": "linux", "hostnames": ["test.com"], "description": "", "mac": null, "credentials": [], "services": [{"name": "webserver", "protocol": "tcp", "port": 80, "status": "open", "version": "unknown", "description": "", "credentials": [], "vulnerabilities": [{"name": "Some vuln text", "desc": "", "severity": "low", "refs": [], "external_id": null, "type": "VulnerabilityWeb", "resolution": "", "data": "", "website": "", "path": "/example.php", "request": "", "response": "", "method": "", "pname": "", "params": "", "query": "", "category": ""}]}], "vulnerabilities": [], "scan_template": "", "site_name": "", "site_importance": "", "risk_score": "", "fingerprints": "", "fingerprints_software": ""}, {"ip": "10.232.62.20", "os": "linux", "hostnames": ["test.com"], "description": "", "mac": null, "credentials": [], "services": [{"name": "webserver", "protocol": "tcp", "port": 80, "status": "open", "version": "unknown", "description": "", "credentials": [], "vulnerabilities": [{"name": "Some other text", "desc": "", "severity": "low", "refs": [], "external_id": null, "type": "VulnerabilityWeb", "resolution": "", "data": "", "website": "", "path": "/login.php", "request": "", "response": "", "method": "", "pname": "", "params": "", "query": "", "category": ""}]}], "vulnerabilities": [], "scan_template": "", "site_name": "", "site_importance": "", "risk_score": "", "fingerprints": "", "fingerprints_software": ""}, {"ip": "10.12.37.24", "os": "linux", "hostnames": ["test.com"], "description": "", "mac": null, "credentials": [], "services": [{"name": "webserver", "protocol": "tcp", "port": 80, "status": "open", "version": "unknown", "description": "", "credentials": [], "vulnerabilities": [{"name": "Yet another vuln text", "desc": "", "severity": "low", "refs": [], "external_id": null, "type": "VulnerabilityWeb", "resolution": "", "data": "", "website": "", "path": "/example.php", "request": "", "response": "", "method": "", "pname": "", "params": "", "query": "", "category": ""}]}], "vulnerabilities": [], "scan_template": "", "site_name": "", "site_importance": "", "risk_score": "", "fingerprints": "", "fingerprints_software": ""}], "command": {"tool": "example_tool", "command": "example_tool", "params": "/path/to/report.xml", "user": "faraday", "hostname": "", "start_date": "2020-04-01T18:43:34.552623", "duration": 2650, "import_source": "report"}}

If you do not have faraday-server installed or don't have the custom_plugins_folder setting, you can use the --custom_plugins_folder parameter with any if the commands (list, detect and process)

Example:

faraday-plugins list-plugins --custom-plugins-folder /home/user/.faraday/plugins/

Logging¶

In the PluginBase there is a logger defined in self.logger that you can use.

If you need to debug for plugins with the command line set this variable:

export PLUGIN_DEBUG=1
faraday-plugins process-report appscan /path/to/report.xml
2019-11-15 20:37:03,355 - faraday.faraday_plugins.plugins.manager - INFO [manager.py:113 - _load_plugins()]  Loading Native Plugins...
2019-11-15 20:37:03,465 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()]  Load Plugin [acunetix]
2019-11-15 20:37:03,495 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()]  Load Plugin [amap]
2019-11-15 20:37:03,549 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()]  Load Plugin [appscan]
2019-11-15 20:37:03,580 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()]  Load Plugin [arachni]
2019-11-15 20:37:03,613 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()]  Load Plugin [arp_scan]
2019-11-15 20:37:03,684 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()]  Load Plugin [beef]
2019-11-15 20:37:03,714 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()]  Load Plugin [brutexss]
2019-11-15 20:37:03,917 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()]  Load Plugin [burp]
2019-11-15 20:37:03,940 - faraday.faraday_plugins.plugins.manager - DEBUG [manager.py:123 - _load_plugins()]  Load Plugin [dig]
...